What's changed: Created Associate Data Practitioner Chapter 1 (Domain 1 "Preparation and ingestion": methodologies and quality = ETL/ELT/ETLT/data quality/cleaning (Cloud Data Fusion/BigQuery/Dataflow)/transfer (Storage Transfer Service/Transfer Appliance); formats and extract/load = CSV/JSON/Parquet/Avro/BigQuery Data Transfer Service/Database Migration Service/CLI; choosing storage and location = destinations/structured classification/regional-dual-multi-zonal).
1.2Data formats and extract/load
Understand data formats (CSV, JSON, Apache Parquet, Apache Avro, structured tables), choosing extraction tools (Dataflow, BigQuery Data Transfer Service, Database Migration Service, Cloud Data Fusion), and loading into Google Cloud storage with the gcloud / bq CLI, Storage Transfer Service, and client libraries.
To ingest data, understand its format, choose an extraction tool based on source and goal, and use the right load method to land it in Google Cloud storage.
1.2.1Data formats
Common formats include row-oriented, human-readable CSV, semi-structured JSON, and analytics-friendly columnar Apache Parquet, plus row-oriented Apache Avro that embeds a schema. For large-scale analytics, columnar formats like Parquet are efficient. Also note whether a format is structured (tables), semi-structured (JSON), or unstructured.
1.2.2Extraction tools and loading
| Goal | Tool |
|---|---|
| Scheduled ingestion from SaaS/services to BigQuery | BigQuery Data Transfer Service |
| Migrate a database (minimal downtime) | Database Migration Service |
| Pipeline with flexible transforms | Dataflow / Cloud Data Fusion |
| Small uploads | gcloud / bq CLI, client libraries |
| Large/continuous object transfer | Storage Transfer Service |
Choose extraction by goal: BigQuery Data Transfer Service for scheduled ingestion from other services into BigQuery, Database Migration Service to migrate an existing DB, and Dataflow or Cloud Data Fusion for flexible transforms. Choose loading by scale: gcloud / bq CLI or client libraries for small data, Storage Transfer Service for large/continuous objects, and loading from Cloud Storage to bring data into BigQuery.
Common: requirement → tool. E.g., "columnar, analytics-efficient" = Parquet; "row-oriented with embedded schema" = Avro; "scheduled ingestion from services to BigQuery" = BigQuery Data Transfer Service; "migrate a DB with minimal downtime" = Database Migration Service; "small upload" = gcloud/bq CLI; "large object transfer" = Storage Transfer Service.
Watch the mix-ups: (1) BigQuery Data Transfer Service (scheduled into BigQuery) vs Storage Transfer Service (into Cloud Storage) target different destinations. (2) Database Migration Service is specifically for DB migration. (3) Columnar (Parquet) is best for analytics.
1.2.3Section summary
- Formats = CSV/JSON (semi-structured)/Parquet (columnar, analytics)/Avro (schema-embedded)
- Extraction = BigQuery Data Transfer Service (scheduled)/Database Migration Service (DB migration)/Dataflow, Cloud Data Fusion
- Loading = gcloud/bq CLI (small)/Storage Transfer Service (large)/load from Cloud Storage
Sign in to track progress — Log in.
Quick check
(just a quick review)Q1. Which columnar, analytics-efficient data format suits large-scale analytics?
Q2. To regularly ingest data from other SaaS/services into BigQuery, which tool is best?
Q3. To migrate an existing database to Google Cloud with minimal downtime, which service is best?
Q4. To upload small data to Cloud Storage or BigQuery from the command line, which is appropriate?
Q5. Which correctly contrasts BigQuery Data Transfer Service and Storage Transfer Service?
Q6. Which row-oriented data format embeds a schema and is used for data exchange?