GCP for Data Engineers cheat sheet
BigQuery, Dataflow, Pub/Sub, Composer, and Dataplex — services, gcloud commands, and cost levers.
The reference architecture
Pub/Sub → BigQuery subscription- The simplest streaming path. No Dataflow job to operate; use it unless you need per-event enrichment or joins.
Pub/Sub → Dataflow → BigQuery- Add Beam only for stateful processing, windowed joins, or heavy transformation. It is real infrastructure to own.
GCS → BigQuery external table / BigLake- Query Parquet in place without loading. BigLake adds fine-grained access control over the same files.
Cloud Composer- Managed Airflow for cross-service orchestration. Overkill if scheduled queries and Dataform cover your DAG.
BigQuery essentials
bq query --use_legacy_sql=false --maximum_bytes_billed=1000000000 'select ...'- Byte ceiling as a guardrail. The query fails fast instead of scanning a surprise 40 TB.
bq mk --table --time_partitioning_field=ordered_at --clustering_fields=customer_id ds.orders- Partition by the date every query filters on, then cluster by the next most common filter.
bq load --source_format=PARQUET ds.orders gs://bucket/orders/*.parquet- Batch load is free; streaming inserts are not. Prefer files when latency tolerance allows.
bq show --schema --format=prettyjson ds.orders- Export the schema for version control or to recreate the table in another project.
bq cp ds.orders ds.orders_backup- Metadata-only copy within a region — near-instant and cheap, the BigQuery equivalent of a clone.
Pub/Sub
gcloud pubsub topics create orders- Topics are cheap; one per event type with a schema beats a single firehose topic.
gcloud pubsub subscriptions create orders-bq --topic=orders --bigquery-table=proj:ds.orders- Direct BigQuery subscription. Removes an entire Dataflow job from the architecture.
--ack-deadline=600 --dead-letter-topic=orders-dlq- Deadline must exceed worst-case processing or messages redeliver forever. A DLQ stops poison pills looping.
--enable-exactly-once-delivery- Available on regional subscriptions. Your sink still needs idempotency for true end-to-end guarantees.
gcloud pubsub subscriptions seek --time=2026-08-01T00:00:00Z- Replay within the retention window. Set message retention deliberately; the default is short.
Dataflow
gcloud dataflow flex-template run job --parameters inputSubscription=...- Flex templates keep job definitions in a container so deploys are reproducible.
--enable-streaming-engine --max-workers=10- Streaming Engine moves shuffle and state off the workers, cutting cost and speeding autoscaling.
--autoscaling_algorithm=THROUGHPUT_BASED- Correct for streaming. Without it a traffic spike backs up the subscription while workers stay flat.
gcloud dataflow jobs drain <id>- Drain finishes in-flight windows; cancel discards them. Use drain for any job whose output must be complete.
Governance and IAM
gcloud projects add-iam-policy-binding --role=roles/bigquery.dataViewer- Prefer dataset-level grants over project-level. bigquery.admin at project scope is the audit finding you will get.
bq add-iam-policy-binding --table ds.orders- Table-level access for shared datasets where team boundaries do not match dataset boundaries.
CREATE ROW ACCESS POLICY ... FILTER USING (region = SESSION_USER())- Row-level security inside BigQuery, evaluated per query and cheaper than maintaining filtered views.
Dataplex data quality scans- Scheduled profiling and rule checks across projects, with results queryable in BigQuery.
Cost control
ALTER PROJECT SET OPTIONS (`region-us.default_time_zone` ...)- Project-level query defaults, including maximum bytes billed, applied without touching every query.
select user_email, sum(total_bytes_billed)/pow(2,40) as tib from `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT where creation_time > current_timestamp - interval 30 day group by 1 order by 2 desc- The monthly spend audit. Nearly always reveals one dashboard scanning an unpartitioned table.
BigQuery editions with autoscaling reservations- Switch from on-demand when monthly scan spend is high and steady. Baseline plus autoscale beats fixed slots.
gcloud billing budgets create --threshold-rule=percent=0.8- Ten minutes of setup that prevents the surprise invoice conversation.
From DataLane — tutorials at/blog, practice SQL live in theplayground.