GCP Data Engineer Interview Questions cheat sheet
BigQuery, Dataflow, Pub/Sub, Dataproc, and Composer questions with the design trade-offs Google Cloud data interviews expect.
Choosing the right service
How do you choose between BigQuery, Bigtable, Cloud SQL, and Spanner?- BigQuery for analytical scans and aggregation over large tables. Bigtable for high-throughput key-value and time-series lookups with single-digit millisecond latency and no joins. Cloud SQL for a normal relational OLTP workload that fits on one machine. Spanner when you need relational semantics with horizontal scale and strong global consistency, at a notably higher price.
When would you use Dataflow instead of Dataproc?- Dataflow is serverless Apache Beam with autoscaling and no cluster to manage, and one pipeline can run in batch or streaming with the same code. Dataproc is managed Spark and Hadoop, which you pick to lift existing Spark jobs, control versions and native libraries, or exploit preemptible VMs aggressively. Familiarity and existing code usually decide it.
What is Pub/Sub and how does it differ from Kafka?- A fully managed global messaging service with no partitions or capacity planning; it scales automatically and delivers at-least-once by default. Unlike Kafka there is no consumer-controlled offset by default, ordering requires an ordering key within a region, and default retention is 7 days. Pub/Sub Lite existed for cost-sensitive partitioned workloads but has been retired.
What is Datastream for?- Serverless change data capture from Oracle, MySQL, PostgreSQL, and SQL Server into BigQuery or Cloud Storage, reading the database log so it captures updates and deletes without querying the source. Its limits are the supported source versions and DDL handling, and initial backfill of a very large table still puts real load on the source.
What does Dataplex add to a lake?- It organizes storage buckets and BigQuery datasets into logical lakes and zones with unified metadata, data quality scans, profiling, and governance policies across them. It is the answer to 'how do you govern data spread across GCS and BigQuery without moving it', and it now carries much of what used to be Data Catalog.
BigQuery essentials
How is BigQuery billed and how do you control it?- On-demand pricing charges per terabyte scanned with a 10 MB minimum per query; capacity pricing buys slots as editions with autoscaling and commitments. Control cost with partition and cluster pruning, avoiding SELECT star, custom quotas per project or user, and maximum bytes billed on a query so a runaway scan fails instead of costing thousands.
Compare partitioning and clustering in BigQuery.- Partitioning splits a table physically by ingestion time, a date or timestamp column, or an integer range, with a limit of 10,000 partitions per table and the ability to require a partition filter. Clustering sorts data within partitions by up to four columns, pruning blocks for filters and joins on those columns. Use partitioning for the date and clustering for the high-cardinality filters.
What is a slot and what happens when you run out?- A slot is a unit of BigQuery compute; a query is decomposed into stages that consume slots in parallel. When demand exceeds the reservation, queries queue rather than fail, so the symptom is rising latency with no error. Diagnose it with INFORMATION_SCHEMA.JOBS slot_ms and the reservation charts, then add autoscaling slots or move workloads to separate reservations.
How do you load data into BigQuery, and which method when?- Batch load jobs from GCS are free of query cost and best for bulk; the Storage Write API is the current streaming path, replacing legacy streaming inserts, with exactly-once semantics using stream offsets. External tables and BigLake query files in place without loading. Avoid single-row DML inserts, which are slow and hit table modification quotas.
What is BigQuery Omni and BigLake?- BigLake tables let BigQuery govern and query open-format data (Parquet, Iceberg) in GCS, S3, or Azure with fine-grained security, and BigQuery Omni runs the BigQuery engine in AWS or Azure regions so you can query data that cannot leave those clouds. Both target the same problem: analytics without a copy job across boundaries.
Dataflow and Beam
What are the core Beam concepts?- A Pipeline contains PCollections (distributed datasets, bounded or unbounded) transformed by PTransforms; ParDo is the general per-element transform and GroupByKey the shuffle primitive. The unified model means the same pipeline runs batch or streaming, differing only in whether the PCollection is bounded and which windowing you apply.
Explain windowing, watermarks, and triggers.- Windowing groups unbounded data by event time into fixed, sliding, or session windows. The watermark is the runner's estimate of how far event time has progressed, and it decides when a window can fire. Triggers say when to emit — at the watermark, early on a count or processing-time interval, or late when stragglers arrive — combined with accumulation mode.
How do you handle late data in a streaming pipeline?- Set allowed lateness so windows retain state past the watermark, add a late trigger to emit corrections, and choose accumulating mode if downstream can absorb restatements or discarding mode if it applies deltas. Data later than the allowance is dropped, so route it to a side output and monitor the volume rather than losing it silently.
What is Dataflow Shuffle and Streaming Engine?- They move shuffle and streaming state off the worker VMs into the Dataflow backend service, so workers become stateless. That makes autoscaling much faster and more aggressive, reduces worker disk requirements, and avoids the long rebalancing that stateful workers caused. Both are the default on new pipelines and are billed as a separate service component.
How do you deal with a hot key in Dataflow?- Symptoms are one worker at full CPU while autoscaling adds idle workers. Fixes are withFanout or withHotKeyFanout on combines to pre-aggregate with a random sub-key, using Combine rather than GroupByKey so partial aggregation happens map-side, or reshuffling with a salted key. Enabling hot key logging identifies the offending key first.
Pub/Sub and streaming design
How do push and pull subscriptions differ?- Pull has the subscriber request messages and acknowledge them, giving flow control and batching, and it is the normal choice for Dataflow and long-running consumers. Push has Pub/Sub POST to an HTTPS endpoint with backoff, which suits Cloud Run and Functions. StreamingPull is the low-latency variant used by most client libraries.
What delivery guarantees does Pub/Sub give?- At-least-once by default, so duplicates are expected and consumers must be idempotent; exactly-once delivery is available within a region for pull subscriptions. Ordering is off unless you set an ordering key and enable message ordering, which limits throughput per key because messages for that key are delivered sequentially.
What is the ack deadline and what breaks when it is wrong?- The time a subscriber has to acknowledge before redelivery, default 10 seconds and extendable to 10 minutes, with client libraries auto-extending while processing. If processing outlasts the deadline without extension, the message is redelivered and you get duplicate work — a common cause of mysterious double-processing in slow consumers.
How do dead letter topics and retry policies work?- A subscription can forward a message to a dead letter topic after a configured number of delivery attempts (5 to 100), and retry policy adds exponential backoff between attempts. Without this, a poison message is redelivered until it expires from retention, consuming capacity. Alert on dead letter volume because it represents unprocessed data.
How would you build a real-time analytics pipeline on GCP?- Pub/Sub for ingestion, Dataflow for windowed enrichment and aggregation with watermarks, the Storage Write API into partitioned and clustered BigQuery tables, and Looker or a materialized view for serving. Write raw events to GCS in parallel as a replay path, because reprocessing from the sink is impossible once the stream retention expires.
Operations and governance
What is Cloud Composer and when do you avoid it?- Managed Apache Airflow on GKE, appropriate when you need cross-service dependency graphs, backfills, and the Airflow operator ecosystem. Avoid it for simple time-based triggers, where Cloud Scheduler plus Workflows is far cheaper, since a Composer environment costs money continuously even when idle.
How does IAM work for data services on GCP?- Roles are bound to principals at the organization, folder, project, or resource level and inherit downward; service accounts are the identities workloads use. Prefer predefined roles over primitive owner and editor, use Workload Identity Federation instead of downloaded service account keys, and remember BigQuery adds dataset, table, column, and row policies below project-level IAM.
How do you implement column and row level security in BigQuery?- Column-level security uses policy tags from a Dataplex taxonomy attached to columns, with the Fine-Grained Reader role controlling access, and dynamic data masking applies a masking rule instead of denying. Row-level security creates row access policies with a filter predicate per grantee. Both are enforced on the table, so views cannot bypass them.
How do you monitor and troubleshoot a failing pipeline?- Cloud Logging with log-based metrics and alerting policies in Cloud Monitoring, the Dataflow job graph for stuck stages and system lag, Pub/Sub oldest unacked message age as the streaming health signal, and BigQuery INFORMATION_SCHEMA for slow or expensive queries. Oldest unacked age is the metric that tells you a consumer is falling behind before anything errors.
How would you keep a multi-team GCP data platform organized?- Separate projects per environment and per domain so quotas, billing, and IAM blast radius are contained, with shared VPC and a central logging project. Datasets get consistent naming and labels for cost attribution, and infrastructure is declared in Terraform. The failure mode to describe is one giant project where nobody can tell whose query caused the bill.
From DataLane — tutorials at/blog, practice SQL live in theplayground.