GCP Professional Data Engineer Practice Question 7: Orchestrating Dependent Pipeline
Google Cloud Professional Data Engineer · Test 1 · question 7 of 10
A nightly workflow must wait for a partner's files to land in Cloud Storage, run a Dataproc job to standardize them, then run two BigQuery queries that depend on that job, and finally export the result with Dataflow. The team needs per-step retries, explicit dependencies between the steps, the ability to re-run the whole workflow for a past date, and one place to see run history.
Which service should orchestrate this workflow?
Show answer and explanation
Answer:
Cloud Composer is managed Apache Airflow, and this is the workload it was built for: a directed acyclic graph whose edges are real dependencies rather than guesses about timing, with sensors that wait for the Cloud Storage objects, operators for Dataproc, BigQuery and Dataflow, per-task retry policies, backfill of a past logical date, and a single web UI showing every run.
Option A cannot express the workflow. Scheduled queries run BigQuery SQL only (they cannot launch the Dataproc or Dataflow steps), and chaining work by clock offsets breaks the first time an upstream step runs long.
Option B can be made to work, but every capability the stem asks for would have to be built by hand: dependency tracking, retry semantics, backfill, and a consolidated run history. That is re-implementing an orchestrator.
Option D is the least managed option of the four. A cron script on a single VM has no retry or backfill semantics, no run history beyond whatever it logs, and the VM itself is a single point of failure for the entire nightly pipeline.
Reference: Cloud Composer overview