GCP Professional Data Engineer Practice Question 10: Updating Streaming Pipeline
Google Cloud Professional Data Engineer · Test 1 · question 10 of 10
A 24/7 Dataflow streaming pipeline reads from Pub/Sub, aggregates into 30-minute session windows, and writes to BigQuery. A bug fix must be deployed to one of the transforms during business hours. The team cannot lose the records already buffered in the running job, the partially accumulated session state must survive the deployment, and the fix renames one of the existing transforms.
How should you deploy the new pipeline code?
Show answer and explanation
Answer:
Updating a streaming job launches a replacement job that takes over the original job's name and carries across its intermediate state and in-flight buffered records, which is the only option here that preserves half-accumulated session windows. Before switching over, Dataflow runs a compatibility check to confirm the state from the prior job's steps can be transferred and that the PCollection coders match; if the check fails, the original job simply keeps running. Because state is matched by step name, a renamed transform requires an explicit transform mapping such as {"oldName":"newName"}, exactly as the stem describes.
Option A throws away the data the stem says must be kept: cancelling stops the job immediately and in-flight buffered records and window state are discarded.
Option B is the safe way to *retire* a pipeline, not to upgrade one. Draining stops pulling new input and flushes in-flight windows early, emitting partial session aggregates, and the replacement job then starts with no state at all, so the sessions are split across two jobs.
Option D is not a deployment mechanism. A Dataflow job runs the pipeline graph it was submitted with; there is no way to swap code under a running job by resizing its worker pool, and dropping to zero workers would only stall processing while the backlog grows.
Reference: Update an existing pipeline