GCP Professional Data Engineer Practice Question 2: Late Data Allowed Lateness
Google Cloud Professional Data Engineer · Test 1 · question 2 of 10
A Dataflow streaming pipeline reads ride events from Pub/Sub and writes per-minute ride counts to BigQuery using one-minute tumbling windows over the event timestamp. Drivers frequently lose connectivity in tunnels, so about 2% of events reach Pub/Sub up to three hours after the moment they describe. Operations notices those events never appear in any count.
The dashboard must keep refreshing within a couple of minutes of the watermark, and the per-minute totals must eventually be corrected to include the delayed events. What should you change?
Show answer and explanation
Answer:
The watermark is Dataflow's estimate of when all data for a window should have arrived; anything arriving after it is late data, and with the Beam default allowed lateness of zero it is discarded, which is precisely the symptom described. Setting allowed lateness to three hours keeps each window's state alive long enough to accept the stragglers, the on-time trigger firing at the watermark keeps the dashboard fresh, and a late firing per element emits a corrected pane. Accumulating mode means each pane carries the full running total for that window rather than a delta, so the downstream consumer can simply take the latest pane for a window.
Option A destroys the very correctness the report depends on. In processing-time windows an event delayed by two hours is counted in the minute it arrived, not the minute it happened, so every per-minute total becomes wrong in both directions.
Option C changes the shape of the output without fixing anything. Hopping windows overlap and still close relative to the watermark with zero allowed lateness, so late events are dropped just the same, and three-hour windows no longer produce the per-minute granularity that is required.
Option D addresses the wrong problem. Ordering guarantees the relative order of messages for a key; it does nothing about an event that was physically unavailable for three hours. Seek replays the whole subscription from a timestamp, which reintroduces every already-processed event as a duplicate and turns a data-freshness requirement into a manual operation.
Reference: Streaming pipelines