DEA-C01 Practice Question 51: Glue Streaming ETL Checkpoints
AWS Certified Data Engineer – Associate (DEA-C01) · Test 6 · question 1 of 10
An AWS Glue streaming ETL job reads from an Amazon Kinesis data stream, enriches each record, and writes Parquet to Amazon S3. The job was stopped for a deployment and restarted an hour later. Operations want to be sure that, after every restart, the job resumes from where it stopped rather than reprocessing or skipping records, and they want the aggregation window widened from the default.
Which statement correctly describes how to meet these requirements?
Show answer and explanation
Answer:
AWS Glue streaming ETL jobs run on Spark Structured Streaming and "use checkpoints rather than job bookmarks to track the data that has been read." The documentation is explicit about restarts: "AWS Glue streaming ETL jobs use checkpoints to keep track of the data that has been read. Therefore, a stopped and restarted job picks up where it left off in the stream. If you want to reprocess data, you can delete the checkpoint folder referenced in the script." The window is configurable as well: "by default, AWS Glue processes and writes out data in 100-second windows… You can modify this window size to increase timeliness or aggregation accuracy."
Option A names the wrong mechanism. The same page states that for streaming jobs "job bookmarks aren't supported"; bookmarks are for batch jobs reading S3 or JDBC sources.
Option C would guarantee the opposite of what operations want. TRIM_HORIZON starts at the oldest record still in the stream, so every restart would reprocess everything within the retention period, mass duplication in the curated table.
Option D asks the team to build the checkpointing the service already provides, and to build it less reliably: a hand-rolled sequence-number table has no transactional relationship with the records actually written to S3, so a crash between write and update produces duplicates or gaps.
Reference: Streaming ETL jobs in AWS Glue