DEA-C01 Practice Question 3: Glue Job Bookmarks Incremental
AWS Certified Data Engineer – Associate (DEA-C01) · Test 1 · question 3 of 10
A nightly AWS Glue Spark ETL job reads newline-delimited JSON files from s3://raw/events/ and appends the transformed rows to a curated Parquet table in Amazon S3. New files arrive in the raw prefix throughout the day and are never modified after they are written. Analysts report that row counts in the curated table roughly double every night, and inspection confirms that each run is reading every file in the raw prefix, not just the ones written since the previous run.
Which change resolves the duplication with the least effort?
Show answer and explanation
Answer:
A job bookmark is persisted state that AWS Glue keeps per job to "prevent the reprocessing of old data". For an Amazon S3 source it records the last-modified time of the objects consumed, and on the next run it filters the listing down to objects newer than the bookmark. Because the raw files are write-once, that filter is exactly right. Bookmarks are off by default; the Disable setting means "the job always processes the entire dataset", which is the behaviour observed, so enabling them is a one-property change with no code rewrite. State is committed when the script calls job.commit(), and it can be reset later if a full reprocess is ever needed.
Option B misunderstands what a crawler does. A crawler populates and updates table and partition metadata in the Data Catalog; it never removes already-cataloged partitions or signals to a job which data is new. The job would still read the full table.
Option C changes the architecture for no benefit. AWS Glue streaming jobs consume from streaming sources such as Amazon Kinesis Data Streams and Apache Kafka, not from an Amazon S3 prefix, and a continuously running job costs far more than a nightly batch.
Option D destroys the source data to work around a state-tracking problem. It also does not fix the bug: files written in the last 24 hours still exist at the next run, so anything ingested twice within a day is still duplicated, and any need to backfill or reprocess is now impossible.
Reference: Tracking processed data using job bookmarks