DEA-C01 Practice Question 13: Step Functions Glue Sync
AWS Certified Data Engineer – Associate (DEA-C01) · Test 2 · question 3 of 10
A nightly pipeline has two steps: an AWS Glue ETL job writes curated Parquet files, then an Amazon Redshift COPY loads them. Today, an AWS Lambda function calls StartJobRun and immediately triggers the COPY step. On several nights Redshift loaded partial data because the Glue job was still running. The Glue job also fails now and then with transient errors that succeed on a rerun.
The team wants the load to start only after the Glue job has actually succeeded. A failed job should be retried up to three times with increasing delays, and if it still fails, an Amazon SNS notification should be sent and the load skipped.
Which solution meets these requirements with the least custom code?
Show answer and explanation
Answer:
The optimized AWS Glue integration in Step Functions supports the Run a Job (`.sync`) pattern. With arn:aws:states:::glue:startJobRun.sync, the Task state starts the job and then waits for the job run to finish before the workflow moves on. If the run fails, the Task state fails too. That fixes the partial-load problem without any polling code. The built-in error handling covers the rest. A retrier's MaxAttempts sets the number of retries, and BackoffRate multiplies the interval between them. When retries run out, a catcher sends the workflow to a fallback state, here an SNS publish. The load state is never reached, so it is skipped.
Option C is the trap. The request-response pattern returns as soon as StartJobRun returns a JobRunId. That is the same race the Lambda function has today, so the COPY still starts while the job is running. Its Retry would only catch errors from the StartJobRun API call itself, not failures of the job run.
Option B replaces the race with a guess. A job that runs long one night, or fails, still leads to a COPY of incomplete data. Nothing retries the job or sends an alert.
Option D only adds a queue. The message is published right after the job *starts*, so the second function runs the COPY just as early as today. Adding the missing wait, retry, and notification logic would mean writing custom code.
References: Start an AWS Glue job with Step Functions; Handling errors in Step Functions workflows