DEA-C01 Practice Question 72: Glue Auto Scaling Variable Workload
AWS Certified Data Engineer – Associate (DEA-C01) · Test 8 · question 2 of 10
A nightly AWS Glue 4.0 Spark job runs with a fixed 60 G.2X workers. Input volume varies a lot from night to night, and CloudWatch shows that many stages use only a few executors while most workers sit idle. The team wants to reduce cost without hand-tuning the worker count for every run. What should they do?
Show answer and explanation
Answer:
With Auto Scaling enabled, *"AWS Glue automatically adds and removes workers from the cluster depending on the parallelism at each stage or microbatch of the job run,"* and *"with the given maximum number of workers, AWS Glue will choose the right size resources for the workload."* AWS lists *"Spark stages running with only a few executors due to over-provisioning"* as a common case where it helps with cost and utilization. It is available for Glue 3.0 or later (option A).
- B is wrong because a Python shell job runs on a single small node without Spark. It cannot process a large, distributed dataset in the same way. - C is wrong because the Standard worker type is not supported by Auto Scaling, and it does not adapt the number of workers to the load. - D is wrong because adding workers to a job that already leaves most of them idle increases cost and does not improve utilization.
Reference: Using auto scaling for AWS Glue