DEA-C01 Practice Question 57: Glue Executor OOM JDBC Fetch
AWS Certified Data Engineer – Associate (DEA-C01) · Test 6 · question 7 of 10
An AWS Glue Spark job reads a 34-million-row MySQL table with sparkSession.read.jdbc(url, tableName, connectionProperties) and writes it to Amazon S3 as Parquet. The job is allocated 10 G.1X workers but fails after a few minutes. The CloudWatch job metrics show average executor memory climbing past 90 percent within a minute and only one executor ever running, and the logs contain Container killed by YARN for exceeding memory limits.
What should the data engineer change?
Show answer and explanation
Answer:
The AWS Glue debugging guide describes this scenario precisely: "The executor ran out of memory while reading the JDBC table because the default configuration for the Spark JDBC fetch size is zero. This means that the JDBC driver on the Spark executor tries to fetch the 34 million rows from the database together and cache them, even though Spark streams through the rows one at a time." The fix is to bound the fetch: "By default, dynamic frames use a fetch size of 1,000 rows that is a typically sufficient value. As a result, the executor does not take more than 7 percent of its total memory", or set the Spark fetchsize property directly. The clue that rules out a capacity problem is in the stem: only one executor ever runs, because "the JDBC data source reads are not parallelized by default", so the memory pressure sits on a single executor regardless of fleet size.
Option B adds executors that have nothing to do. Without a partitioning column and multiple connections, a JDBC read is single-threaded; nine of the ten workers already sit idle, and forty would mean thirty-nine idle.
Option C solves the other OOM scenario in the same guide, a *driver* OOM caused by listing a very large number of small Amazon S3 files. groupFiles is an S3 read option and has no effect on a JDBC source.
Option D is throwing memory at an unbounded allocation. A bigger heap raises the row count the executor can cache before dying, but the driver still tries to materialize the whole result set, so the job fails again as soon as the table grows, at eight times the worker cost.