DEA-C01 Practice Question 22: Glue Group Small Files

The symptom set (millions of sub-second tasks, low CPU, driver pressure during enumeration) is the classic small-files problem. Spark creates roughly one task per input file, so per-task scheduling overhead dominates and the actual work is trivial. AWS Glue's file grouping exists for exactly this: setting groupFiles to inPartition makes "each ETL task read a group of input files into a single in-memory partition", and groupSize sets the target size of each group in bytes. Millions of 20 KB files collapse into a few thousand sensibly sized partitions, task count drops by orders of magnitude, and the workers finally do real work. The option is supported for csv, ion, grokLog, json, and xml; JSON here qualifies. (Glue enables grouping automatically above 50,000 input files, but setting groupSize explicitly is what tunes it to the cluster.)

Option A adds capacity to a job that is not capacity-bound. CPU utilization is already low, so more and bigger executors mostly add more idle cores while the per-task overhead and the driver-side enumeration stay exactly the same, a more expensive job with a similar runtime.

Option B addresses a different problem. Bookmarks prevent reprocessing of data from *previous* runs. The 2 million files are the current day's arrivals; they are all new, so a bookmark excludes none of them.

Option D tunes the write side, and in the wrong direction. The bottleneck is reading millions of input splits, not writing. Forcing a single output partition funnels the entire result through one task, removing write parallelism and usually making the job slower still.

Reference: Reading input files in larger groups