Databricks ML Professional Practice Question 13: Cache Training Dataframe
Databricks Certified Machine Learning Professional · Test 2 · question 3 of 10
A Spark ML CrossValidator tunes a classifier over a training DataFrame built from four joins and several filters across Delta tables. The Spark UI shows the same joins and file scans repeating on every model fit, and the tuning job takes hours. The assembled DataFrame fits comfortably in the cluster's memory.
What is the most effective change?
Show answer and explanation
Answer:
Spark DataFrames are lazy: the DataFrame is a plan, not data, and every action re-executes that plan from the source files. Cross-validation is an iterative algorithm that reads the training data once per fold per parameter combination, so the four joins are being recomputed dozens of times. Calling cache() (or persist()) and then forcing materialisation with an action stores the computed result in executor memory, and every subsequent fit reads from that instead of rebuilding it. Caching the input to an iterative algorithm is the standard remedy, and the DataFrame fitting in memory is what makes it safe here.
Option A confuses planning with execution. Catalyst already optimises the plan cheaply on the driver; the cost is the repeated distributed execution on the workers, which driver memory does not touch.
Option C persists the wrong side of the problem. The models are small outputs; the expense is regenerating the input, and writing models to Delta adds I/O without eliminating a single join.
Option D abandons the tuning strategy to save I/O, and CrossValidator requires at least two folds. Even granting the intent, one split still re-evaluates the plan for each parameter combination.
Reference: RDD persistence (Apache Spark)