Databricks ML Associate Practice Question 16: randomSplit Reproducibility
Databricks Certified Machine Learning Associate · Test 2 · question 6 of 10
A notebook builds a training DataFrame from a chain of joins and filters over Delta tables, then splits it:
train_df, test_df = df.randomSplit([0.8, 0.2], seed=42)
The seed never changes, but re-running the notebook produces a different number of training rows each time, and a few records show up in both splits. The upstream Delta tables are not being modified.
What is the most likely cause, and what should the data scientist do?
Show answer and explanation
Answer:
randomSplit applies a sampling filter per output DataFrame, and each one re-evaluates the source plan. The seed fixes the random number sequence, but that sequence is applied *within* partitions: the split is only deterministic if every re-evaluation produces the same rows in the same order in the same partitions. A plan built from joins and filters offers no such guarantee, so rows can land in different partitions on different evaluations and be sampled into both splits or neither. Materialising the DataFrame first (df.cache(), or a repartition on a column, or writing it out and reading it back) fixes the layout so both sampling passes see identical input.
Option A misattributes the effect to the seed's magnitude. The seed is a starting point for a pseudorandom sequence, not a partition count, and no value of it stabilises a changing plan.
Option B is wrong on the design point and does not help. randomSplit is reproducible over a stable input, and the two sample calls it proposes would re-evaluate the plan exactly the same way, reproducing the overlap.
Option C would serialise every shuffle in the job for a severe performance cost, and a single partition still does not guarantee a stable row order across re-evaluations.
Reference: Behavior of the randomSplit method (Databricks Knowledge Base)