Databricks ML Professional Practice Question 3: Spark ML Pipeline Leakage
Databricks Certified Machine Learning Professional · Test 1 · question 3 of 10
An engineer builds a Spark ML Pipeline with three stages: an Imputer that fills missing values with column means, a StandardScaler, and a LogisticRegression estimator. The dataset has 500 million rows. The engineer needs an honest estimate of how the model will perform on data it has never seen.
What is the correct workflow?
Show answer and explanation
Answer:
Calling fit on a Spark ML Pipeline fits every estimator stage in order: the Imputer learns the column means, the StandardScaler learns the mean and standard deviation, and LogisticRegression learns the coefficients. The result is a PipelineModel. That model applies the same learned statistics when it transforms any DataFrame. If you fit only on the training split and transform the held-out split, no test-set statistic reaches the model. That is what makes the evaluation honest (for example, AUC from BinaryClassificationEvaluator).
Option A leaks test data. The imputation means and scaling statistics are calculated with the test rows included, so the score is optimistic.
Option B refits every stage on the test data. The coefficients you evaluate are then learned from the test set, so the score measures memorisation, not generalisation.
Option C has the same leakage as option A, just done by hand. The preprocessing statistics still come from the full dataset. It also splits preprocessing away from the model, so they can no longer be saved and served together as one PipelineModel.
Reference: ML Pipelines (Apache Spark documentation)