Databricks ML Associate Practice Question 7: Cross Validation Leakage
Databricks Certified Machine Learning Associate · Test 1 · question 7 of 10
A data scientist target-encodes a high-cardinality merchant_id column: each merchant is replaced with its mean label value, computed over the entire dataset. They then run 5-fold cross-validation on a gradient-boosted classifier and get a mean ROC AUC of 0.94. On a holdout set collected the following month, the same model scores 0.81.
What is the most likely cause, and what should they change?
Show answer and explanation
Answer:
Target encoding computes a statistic from the label. Because it was fit on the whole dataset before cross-validation, each validation fold's encoded values already include that fold's own labels. The model is scored partly on information it could never have at prediction time, so the cross-validation AUC is inflated. Truly unseen data exposes the gap. The fix is to make every learned preprocessing step part of the model: put the encoder and the classifier in one pipeline and pass the pipeline to the cross-validator, so the encoder is fit on each fold's training portion only. The same rule applies to imputers and scalers fit on data.
Option A treats the gap as a variance problem. More folds give a slightly different estimate, but each fold still contains leaked labels, so the inflation stays.
Option B misreads the evidence. A model that scores much better in validation than on new data is not underfitting, and adding complexity would make it worse.
Option C changes the reported number without fixing the flawed evaluation. Accuracy is also a weaker choice than AUC for most imbalanced classification problems.
Reference: Common pitfalls and recommended practices: data leakage (scikit-learn)