MLA-C01 Practice Question 3: Target Leakage Training Data
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 1 · question 3 of 10
A churn model trained on a tabular customer dataset scores 0.98 AUC on a held-out validation split that was drawn randomly from the same dataset. After deployment, the endpoint's measured AUC against real outcomes is about 0.61. Reviewing the feature list, an ML engineer notices a column named retention_offer_sent, which the retention team populates only after an analyst has already flagged the account as likely to churn. The column is blank for almost every live inference request.
What is the most likely cause, and what is the correct remediation?
Show answer and explanation
Answer:
Target leakage is data in the training set that is strongly correlated with the label but is not available in the real world at prediction time. retention_offer_sent is written only after a human has decided the account is churning, so in the training data it is effectively a copy of the answer. Both the training split and the validation split contain it, which is why offline scores are spectacular and honest, and why they collapse the moment the model sees live records where the column is empty. The fix is to remove the feature (or reconstruct it from a point in time strictly before the prediction moment) and retrain. SageMaker's target leakage analysis flags exactly this pattern: a feature whose standalone predictive score approaches 1.
Option B does not match the evidence. Overfitting shows up as a gap between training and validation scores; here the validation score is also inflated, because the leaked feature is in both splits. Regularization would not remove a feature that is genuinely predictive of the label in the data it was given.
Option C misreads the symptom. Imbalance distorts metrics like accuracy, but the reported metric is AUC, and oversampling would not explain a validation score that is 0.37 higher than production.
Option D describes a real failure mode, but drift produces gradual degradation from a realistic starting point, not a model that is near-perfect offline and near-random from day one. Monitoring would report the problem without fixing it.
Reference: Analyze and Visualize (Target Leakage)