Databricks ML Associate Practice Question 26: Post Outcome Feature Leakage
Databricks Certified Machine Learning Associate · Test 3 · question 6 of 10
A model predicts, at the moment a support ticket is opened, whether that ticket will breach its SLA. It is trained on a table of closed tickets whose columns include priority, queue, customer_tier, reopen_count and minutes_to_first_response. Offline evaluation on a held-out set of closed tickets gives 0.96 AUC. In production, scoring tickets at the instant they are created, the model performs barely better than guessing.
What is the most likely cause, and what should the team do?
Show answer and explanation
Answer:
minutes_to_first_response and reopen_count do not exist when a ticket is created; they are filled in as the ticket is handled, and they correlate strongly with whether it eventually breached. Training on a table of *closed* tickets hands the model these after-the-fact columns, so it learns a relationship it can never use, and the held-out set of closed tickets has the same columns filled in, so offline evaluation never exposes the problem. In production those fields are null or zero at scoring time and the signal disappears. The fix is to define each feature by *when its value is known*: reconstruct the training rows as of the moment of ticket creation and drop anything that is not available then. This is train/serve skew, and no amount of tuning detects it, because the offline setup is the thing that is wrong.
Option B misreads the symptom. Overfitting shows up as a gap between training and held-out performance; here the held-out score is excellent and only production is poor, which points at a difference between the offline and online feature values rather than at model capacity.
Option C scales up the same flawed table. More closed tickets bring more rows with the same post-outcome columns populated, so the offline score stays high and production stays unchanged.
Option D changes how the model is scored without changing what it was given. The threshold and metric are reasonable things to revisit, but a model relying on features that are absent at inference time performs poorly at every operating point.
Reference: Rules of Machine Learning, Rule #29: training-serving skew (Google)