MLA-C01 Practice Question 42: Oversampling Split Order
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 5 · question 2 of 10
A bank is training a binary classifier to flag money-laundering transactions, which make up 0.8% of its 400,000 labeled examples. A data scientist applied SMOTE to the full dataset to bring the positive class to 30%, then randomly split the result 80/20 into training and validation sets. The model scored 0.97 recall on validation. In a shadow deployment against live transactions, recall dropped to 0.41, and no data drift was found between training data and live traffic.
What is the most likely cause, and how should the process be corrected?
Show answer and explanation
Answer:
SMOTE creates each synthetic minority sample by interpolating between real neighboring minority samples. Applying it *before* the split means many validation positives are near-copies of training positives, or share parents with them, so the model is effectively tested on examples it has already seen. The validation set is also 30% positive when reality is 0.8%. Both inflate the validation score, and production exposes the gap. The fix is to split the original data first, resample only the training split, and evaluate on validation data that keeps the real class distribution (option A).
- B is wrong because more oversampling makes the leak worse. The problem is where the resampling happened, not how much. - C is wrong because stratifying *after* SMOTE still spreads synthetic near-duplicates across both sets and still validates at an unrealistic 30% positive rate. Stratification is useful, but on the original data, before any resampling. - D is wrong because a model that scores 0.97 recall on validation is not underfitting. Adding capacity makes it more likely to memorize the synthetic points, which widens the gap.
Reference: Transform Data - Balance Data (Amazon SageMaker Data Wrangler)