MLA-C01 Practice Question 24: k Fold Cross Validation
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 3 · question 4 of 10
A clinical dataset has only 3,000 labeled rows and cannot be expanded. An ML engineer holds out a random 20% of the rows for validation. Re-running the same training code with a different random split produces validation AUC anywhere between 0.71 and 0.84, so the team cannot tell whether one candidate model is genuinely better than another.
Which change gives a more reliable estimate of model performance?
Show answer and explanation
Answer:
With few rows, a single 20% holdout is a small, noisy sample, so the score swings with whichever rows happen to land in it. k-fold cross-validation splits the data into k equally sized folds, trains on k-1 of them and validates on the remaining one, repeats that k times so every row is validated exactly once, then averages the k scores. The averaged metric has far less variance and uses all the data for both training and validation, which is why SageMaker Autopilot applies k-fold cross-validation automatically on small datasets (option D).
- A is wrong because it trades one problem for another: the validation set gets steadier, but the model now trains on only 1,500 rows, so the measured performance no longer reflects the model the team would ship. - B is wrong because a training metric measures memorization, not generalization. On 3,000 rows a flexible model can score near-perfectly on data it has already seen. - C is wrong because fixing the seed makes the number reproducible, not accurate. It locks in one arbitrary split and hides the variance rather than reducing it.
Reference: Metrics and validation - Cross-validation in Autopilot