MLA-C01 Practice Question 4: Imbalanced Classification Metric
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 1 · question 4 of 10
A payments company trains a binary classifier on transactions in which 0.4% of records are fraudulent. The SageMaker AI hyperparameter tuning job is currently configured with Accuracy as its objective metric, and the winning model reports 99.6% accuracy while catching almost no fraud. The business needs the model to find as many fraudulent transactions as possible, but a small manual review team means a flood of false positives is equally unacceptable.
Which objective metric should the tuning job use instead?
Show answer and explanation
Answer:
The requirement names both halves of the trade-off: catch as much fraud as possible (recall) without burying the review team in false alarms (precision). F1 is the harmonic mean of the two, so a model can score well only by doing well on both; a candidate that flags everything gets perfect recall and terrible precision, and F1 punishes it. That makes F1 the objective that actually encodes the stated business constraint.
Option A is the metric that already failed. With a 0.4% positive rate, a model that predicts "not fraud" for every transaction scores 99.6% accuracy, which is exactly what the tuner optimized toward. Moving the threshold afterwards does not change which hyperparameters the tuner selected, because the tuner never scored candidates on their minority-class behaviour.
Option B is a real improvement over accuracy and is a common choice, but ROC AUC is threshold-independent and evaluates ranking quality across every possible threshold. Because the false positive rate is computed against an enormous negative class, ROC AUC stays optimistic under heavy imbalance and says nothing about precision at the single operating point the review team will actually work at.
Option C optimizes only one side. A model tuned purely for recall is rewarded for flagging more transactions, and the degenerate solution, flag everything, scores a perfect 1.0 while violating the review-capacity constraint outright.
Reference: Metrics and validation