Databricks ML Associate Practice Question 8: Imbalanced Metric Choice
Databricks Certified Machine Learning Associate · Test 1 · question 8 of 10
In a payments dataset, 0.4% of transactions are fraudulent. Flagged transactions go to a small team of analysts, so the model must catch as much fraud as possible without flooding the team with false alarms. A colleague proposes selecting the model by accuracy.
Which metric should the team use to select the model instead?
Show answer and explanation
Answer:
With 0.4% positives, a model that predicts "not fraud" for every transaction scores 99.6% accuracy while catching nothing. The goal names two competing costs: missed fraud (recall) and wasted analyst time (precision). The F1 score is the harmonic mean of precision and recall for the positive class, so it is high only when the model does well on both. That makes it a good selection metric here. f1 is also the default primary metric for AutoML classification.
Option A is exactly the trap the stem describes. On heavily imbalanced data, accuracy is dominated by the majority class and hides a useless model.
Option B is a regression metric. It measures explained variance of a continuous target and does not apply to a binary classifier.
Option D is also a regression metric. Applied to 0/1 labels it reduces to the error rate, which is just 1 minus accuracy and has the same flaw.
Reference: Metrics and scoring: precision, recall and F-measures (scikit-learn)