AI-300 Practice Question 43: Error Analysis Find Failing Cohorts
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 5 · question 3 of 10
A loan-approval classifier has 92 percent overall accuracy, but support staff report that it fails noticeably more often for some kinds of applicants. The team does not know which groups are affected, and it has not defined any sensitive attribute in advance. Before deploying the model, they want the Responsible AI dashboard to reveal automatically where the errors are concentrated. Which component should they use first?
Show answer and explanation
Answer:
The Responsible AI dashboard's error analysis component *"helps you get a deeper understanding of model failure distribution and quickly identify erroneous cohorts (subgroups) of data."* It is meant for the situation in this scenario, where overall accuracy hides poor performance in subgroups the team has not yet identified (option D).
- A is wrong because counterfactual what-if analysis shows how perturbing features changes a prediction. It helps understand individual cases, not find which cohorts fail. - B is wrong because the fairness assessment *"defines groups in terms of sensitive attributes like sex, race, and age"* and then evaluates how predictions affect those groups. It compares the groups defined by sensitive attributes, so it is not the tool for discovering unknown failing cohorts. - C is wrong because the model overview aggregates model assessment metrics in a high-level view. It reports metrics for the dataset and for cohorts that are already defined, and an overall accuracy alone is exactly what hides the problem.