AIF-C01 Practice Question 78: Annotator Introduced Bias
AWS Certified AI Practitioner (AIF-C01) · Test 8 · question 8 of 10
A content-moderation training set was labelled by a pool of 40 contractors. The dataset represents every demographic group in the expected user population in proportion, and overall model accuracy is high, yet the deployed classifier removes posts written in one dialect at a far higher rate than others. An audit finds that a subset of annotators consistently marked that dialect as "offensive" where other annotators reviewing the same posts did not. Which action addresses the actual source of the disparity?
Show answer and explanation
Answer:
The bias here entered through the labels, not the sample. The AWS Responsible AI Lens best practice on label quality tells you to *implement quality control mechanisms for human annotators including training processes, unwanted bias identification, and inter-rater agreement measurements*, and to *hunt for sources of human bias in your labeling process by looking at whether certain annotators consistently label some groups differently than others*. It also calls for *processes for fixing labels that don't meet your quality standards to block similar problems in future annotation work*, which is why correcting the affected labels is part of the answer, not just tightening the process (option D).
- A is the trap, and the stem rules it out: representation is already proportional. More correctly-sampled posts routed through the same annotation process simply acquire the same skewed labels, teaching the model the identical bias with greater confidence. - B is wrong because raising the threshold suppresses removals across every dialect equally. The *ratio* between groups, which is the disparity being complained about, is unchanged, and genuine violations now survive too. - C is wrong as a fix, though it is defensible as a temporary mitigation. It treats the symptom at serving time forever, scales poorly, and leaves the corrupted labels sitting in the training set to contaminate every future version of the model.
Reference: RAIDP02-BP03 Validate the quality of human and generated labels and features in your dataset