AI-300 Practice Question 20: Preference Alignment Method
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 2 · question 10 of 10
A model that has already been supervised fine-tuned on a company's support corpus answers accurately but far too bluntly. Reviewers can reliably say which of two candidate replies has the better tone, but they say they cannot write the ideal reply themselves, and no automated grader exists that can score tone. Which customization technique should you apply next to the fine-tuned model?
Show answer and explanation
Answer:
Direct preference optimization (DPO) is "an alignment technique for large language models, designed to adjust model weights based on human preferences". Unlike RLHF it "doesn't require fitting a reward model and uses binary preferences for training", which is precisely what these reviewers can supply. Microsoft even documents the sequence: "first use supervised fine-tuning (SFT) to create a customized model optimized for your use case, then use DPO to align the responses to your specific preferences" (option B).
- A is wrong because supervised fine-tuning learns from prompt/completion pairs; you have to supply the ideal answer, which the reviewers said they cannot write. Re-running it over the same blunt corpus at a lower learning rate only re-learns the same style more gently. - C is wrong because reinforcement fine-tuning optimizes behavior through iterative feedback from graders, and the stem states there is no grader that can score tone. Without a reward signal there is nothing for RFT to optimize, and the technique is aimed at complex or dynamic reasoning behavior rather than style alignment. - D is wrong because rewriting the corpus in the desired tone is the SFT path again, and it needs exactly the ideal answers that are unavailable. It also discards the customized model that the documented SFT-then-DPO sequence is designed to build on.
Reference: Microsoft Foundry fine-tuning considerations