Databricks ML Professional Practice Question 28: Pipeline Expectations Quarantine
Databricks Certified Machine Learning Professional · Test 3 · question 8 of 10
A declarative pipeline computes features from an upstream feed that occasionally emits a negative amount or a null customer_id. These rows must not reach the feature table, the pipeline must keep processing the good rows in the same batch, and the team needs the number of rejected records visible per run so they can raise it with the upstream owner.
Which approach meets all three requirements?
Show answer and explanation
Answer:
The three requirements map onto the three expectation actions, and only the drop action satisfies all of them. expect_or_drop removes records that violate the constraint before they are written, the update proceeds with the remaining rows, and the count of dropped records is logged alongside the dataset's other metrics, so the rejection rate is queryable per run and can be charted or alerted on. Declaring the rule as an expectation rather than as ad-hoc code is what makes the observability automatic.
Option A achieves the filtering but is silent. A filter leaves no record of how many rows it removed, so the team has no number to take to the upstream owner and no way to notice the rate changing.
Option B satisfies the first requirement and breaks the second. The fail action halts the update when a violation appears, rolling back the transaction, so a single malformed row stops the good rows from being processed. It also produces no metrics, because the update stops on detection.
Option D publishes the bad rows first, which is the outcome the first requirement forbids. Anything reading the feature table between the write and the cleanup sees invalid features.