Databricks ML Associate Practice Question 19: Permutation Importance
Databricks Certified Machine Learning Associate · Test 2 · question 9 of 10
A random forest is trained on customer data. Ranking the model's feature_importances_ puts account_number, a near-unique numeric identifier, at the top. A stakeholder asks whether account numbers really drive the prediction.
How should the data scientist check this?
Show answer and explanation
Answer:
Impurity-based importance sums how much each feature reduced impurity over all splits, measured on the training data. A high-cardinality numeric column offers an enormous number of candidate split points, so it can carve the training set into pure leaves and accumulate a large score while carrying no generalisable signal, exactly what an identifier does. Permutation importance instead shuffles one column in a held-out set and measures how much the score drops. A feature the model does not genuinely rely on produces almost no drop, which settles the stakeholder's question.
Option A trusts the statistic whose known failure mode is on display. Being computed from the fitted trees is precisely why it reflects training-set behaviour rather than predictive value.
Option B changes the model instead of testing the claim. Refitting on a reduced feature set produces a new ranking that inherits the same bias, so the identifier can simply rank first again.
Option C misidentifies the cause. The bias is structural (it comes from cardinality creating more split opportunities), and averaging over more trees does not remove it.