Databricks ML Associate Practice Question 20: Spark UDF Batch Scoring
Databricks Certified Machine Learning Associate · Test 2 · question 10 of 10
A nightly job must score a 500-million-row Delta table with a model registered in Unity Catalog. The current implementation calls toPandas() on the table and then model.predict(), and the job fails with a driver out-of-memory error. The team wants the scoring spread across the cluster's workers.
What should they do?
Show answer and explanation
Answer:
mlflow.pyfunc.spark_udf loads any logged MLflow model and returns it as a Spark UDF. Applying that UDF with withColumn keeps the data distributed: each executor loads the model once and scores its own partitions in Arrow-backed batches, so nothing passes through the driver and the job scales with the cluster. This is the standard pattern for offline scoring of a large table, and env_manager can restore the model's recorded dependencies inside the UDF when the cluster's environment differs from the training environment.
Option A treats a distribution problem as a sizing problem. toPandas() pulls all 500 million rows into driver memory no matter how large the driver is, and the single-threaded predict() that follows uses one node of the cluster.
Option C uses the wrong serving mode. Real-time endpoints are sized for low-latency individual requests; 500 million HTTP round trips a night is far slower and more expensive than scoring in place, and the data already lives next to the compute.
Option D still collects every row to the driver, so it hits the same out-of-memory failure, and a row-at-a-time Python loop discards vectorisation as well. Broadcasting matters only when work is actually running on executors.
Reference: Log, load, and register MLflow models