Databricks ML Professional Practice Question 21: Spark ML vs Single Node
Databricks Certified Machine Learning Professional · Test 3 · question 1 of 10
A training dataset is 8 GB and the cluster's worker nodes have 128 GB of memory each. The team is comfortable with scikit-learn and is deciding whether to rewrite their training code in Spark ML because "the platform is Spark".
What should they do?
Show answer and explanation
Answer:
Spark ML earns its cost when a dataset is too large for one machine: it pays for shuffles, serialization and coordination in exchange for splitting work across the cluster. At 8 GB against 128 GB of node memory there is nothing to split, so a single-node scikit-learn fit is typically faster as well as simpler, and it keeps access to scikit-learn's wider algorithm and ecosystem support. Databricks supports this directly through single-node clusters. The distributed alternatives worth reaching for at this scale are parallel *tuning* or per-group training, not distributing a single fit.
Option A asserts a requirement that does not exist. Databricks clusters run ordinary Python on the driver, and single-node cluster types exist precisely for this workload.
Option C is false. MLflow tracks and logs scikit-learn models as a first-class flavour, and autologging supports it.
Option D is simply wrong about Spark ML, which provides logistic regression, decision trees, random forests, gradient-boosted trees and more. The reason to prefer scikit-learn here is scale, not capability.
Reference: Classification and regression (Spark ML)