Databricks ML Professional Practice Question 2: Optuna Distributed Tuning
Databricks Certified Machine Learning Professional · Test 1 · question 2 of 10
A team is moving a tuning notebook from Hyperopt SparkTrials to a cluster running Databricks Runtime 17.3 LTS ML. The new setup must meet all of these requirements:
- Trials run in parallel on the Spark executors, not only on the driver. - All trials share one study, so the sampler learns from every completed trial. - Every trial is tracked in an MLflow experiment. - Unpromising trials can be pruned early. - No external database is provisioned to store the study.
Which approach satisfies every requirement?
Show answer and explanation
Answer:
Databricks documents this pattern for distributing Optuna. MlflowStorage (from mlflow.optuna.storage) lets Optuna use the MLflow Tracking Server as its study storage, so no separate database is needed. MlflowSparkStudy (from mlflow.pyspark.optuna.study) runs trials in parallel on PySpark executors. n_jobs sets how many executor tasks run at once. All trials write to the same MLflow-backed study, so the sampler (TPE by default) learns from every result, and the default MedianPruner stops unpromising trials early. MLflow 3 is preinstalled on Databricks Runtime 17.0 ML and above.
Option B fails at the first step. Hyperopt is no longer maintained, and Databricks removed it from Databricks Runtime ML after 16.4 LTS ML. Databricks recommends migrating to Optuna or Ray Tune.
Option C uses a single shared study, but n_jobs in plain study.optimize only runs threads in the driver process. No trial runs on the executors, so it fails the first requirement.
Option D does put work on the executors, but it creates eight separate in-memory studies. They share no history, so no sampler learns from all trials. The results also end up scattered instead of in one study.
Reference: Hyperparameter tuning with Optuna