Databricks ML Professional Practice Question 22: Spark ML Class Weights
Databricks Certified Machine Learning Professional · Test 3 · question 2 of 10
A Spark ML LogisticRegression is trained on a 2-billion-row table in which 0.5% of rows are positive. The fitted model predicts the negative class for almost every record. The team requires that the *fitted coefficients themselves* account for the higher cost of missing a positive, and they will not materialise a resampled copy of a table this size.
What should they do?
Show answer and explanation
Answer:
weightCol makes each row contribute to the loss in proportion to its weight, so weighting positives by roughly the inverse of their frequency makes the 0.5% minority carry comparable total influence to the majority. The optimiser then fits coefficients that reflect that cost (which is exactly what the requirement asks for), and it does so by reading a weight column, with no copy of the data. This is the standard Spark ML approach to imbalance at scale and is supported across its classifiers.
Option A changes how the fitted model's scores are turned into labels, not what the model learns. Adjusting thresholds is a legitimate complement and would indeed produce more positive predictions, but the coefficients are identical to before, so it fails the stated requirement.
Option B does influence the fit, but by writing a physically larger training table, the materialised resample the team has ruled out. On 2 billion rows it multiplies storage and shuffle cost to achieve what a weight column achieves for free.
Option C misdirects entirely. standardization controls whether features are scaled before the penalty is applied; it addresses feature scale, not class frequency, and leaves the imbalance untouched.
Reference: pyspark.ml.classification.LogisticRegression