GCP Professional Data Engineer Practice Question 5: Ephemeral Dataproc Clusters
Google Cloud Professional Data Engineer · Test 1 · question 5 of 10
A team is migrating a set of Apache Spark batch jobs off an on-premises Hadoop cluster. The jobs run for about four hours each night and read and write data currently held in HDFS. The migration budget does not allow the Spark code to be rewritten, and the first cut on Google Cloud (one always-on Dataproc cluster sized for the nightly peak, with data in cluster HDFS) costs far more than expected.
What is the most cost-effective way to run these jobs?
Show answer and explanation
Answer:
Separating storage from compute is the core Dataproc cost pattern. (Google's docs now also call Dataproc on Compute Engine *Managed Service for Apache Spark*; the exam guide still says Dataproc.) With data in Cloud Storage, Spark reads it directly through the Cloud Storage connector using gs:// paths instead of hdfs:// (a one-line configuration change, not a rewrite), and the data survives the cluster. That in turn makes the cluster disposable: create it for the nightly run, let the job start as soon as the workers come up, and delete it (with a scheduled deletion policy or a workflow template) so the remaining 20 hours a day cost nothing.
Option A keeps paying for the idle window. Autoscaling can remove secondary workers, but the primary workers and their persistent disks must stay up, and HDFS pins the data to those workers, so the cluster can never scale to zero.
Option C violates the stated constraint. Porting Spark to Beam is a rewrite of every job, and these are bounded nightly batch jobs, not streaming ones.
Option D is a rewrite too, and a larger one; every transformation would have to be re-expressed in SQL, which the budget explicitly rules out.
Reference: Cloud Storage connector