GCP Professional ML Engineer Practice Question 15: Reduction Server Gradient Bottleneck
Google Cloud Professional Machine Learning Engineer · Test 2 · question 5 of 10
A PyTorch model is trained with DistributedDataParallel across five nodes, each with eight GPUs, as an Agent Platform custom training job. Profiling shows the GPUs idle for a large share of each step while NCCL all-reduce exchanges gradients between nodes, and wall-clock time per epoch barely improved when the fourth and fifth nodes were added. Accuracy is acceptable and the team does not want to change the model or the effective batch size.
What should you do to increase training throughput?
Show answer and explanation
Answer:
Reduction Server is an all-reduce algorithm Agent Platform provides as a container image for a worker pool, aimed exactly at the case where communicating gradients between nodes contributes significant latency in multi-node GPU training. It applies when the workers use GPUs and the code is configured for multi-host data-parallel training with NCCL all-reduce, which DistributedDataParallel is. You specify the reductionserver image in workerPoolSpecs[2], use no GPUs on those nodes, and size the pool so its total network bandwidth matches or exceeds that of the primary and worker pools, the documented configuration rule (option C).
- A is wrong because it caps the job at whatever one machine type offers and throws away the scale-out the team just paid for. It is a retreat from distributed training, not a fix for it. - B is wrong because a parameter server centralizes gradient traffic on a few replicas, which usually makes a bandwidth-bound synchronous GPU job worse, and it means rewriting the training code away from DistributedDataParallel. - D is wrong because accumulating gradients across micro-batches changes the effective batch size, which the team explicitly ruled out, and it reduces all-reduce frequency rather than making each all-reduce faster.
Reference: Distributed training