AI-300 Practice Question 33: PyTorch Distribution Process Count
Microsoft Machine Learning Operations Engineer Associate (AI-300) · Test 4 · question 3 of 10
A team runs a PyTorch DistributedDataParallel training script as an Azure Machine Learning command job on a compute cluster of two nodes, each with 4 GPUs. The job succeeds, but monitoring shows that only two GPUs are busy in total, one on each node. The training script already reads the rank and world size from the environment variables. What should the team change in the job definition?
Show answer and explanation
Answer:
For a distributed PyTorch job you *"create a command, and specify the type as PyTorch and the process_count_per_instance in the distribution parameter."* The documentation adds that process_count_per_instance *"corresponds to the total number of processes you want to run for your job, and should typically equal the number of GPUs per node. If you don't specify process_count_per_instance, Azure Machine Learning launches one process per node by default"*, which explains one busy GPU per node. Setting it to 4 with two instances gives a world size of 8, and Azure Machine Learning sets MASTER_ADDR, MASTER_PORT, WORLD_SIZE, NODE_RANK, RANK and LOCAL_RANK for the script (option A).
- B is wrong because the documentation says *"you don't need to use a launcher utility like torch.distributed.launch"* to run a distributed PyTorch job. The distribution setting does the launch. - C is wrong because the cluster has two nodes. Asking for eight instances requires eight nodes, and with the default of one process per node it would still leave three GPUs on each node idle. - D is wrong because a value of 1 keeps one process per node, which is the behavior the team is already seeing, and MPI is a different launch mechanism than the PyTorch distribution.
Reference: Distributed GPU training guide (SDK v2) - Azure Machine Learning