MLA-C01 Practice Question 15: Model Parallel Large Model
AWS Certified Machine Learning Engineer – Associate (MLA-C01) · Test 2 · question 5 of 10
An ML engineer is fine-tuning a 30-billion-parameter transformer on SageMaker AI with PyTorch DistributedDataParallel (DDP) across two instances, each with eight 80 GB GPUs. The job fails with a CUDA out-of-memory error on every GPU during the first forward pass, even with a per-GPU batch size of 1. Profiling shows that the model's weights, gradients, and optimizer states together need several times more memory than a single GPU has.
What should the engineer do to make the job train?
Show answer and explanation
Answer:
The diagnostic detail is "batch size 1 still runs out of memory". That rules out anything to do with data volume: the problem is that each GPU is being asked to hold a complete copy of the model and its training state. Data parallelism, by definition, places a full replica on every GPU and requires that at least the model and a single record fit on one device. When the model itself does not fit, AWS's guidance is model-parallel training, partitioning the model across GPUs so that each device holds only a subset. The SageMaker model parallelism library (SMP v2) does this with hybrid sharded data parallelism and tensor parallelism, and it aligns with PyTorch FSDP, so an existing PyTorch script needs relatively few changes.
Option A is still data parallelism. SMDDP optimizes the gradient all-reduce between replicas, but every GPU still holds the whole model, and adding instances only adds more replicas that each run out of memory.
Option C reduces activation memory per step for a given effective batch size, which helps when the *batch* is the problem. It does nothing for the weights, gradients, and optimizer states, which are the same size whether the micro-batch is 1 or 64.
Option D changes the number of GPUs, not the memory on each one. More replicas under DDP means more copies of a model that already does not fit on a single device.
References: Distributed training in Amazon SageMaker AI; SageMaker model parallelism library v2