Large-Scale AI Engineering — exercise notes

Assignment 3 #

Scheduling Multi-Node Runs with SLURM

• torch.distributed

• torchrun

• All-reduce: synchronise gradients in data-parallel training across multiple GPUs and nodes

All-Reduce and Measuring Throughput

• torch.cuda.synchronize: to wait for all gpus to finish

• Measuring time

  • Need warmup to gain significant speedup

Reduce and Broadcast: all reduce split into 2 steps

• dist.reduce(gradients, dst=ROOT_RANK, op=dist.ReduceOp.SUM)

• dist.broadcast(parameters, src=ROOT_RANK)

Send and Receive in a Ring

• send and recv are blocking so we have if can alternatively use ````isendand irecv`

Assignment 4: Data and Tensor Parallelism #

Assignment 5: Pipeline Parallelism #