Large-Scale AI Engineering — lecture notes

see full notes in the pdf. I will at some point update this with the latest notes.

Clariden is the Swiss AI partition of the Alps supercomputer at CSCS

Super computer: 4-6 years also called HPC (high performance computing)

Used for tasks that require large compute or memory
Huge energy consumption
HPC Parts:
- Compute nodes, interconnect network, storage system, cooling system, power supply system, software stack
Networking: Latency, Bandwidth, Radix (Ports)
Storage: Petabytes of data, high throughput and reliability, parallel file systems, Lustre (open source), GPFS (IBM)
Providers: Hyperscalers, Neoclouds (AI workload) SLURM Workload Manager
Open source cluster management and job scheduling system for linux clusters

Supercomputer lecture by CSCS

Communication overhead

Weak scaling law: increase batch size when increasing nodes

Model Flops utilization: 30-50%, measure how well configured your system is

Different types of collective communication

Torch.distributed commands

Performance Analysis

Batch size: Gradient accumulation

Activation Recomputation: only save parts to reduce memory consumption

Data parallelize:

Model Parallelize:

Mixture of Expert Models

Efficient Models

Attention variations: to reduce large attention matrix

Associative RNNs