Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

Origin of tensor parallelism for transformers: shards each layer's weight matrices across devices and inserts two communication ops per layer, an all-reduce between the attention block and the MLP and an all-gather of the sequence dimension after the MLP. Abstract reports 8.3B parameters on 512 GPUs at 15.1 PFLOPs with 76% scaling efficiency vs a 39 TFLOP/s single-GPU baseline (30% of peak), which is the framing this repo uses for the per-layer communication cost. The paper's 'few communication operations in native PyTorch' wording is the primary-source basis for 'the all-reduce is a fixed small number of ops per layer, not a stream of small ones'. | SOURCE AUDIT 2026-10-03: the 'publisher' field held a paragraph of prose (authors + affiliations, 68 chars) instead of a publisher name; trimmed to 'arXiv' with the authorship detail moved here.