Carbon Emissions and Large Neural Network Training

ML compute demand has grown fast with real energy and carbon costs, but energy use is hard to estimate without detailed information, so the cost is invisible and the optimisation target is undefined.

Measurement methodology plus sensitivity analysis rather than an optimisation algorithm: compute energy use and carbon footprint for several large models (T5, Meena, GShard, Switch Transformer, GPT-3), refine earlier estimates for the evolved transformer architecture search, and attribute the totals to the individual choices of DNN architecture, datacenter location, time-of-day, and processor type.

Not a speedup paper. The reportable findings, each with its own baseline: large but SPARSELY ACTIVATED DNNs can consume LESS THAN 1/10TH the energy of large dense DNNs without sacrificing accuracy, despite using as many or more parameters; geographic location matters because the carbon-free fraction and resulting CO2e vary ~5X-10X even within the same country and the same organization; cloud datacenters are ~1.4-2X more energy efficient than typical datacenters, and ML-oriented accelerators inside them are ~2-5X more effective than off-the-shelf systems; and the COMBINED choice of DNN, datacenter, and processor can reduce the carbon footprint up to ~100-1000X.

ADOPTION: in-production as a REPORTING STANDARD rather than a technique - the authors state they are collaborating with MLPerf developers to include energy usage during training AND inference in that industry-standard benchmark, which is why this 2021 paper is still the citation for energy-per-model figures. VENUE NOTE: no peer-reviewed venue verifiable via Semantic Scholar at verification time (rate-limited) and the arXiv comment carries no venue, so venue = arxiv-preprint, year = ArXiv v1 (Apr 2021). THE NUMBER TO CARRY FORWARD IS THE ~100-1000X SPREAD: the authors use it to argue that retroactive energy estimates are difficult and error-prone, which is the honest framing - a single carbon figure for a model is meaningless without the datacenter, region, time, and hardware recorded alongside it. Note also that the 1/10th-energy sparse-activation result is the origin of the sparsity-is-a-carbon-intervention argument that later work on MoE serving builds on. DANGLING REF STILL OPEN (2026-10-04 paper-read pass; supersedes the earlier 'LEFT UNRESOLVED' note, which reached the same conclusion but only from this record's own text - this time the PAPER itself was read). No A100 anywhere in the paper, and it is a carbon-accounting paper about TPU v2/v3/v4 and P100/V100 comparisons ("Google's custom TPU v2 processor runs Transformer (Big) 4.3X faster than P100 GPUs", "we assume it runs at 40% of the peak floating point performance of Quartz-Tundra (3.2 PetaFLOPS/sec)"). Source: https://arxiv.org/pdf/2104.10350 . Its existing 'nvidia-tesla-p100' sibling ref in the same record is ALSO unsupported: the paper names P100 only as the contemporaneous comparison baseline for Google TPU v2, and there is no P100 record in data/accelerators/ at all, so that ref cannot be resolved either - it needs a new record, not a rename. RESOLVED (2026-10-04 cross-reference repair): 'nvidia-tesla-v100' -> nvidia-v100-sxm2-32gb. data/accelerators/ holds exactly one V100 record, so it is the only possible target - but note the capacity and form factor (32 GB, SXM2) are properties of that record, not claims this paper makes, so do not read the entry as a statement that the paper used a 32 GB SXM2 module. AFFILIATIONS FILLED 2026-10-04 (paper-verification pass): from the Semantic Scholar graph API (paper/arXiv:2104.10350), which is the only machine-readable source that carries author affiliations — the arXiv Atom feed returns author names only, and no ORCID is exposed by either API. One known upstream typo corrected: "Tel Aviv Univesity" -> "Tel Aviv University". S2 affiliation coverage is partial (120 of 249 corpus records carry any), so these lists may be incomplete rather than exhaustive. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.