08 — Cost per output token

This doc closes the gap that 01-hardware-selection.md listed as its #1 unanswered question: "Cost per token. No price↔throughput join exists." The repo held both halves — ~50 priced supply records and 38 throughput benchmark records — and nothing joined them.

It now exists, and it is a tool, not a spreadsheet:

python tools/cost_per_token.py            # the table
python tools/cost_per_token.py explain    # the arithmetic for every row
python tools/cost_per_token.py gaps       # what the records cannot answer

Every figure below is the output of that tool against the current records. Each row carries two record ids: one supply price and one benchmark throughput. Nothing here is interpolated, extrapolated or averaged; where a number is not in a record the cell says no record.

The formula, and the two ways to get it wrong

USD per million output tokens = (USD per GPU-hour) / (output tokens/s per GPU) x 1,000,000

Both denominators are the trap. The price side, because price_usd is not per-GPU-hour — a record priced "per 8-GPU node per HOUR" is 8x a per-GPU-hour quote, and getting that backwards is an 8x error. The throughput side, because value is not per-GPU — a TP=8 record is the whole 8-GPU group.

The tool therefore refuses to print anything it cannot show is in a record. Each price quote stores a literal substring of the price_basis it was read from, and each GPU count stores a literal substring of the benchmark's unit or methodology. If a record is edited and no longer contains that substring, the tool exits non-zero and prints UNSOURCED INPUT rather than a stale number. That check runs in tools/test_tools.py against the real data/, so an 8x price-basis error cannot reach this doc quietly.

ASSUMPTIONS — read before quoting any number here

Every row below requires all of these. They are not hedges; they are the conditions under which the arithmetic is even meaningful.

1. The throughput is almost all AGGREGATE, not single-user. 28 of 29 joinable records in the repo are tps_aggregate (--single-user returns 1 row). A row here is a fleet-average cost at whatever concurrency that benchmark used — it is not what one user pays. The repo documents exactly how wrong this can be: Per-user decode tok/s, MPT-7B static batch 1, 1x A100 gives 57.6 tok/s for one user at batch 1, Aggregate output throughput, MPT-7B static batch 64, 1x A100 gives 800 aggregate at batch 64, and Per-user decode tok/s, MPT-7B static batch 64, 1x A100 gives 12.5 tok/s for one user in that same batch-64 run. Aggregate rose 13.9x while the experienced rate fell 4.6x. A single-user cost cannot be derived from these records; see "What the records cannot answer".

2. Price tier is not interchangeable, and rows are NOT comparable across tiers. The tier column is load-bearing:

A spot row and an on-demand row for the same model differ by up to 5.4x (Azure H100, 8B: $151.50 spot vs $819.79 on-demand). The cheapest row in this doc is a spot row and is not available on demand at that price.

3. Used-market and auction records are excluded on purpose. Every used-market, auction, regional-reseller, enterprise-distributor, oem-direct and integrator price in the repo is a price per card or per configured system, not per GPU-hour, and the repo holds no amortised runtime, power draw or utilisation figure that would turn Used / secondary market: NVIDIA H100 80GB SXM5 (bare module, ASKING band) or Secondary / integrator: refurbished 8x H100 SXM5 server node (whole-system band) into a cost per token. That is 24 records, listed by --gaps and not in the table. Any $/Mtok built from them would require inventing an hours-of-use denominator, which the repo's own rules forbid. Four further priced records are excluded for other reasons: two accelerator families with no throughput record (AWS EC2 Trn1 / Trn2 (AWS Trainium and Trainium2), Google Cloud TPU v4 / v5e / v5p (per chip-hour, in a pod)), one consumer-GPU rental index (Peer-to-peer rental index: consumer GeForce (4090 / 3090 / 5090 per-GPU-hour)), and the electricity tariff in Cost to RUN, not to buy: US average industrial electricity price, 8.91 cents/kWh trailing-12-months to July 2026 (EIA Electric Power Monthly Table 5.3); 8.13 was the CY2024 figure, which is a $0.0813/kWh energy price rather than any compute price at all.

4. The throughput belongs to THAT model, at THAT precision, on THAT engine. Do not generalise across models. A B200's 70B FP4 row does not predict its 8B FP8 cost, and nothing here predicts the cost of a model the repo has no benchmark for. Likewise the TensorRT-LLM rows are NVIDIA-engine numbers; vLLM and SGLang rows are different engines on the same silicon.

5. Throughput is vendor-reported unless stated. Of the joinable benchmarks, 27 of 29 are measured_by: vendor. Two are third-party: Perplexity: Llama 2 70B FP8 on 8x H100 80GB, TP=2 / batch 128, 767 output tokens/s per GPU and SGLang: DeepSeek-V3 671B FP8 on 96x H100 with PD disaggregation + large-scale EP, 22300 output tokens/s per node. Vendor aggregate figures are published under an offline infinite-rate feed — the best point on a concurrency sweep the vendor chose, not a sustained production operating point.

6. Concurrency is usually not stated. Most joined records say so explicitly ("concurrency NOT stated and NOT fixed"). The aggregate rate depends on concurrency, so a row's figure is valid only at the concurrency the source used, which is frequently off the record. This is the single largest source of optimism in the table.

7. Price and throughput are observed on different dates and in different places. Supply prices were captured 2026-10-03; benchmarks were published earlier and may predate a price change. AWS's own schedule lowers p4d Capacity Blocks and raises p5en Capacity Blocks for 2026-10-07 — see the notes in AWS EC2 P5 (NVIDIA H100 SXM, on-demand + Capacity Blocks). Prices are the most perishable data in this repo.

8. This is GPU cost only. No CPU, host RAM, storage, egress, idle-time or PUE overhead is in the figure. Modal serverless GPU containers (per-second, autoscale to zero) charges egress at $0.04/GiB separately. A real bill is higher.

The table

provider × model × hardware (220 aggregate, 1 single-user). Full 221-row output: python tools/cost_per_token.py.

Llama 3.1 8B FP8 on 1x H100 SXM — the best-served shape in the repo

Throughput from TensorRT-LLM: Llama 3.1 8B FP8 on 1x H100 SXM 80GB, ISL/OSL 1000/1000: 14,991.62 output tok/s aggregate on one H100 (TP=1), so no GPU-count division is needed.

5.4x spread between the cheapest spot row and the dearest on-demand row, same GPU, same model, same benchmark.

Llama 3.1 8B FP8 on 1x H200 SXM

Throughput from TensorRT-LLM: Llama 3.1 8B FP8 on 1x H200 SXM 141GB, ISL/OSL 1000/1000: 17,162.49 tok/s on one H200.

Azure H200 v5 has no cheaper spot row because its spot price equals its on-demand price ($84.80/8) — a 0% discount, which per the record's own notes is what a sold-out fleet looks like in a price API.

Llama 3.3 70B FP8 on 2x H100 SXM

Throughput from TensorRT-LLM: Llama 3.3 70B FP8 on 2x H100 SXM 80GB (TP=2), ISL/OSL 1000/2000: 3,708.93 tok/s across a TP=2 group = 1,854.46 tok/s per GPU. The division by 2 is where a naive join would have understated every figure below by 2x.

Llama 3.3 70B FP4 on 1x B200

Throughput from TensorRT-LLM: Llama 3.3 70B FP4 on 1x B200 180GB, ISL/OSL 1000/2000: 6,725.03 tok/s on a single B200 (the FP4 checkpoint fits one GPU, TP=1).

The B200 at FP4 is 1.88x cheaper per token than the H100 running the same 70B at FP8 on the same provider and the same tier (Runpod on-demand: 1,009.66 vs 1,881.94), on one GPU instead of two. Against the H200 on-demand (2,177.76, CoreWeave) it is 2.16x.

DeepSeek-V3 671B FP8 per 8xH100 node

Throughput from SGLang: DeepSeek-V3 671B FP8 on 96x H100 with PD disaggregation + large-scale EP, 22300 output tokens/s per node: 22,300 tok/s per 8-GPU node = 2,787.5 tok/s per GPU.

The controlled comparison: two models, one node, one power envelope

8xH200 SXM-141GB (MaxQ), MLPerf v4.1 llama2-70b Offline: 25262.1 output tok/s at 5806.9 W mean node draw = 0.2299 J/output token and 8xH200 SXM-141GB (MaxQ), MLPerf v4.1 mixtral-8x7b Offline: 48987.9 output tok/s at 5854.3 W mean = 0.1195 J/output token are the same 8xH200 MaxQ node, same instrument, same locked 1000 MHz clock. Only the model changes. This is the cleanest cost-per-token contrast the repo can make, and it needs no price join at all — the records already carry joules per token (0.2299 vs 0.1195 J/output token).

At one CoreWeave spot rate ($2.61625/GPU-hr), the join reproduces the energy result exactly, which is a good check that both denominators are right:

MoE is 1.94x cheaper per token than a dense 70B on identical hardware. The ratio of costs (0.5157) equals the reciprocal of the throughput ratio to four decimal places, as it must when only the model changes.

But do not read this as "MoE is cheaper." The records say so themselves: the two models are not of comparable quality, and ~12.9B of Mixtral's 46.7B parameters are active per token. Per useful token, per byte of weight shipped, or per unit of output quality, this comparison does not settle anything.

Llama 2 70B on 8xH200 — the power cap has a price

Power-capping a node makes each token 1.24x more expensive. The cap saves power but costs throughput, and at a fixed hourly rate throughput loss is straight-through cost. This is the repo's clearest demonstration that $/Mtok and J/token are different quantities — J/token improves under the cap while $/Mtok gets worse.

Llama-2-70B FP8, 128-concurrency, third-party

Throughput from Perplexity: Llama 2 70B FP8 on 8x H100 80GB, TP=2 / batch 128, 767 output tokens/s per GPU: 767 tok/s per GPU (already per-GPU in the record's own unit string), batch 128. Independent third party on their own AWS p5 fleet.

The dearest rows, and why they are not interesting

vLLM: Meta-Llama-3.1-405B-Instruct-FP8 on 8x H100 (TP=8), aggregate output throughput reports 291.53 output tok/s across 8xH100 = 36.44 tok/s per GPU, at 1024-in/128-out. That record's own methodology says the concurrency was neither stated nor controlled and the post calls the figures "preliminary". Dividing an hourly rate by it produces $274,414 to $337,255 per million tokens on on-demand providers.

Those numbers are arithmetically correct and should not be used as a cost estimate. The record is a latency-sensitive short-output workload measured at an unstated concurrency; a $/Mtok from it describes nothing purchasable. It is listed here so the range is complete and so nobody later mistakes the top of the table for a finding.

Price-basis errors nearly made

Each of these was a live wrong answer until the record text was read.

What the records cannot answer yet

python tools/cost_per_token.py gaps is the machine version of this section.

1. Any single-user cost per token. Zero rows. This is the biggest gap and it is structural: the repo has per-user records only where accelerator_ids is empty — Per-user decode tok/s, MPT-7B static batch 1, 1x A100, Per-user decode tok/s, MPT-7B static batch 64, 1x A100 — so they cannot be priced. The one real per-user record on real hardware, vLLM disaggregated PD serving: Qwen3.8-2.4T-A95B NVFP4 on GB300 NVL72, 180 generated tokens/s per user (180 tok/s/user), has a sibling aggregate that is a total-tokens figure, so the pair needed for a per-output-token cost does not exist. Filling this needs a benchmark with both tok_s_per_user and an output-token tps_aggregate on the same accelerator.

2. AMD Instinct is now priceable, and the join still cannot use it — because the tool's price table has not caught up. This correction is recorded because the previous version of this document asserted the opposite, and the assertion was wrong in a way that generalises.

What is true now. Oracle OCI bare-metal AMD Instinct MI300X / MI355X per-GPU-hour list price prices AMD Instinct per GPU-hour: MI300X at $6.00 and MI355X at $8.60, read off Oracle's own accelerated-compute table, in the same column and on the same page as NVIDIA BM.GPU.H100.8 at $10.00 (Oracle Cloud Infrastructure BM.GPU.H100.8 / BM.GPU.H200.8 prices the H100 and H200 at $10.00). The per-GPU-hour denominator this document called impossible exists, it is first-party, and it sits beside the NVIDIA figure a like-for-like comparison needs.

What the earlier claim got wrong, and why it was not merely out of date. It cited four AMD supply records — amd-instinct-oem-systems, dell-poweredge-xe-ai, hpe-proliant-compute-xd, qct-gpgpu-servers — all of which genuinely have price_usd: null. Those four are correct as they stand: they describe OEM and reseller channels, which do not quote per-GPU-hour because per-GPU-hour is not a unit an OEM system sale is priced in. The error was one of scope: the claim was stated as "AMD hardware is unpriceable" when the only thing established was "no supply record in this repo priced AMD per GPU-hour". The channel that publishes per-GPU-hour rates for AMD is a cloud provider, not an OEM, so the negative generalised across a boundary it was never tested at. This is the same shape of error as rent-cerebras-api-token-pricing-negative elsewhere in the repo: a negative asserted on one channel, generalised to the whole market.

What is still true, and why gaps still reports AMD as unpriceable. The gap is closed on the supply side and still open in the tooling. tools/cost_per_token.py is driven by a hand-maintained PRICE_QUOTES table rather than by scanning data/supply, so the new record does not by itself reach the tool: python tools/cost_per_token.py gaps still lists 11 records on amd-instinct-mi355x and 2 on amd-instinct-mi300x as having no priced per-GPU-hour supply record. Adding two entries of the form dict(accel="amd-instinct-mi300x", usd=6.00, tier="on-demand", gpus=1, basis="BM.GPU.MI300X.8 = $6.00") and its MI355X equivalent is an edit in tools/, outside this document's scope. The 19 AMD MI3xx throughput records in the repo — including MI355X, MLPerf Inference v6.0 Offline: Llama 2 70B WMXFP4 on 8xMI355X, aggregate output throughput at 103,480 tok/s, Dell XE9785L 8xMI355X, MLPerf Inference v6.1 Offline: Llama 3.1 8B, third-party OEM reproduction of AMD silicon at 158,458 tok/s, MI355X, MLPerf Inference v6.0 Offline: gpt-oss-120b (MXFP4/FP8 MoE) on 8xMI355X, aggregate output throughput at 95,004 tok/s and llama.cpp on 8xMI300X: DeepSeek-V3 671B Q4_K_M decode throughput at batch 1, 256 generated tokens at 36.53 tok/s single-user — would all become priceable on that one edit.

What it buys, stated at the limit. $6.00 is an on-demand asking rate on a bare-metal shape with no vCPU/NRAM bundling. For a like-for-like AMD-vs-NVIDIA comparison use Oracle's own H100 at $10.00, not AWS p5's $6.88-equivalent, which bundles 24 vCPU and 256 GB of RAM into the GPU hour. And Oracle's availability on GPU shapes is unverified from any public page — its H100 record says so directly — so these figures describe price, not purchasability.

3. GB200/GB300 NVL72 racks are unpriceable per GPU. SGLang: DeepSeek 671B PD-disaggregated + large-scale EP on GB200 NVL72, 7583 decode tokens/s per GPU (7,583 tok/s per GPU) and vLLM disaggregated PD serving: Qwen3.5-397B-A17B-NVFP4 on GB200 NVL72, 25K total TPS per GPU (25,000 tok/s per GPU) have no per-GPU-hour supply record. CoreWeave's $42.00/h for a whole 72-GPU Superchip is in CoreWeave Cloud (HGX H100 / H200 / B200 / A100 nodes), but the benchmark's rack (12 of 14 nodes, 768 GPUs) is not that shape.

4. A100, L4, L40S and A10 are priced but have no throughput record at all. nvidia-a100-80gb-sxm4 is priced by 9 supply records (AWS EC2 P4d / P4de (NVIDIA A100 SXM4 40GB and 80GB), Microsoft Azure ND A100 v4 (ND96asr 40GB and ND96amsr 80GB), Google Compute Engine A2 (NVIDIA A100 40GB / 80GB), Lambda AI cloud GPU instances (NVIDIA H100 SXM, self-serve), Modal serverless GPU containers (per-second, autoscale to zero), Oracle Cloud Infrastructure BM.GPU4.8 / BM.GPU.A100-v2.8 / L40S, Runpod Pods (dedicated H100 / H200 / A100 GPU instances), Runpod Serverless (per-worker inference endpoints), CoreWeave Cloud (HGX H100 / H200 / B200 / A100 nodes)) and not one throughput record exists for it. Same for nvidia-l40s (6 records), nvidia-l4 (4), nvidia-h100-pcie (2), nvidia-a10 (1). Price is abundant and throughput is absent — the mirror image of the AMD problem, and much cheaper to fix.

5. Consumer and non-NVIDIA accelerators are entirely unpriceable. llama.cpp under ROCm on 2x RX 7900 XTX (gfx1100): Qwen3.8-27B Q4_K_XL batch-1 tensor-split decode throughput, llama.cpp on Radeon AI PRO R9700 (gfx1201, RDNA4): Bonsai-27B Q1_0 decode throughput, batch 1, MLX-LM on 64GB Apple M4 Max: Qwen3-30B-A3B-Instruct-2507 4-bit generation throughput (128 tokens), MLX-LM on MacBook Pro 14-inch M3 Pro 18GB: Qwen3-4B-Instruct-2507 4-bit generation throughput, AMD Ryzen AI Max+ Radeon 8060S iGPU (RDNA 3.5) under Ollama: Qwen3.5 35B-A3B MoE generation throughput, 100% GPU offload, Intel Arc Pro B70, MLPerf Inference v6.1 Offline: Llama 2 70B on 4x Arc Pro B70 (GDDR6), aggregate output throughput, Intel Gaudi 2, MLPerf Inference v4.0 Offline: Llama 2 70B fp8 on 8xGaudi 2, aggregate output throughput — seven real single-user measurements with no per-GPU-hour price for their hardware.

6. Two records measure total tokens, not output tokens, and are excluded. Silex (third-party genai-perf): Llama 3.3 70B Instruct bf16 on 4x H100 80GB SXM, ~7000 total tokens/s at 200/200 (7,000 tok/s at 200-in/ 200-out) and vLLM disaggregated PD serving: Qwen3.8-2.4T-A95B NVFP4 on GB300 NVL72, 5000 total TPS per GPU (5,000 tok/s/GPU at 8192-in/1024-out) report prompt plus generated tokens. At 8192 prompt tokens the prompt share dominates, so a $/output-token from them would be badly wrong. No record in the repo carries an output-only rate for these.

7. No utilisation figure exists anywhere. The $/Mtok above assumes a GPU-hour is fully used. Real serving is bounded by KV-cache capacity and by Sustained load drives SW Power Cap and thermal throttling, which move the tail of the latency distribution rather than its mean - so a benchmark that reports only p50 hides the regression entirely-class effects. There is no utilisation field in the supply schema and no achieved-throughput-below-peak record, so cost per token actually delivered to a user cannot be computed — only cost per token at vendor peak.

8. Not answerable at all: input-token cost. Every figure here is output. Silex (third-party genai-perf): Llama 3.3 70B Instruct bf16 on 4x H100 80GB SXM, ~7000 total tokens/s at 200/200 shows prefill is a different rate (52.3k input vs 22.3k output tok/s per node on SGLang), but no joined record pairs a prefill price with a prefill rate.

Coverage: what the join does and does not reach

Of 145 benchmark records with a positive value:

• 29 joinable — 20.0%. 221 rows, 18 supply records, 17 models, 20 hardware×model combinations, 145 provider×model pairs.
• 51 throughput records named in --gaps as unpriceable, which together with the 29 joined accounts for all 80 records whose metric is one of decode_tok_s, tok_s_per_user or tps_aggregate.
• The other 65 positive-value records are not throughput records at all and are therefore outside this join's denominator by construction: 33 quality, 10 dimensionless_ratio, 8 ttft_ms, 7 prefill_tok_s, 4 speedup_ratio, 3 itl_ms.

This document names 28 of the 145 positive-value records; 117 are never mentioned here, and the gap is concentrated in exactly the classes a cost model needs and this join cannot serve: 33 of the 51 tps_aggregate records and 17 of the 24 decode_tok_s records appear nowhere in this doc, because they sit on hardware no price record covers (AMD Instinct, GB200/GB300 NVL72, Apple, consumer Radeon, Gaudi2, Arc) or on a benchmark-only channel with no accelerator id to join on. 65 positive-value records have no accelerator id at all and are structurally unpriceable regardless of how many prices exist. Where another document carries those records, cite it; the point of stating the count here is that a 20.0% join is not a pricing model, it is a worked example on the fifth of the corpus the price records happen to reach.

Almost every joinable row is aggregate (220 aggregate, 1 single-user). The join answers "what does a saturated fleet cost per million output tokens", which is a real and useful question. It does not answer "what will my user pay per token", which is usually the question being asked.

Three findings the join makes visible that were not obvious from the records alone:

1. Price tier dominates hardware choice for a fixed model. On 8B FP8 on H100, spot Azure ($151.50) beats on-demand Runpod ($232.80) beats on-demand hyperscaler Oracle ($667.04) — a 2.9x spread from when and how you buy alone, before any hardware decision. Across all 16 H100 8B rows the full spread is 5.4x ($151.50 spot to $819.79 on-demand).
2. A newer GPU at lower precision beats an older one at higher precision. B200 FP4 at $1,009.66 vs H100 FP8 at $1,881.94 for 70B on the same provider and tier — 1.86x, on one GPU instead of two.
3. Power caps trade joules for dollars in the wrong direction for this metric. J/token improves 17% under the MaxQ cap; $/Mtok worsens 24%.

Reproducing

python tools/cost_per_token.py                # 221 rows
python tools/cost_per_token.py explain        # per-row arithmetic
python tools/cost_per_token.py gaps           # the unpriceable records
python tools/cost_per_token.py --single-user  # 1 row
python tools/test_tools.py                    # the evidence check is tested

The tool stores, for every price, the literal substring of the price_basis it was read from, and for every throughput, the literal substring establishing the GPU count. Editing a record to change a price makes the tool exit non-zero. If a row in this doc stops matching the tool's output, the tool is right and this doc is stale.