Reconciliation of the 'only ~40% of throughput benchmarks are priceable' claim against the corpus as it stands on 2026-10-05
Fields
supply/w5s-throughput-priceable-coverage-2026-10-05Notes
THE CLAIM UNDER TEST. docs/08-cost-per-token.md, 'Coverage', states: 'Of 37 throughput benchmark records with a positive value: 15 joinable (40.5%) - 203 rows, 17 supply records, 11 models, 13 hardware x model combinations, 136 provider x model pairs; 22 not joinable'. The same section opens with 'the repo held both halves - ~50 priced supply records and 38 throughput benchmark records'. The parent brief repeats it as 'only ~40% of throughput benchmarks are priceable'.
WHAT THE CORPUS ACTUALLY HOLDS ON 2026-10-05, measured with python tools/cost_per_token.py and a direct count of data/benchmarks/: - data/benchmarks/ now holds 150 records, of which 145 have value > 0. The doc's denominator of 37 is stale by a factor of ~4. - The join table emits 203 rows, which MATCHES the doc. - The gaps list names 67 unpriceable throughput benchmarks. - So the live arithmetic is 203 rows from 15 joinable benchmark records, with 67 further benchmarks named as unpriceable: 15 + 67 = 82 of the 145 positive-value records, leaving 63 positive-value records the tool does not enumerate in either list. That gap of 63 is itself a finding and is described below. - data/supply/ now holds 119 records of which 86 carry a price, not the doc's '~50 priced'.
SO THE 40.5% FIGURE IS NOT MERELY STALE, IT IS NOW UNCOMPUTABLE AS WRITTEN, because its denominator moved from 37 to 145 while its numerator did not move at all. Recomputed on today's denominator the joinable share is 15/145 = 10.3%. Both numbers are true about different corpora and the doc quotes the wrong one for the corpus a reader now has. A reader who reads '40.5% of benchmarks are priceable' and then opens data/benchmarks/ sees ~63 records the doc never mentions.
WHY THE NUMERATOR DID NOT MOVE, WHICH IS THE REAL STORY. 15 of 145 means the corpus grew by benchmarks and not by prices. data/benchmarks/ has gained the MLPerf v6.0/v6.1 submissions - Cisco H200, CoreWeave GB200 and GB300 NVL72, Dell and AMD MI355X, Supermicro, and others - and almost all of them are aggregate throughput at scale, which is exactly the shape that needs a per-GPU-hour denominator the repo does not carry. Growth on the benchmark side without matching growth on the price side lowers coverage automatically. Nobody broke anything; the ratio simply decayed as the corpus grew.
WHICH BENCHMARKS REMAIN UNPRICEABLE, AND WHY, GROUPED BY THE ACTUAL CAUSE (67 records, from the tool's gaps output): (1) MISSING RENTAL RATE - the accelerator has no per-GPU-hour supply record anywhere. This is the largest group by far: 11 x amd-instinct-mi355x, 4 x nvidia-gb200-nvl72, 4 x amd-radeon-instinct-mi50x, 4 x apple-m4-max, 3 x nvidia-rtx-pro-6000-blackwell, 3 x nvidia-rtx-5090, 2 x amd-instinct-mi300x, and one each for intel-gaudi2, intel-arc-pro-b70, cerebras-wse-3, aws-trainium2, aws-inferentia2, google-tpu-v5e, google-tpu-v6e, nvidia-rtx-4090, nvidia-rtx-5080, nvidia-rtx-5070, local-apple-m5-max, local-apple-m5-ultra, local-nvidia-rtx-5070, amd-radeon-rx-7900-xtx, amd-radeon-ai-pro-r9700, amd-radeon-8060s-ryzen-ai-max. IMPORTANT: 13 of these are now WRONG rather than merely open - Oracle published MI300X $6.00 and MI355X $8.60 per GPU-hour on 2026-10-05, so the amd-instinct rows are no longer a market gap, only a missing supply record and a missing PRICE_QUOTES entry. See w5s-oracle-oci-amd-mi300x-mi355x-gpu-hour. (2) MISSING POWER FIGURE - the record measures energy per token rather than throughput, so there is no tok/s denominator to pair with any rate. These are the two Apple M4 Pro energy records and apple-m4-pro-5-model-system-joules-per-token-range, which carry joules per output token instead. Note this is a MEASUREMENT-SHAPE gap, not a pricing gap: the repo has the joules, and a $/kWh record exists, so the missing piece is a throughput record for the same run, not a price. (3) A LICENCE OR TERM THAT DOMINATES - the price exists but is not comparable. aws-ec2-trn1-trn2-trainium prices Trainium at $1.3438/chip-hour but has no Trainium throughput record, and the alt-aws-trn2 and alt-aws-inf2 records are peak-compute specifications (20.8 PFLOPS FP8, 2280 TFLOPS FP16) rather than token rates; gcp-cloud-tpu-v4-v5e-v5p prices TPU hours but the alt-tpu-v5e and alt-tpu-v6e records are again peak compute (197 and 918 TFLOPS BF16). These four are excluded for a different reason from the rest: they are not missing a price and not missing a power figure, they are measuring a quantity that cannot be divided by an hourly rate to give tokens. (4) THE UNIT IS NOT AN OUTPUT-TOKEN RATE - two records report prompt PLUS generated tokens, so the prompt share is not separable: silex-llama33-70b-4xh100-tp4-tps-200x200 (7,000 tok/s at 200-in/200-out) and vllm-qwen38-nvfp4-gb300-nvl72-tps-per-gpu (5,000 tok/s/GPU at 8192-in/1024-out). At 8192 prompt tokens the prompt share dominates, so a $/output-token from either would be badly wrong. No output-only rate exists in the repo for these two. (5) THE PER-USER/AGGREGATE PAIR DOES NOT EXIST - vllm-qwen38-nvfp4-gb300-nvl72-interactivity reports 180 tok/s/user, the only real per-user figure on real hardware, but its sibling aggregate is a TOTAL-tokens figure, so the pair needed for a per-output-token cost does not exist. (6) NO ACCELERATOR NAMED - 5 records: accelerator_ids is empty. Three are the mpt7b per-user records plus two others. These cannot be priced for the boring reason that nothing identifies the hardware. (7) HEAD-TO-HEAD ROWS THAT MUST NOT BE JOINED - 2 records carry accelerator_ids listing MI300X and H100 together because they are the head-to-head rows; the H100-only and MI300X-only twins carry the split values and are the ones that should join. (8) 7 records are listed as 'see EXCLUDED in tools/cost_per_token.py', i.e. deliberately withheld from the join rather than blocked by it.
THE OTHER DIRECTION, WHICH IS THE CHEAPER FIX. The tool's own second list names priced accelerators with no throughput record at all. The doc claims nvidia-a100-80gb-sxm4 is priced by 9 supply records and has zero throughput records, and the same for nvidia-l40s (6 records), nvidia-l4 (4), nvidia-h100-pcie (2), nvidia-a10 (1). Price is abundant; throughput is absent. That direction is worth more per unit of effort than the AMD gap because the price records already exist and only a benchmark run is missing.
WHAT THIS RECORD DELIBERATELY DOES NOT DO. It does not edit docs/08-cost-per-token.md, tools/cost_per_token.py or tools/ref_budget.json, all outside this wave's write scope, and it does not restate the 40.5% figure as current. The specific edits a sibling should make are: restate the denominator as 145 positive-value records with 15 joinable (10.3%) and name the 63 positive-value records that appear in neither list, or explain why the tool does not enumerate them; replace the '~50 priced supply records' figure with 86; and rewrite item 2 of 'What the records cannot answer yet', which is now false. The safest single change is to stop quoting a percentage whose denominator has quadrupled.
Cross-references
This record references no other records — it is a leaf. It is cited by 0.