id
gotchas/flops-row-had-no-basis-field
Concepts
spec-sheet-transcriptionprovenanceschema-validationdense-vs-sparseprecision-enumdead-enum-values
Symptom
The accelerator schema's flops[] items required exactly two fields - precision and tflops - and permitted five optional ones (dense, vendor_claim, source_id, notes, unit). A row could therefore state a TFLOPS figure, set vendor_claim:false, leave source_id null, and carry its entire justification in free-text notes, and it would pass `python tools/validate.py` cleanly. That is precisely the shape in which the groq-lpu record's two values survived (188 TFLOPS FP16, 750 TOPS INT8, both with source_id:null and both attributed only to 'press coverage of the original GroqChip, widely attributed to Groq's Hot Chips 2021 presentation'). Nothing in the schema FORCED a derivation to be shown, so nothing forced a reviewer to notice the values appeared in no source in data/sources/. The same gap is why a record could carry EIGHT flops rows with no per-row provenance: there was no field on which per-row provenance could even be recorded, only a whole-record `sources` array and a per-row `source_id` that the schema permitted to be null. A second, quieter failure lived in the precision enum, which had no 'fp64' member. Every FP64 figure in the repo was therefore filed as precision:'other', which made an FP64 row indistinguishable from a row whose precision nobody had determined at all. Counting on 2026-10-04: 19 rows were filed 'other' and they were in fact FOUR different things - 16 genuine FP64 rows (NVIDIA A100 x3, H100 SXM/PCIe/NVL, H200 SXM/NVL, V100, GB200, MI355X/MI350X/MI350P), 1 INT16 TOPS row (Cambricon MLU370-X8), and 2 that were not that at all: huawei-ascend-950dt's 1560 TFLOPS row, which was a byte-identical DUPLICATE of the record's existing fp4_block_scaled row (both derived the same way, 12.48 PFLOPS server / 8 NPUs), and horizon-journey-6p's 560 TOPS row, which was an unattributed-precision duplicate of that record's own int8 row.
Root cause
The repo's contract (AGENTS.md rule 4) requires that 'numbers derived from other numbers carry a *_basis field explaining the derivation', and accelerator.schema.json enforces that convention on memory_bandwidth via memory_bandwidth_basis. It was not enforced inside flops[], where the numbers actually live. A convention enforced in one place and not another is not a convention, it is a suggestion - and validate.py is a schema-subset validator, so anything the schema does not require is not checked at all. The precision enum compounded it: 'other' was doing duty as a catch-all for both 'a real precision this enum lacks' and 'nobody knows what this is', and an enum member that means two different things hides both.
Workaround
Treat every flops[] row as requiring a stated derivation regardless of what the schema says, and record the derivation where it cannot be missed. Concretely, for each row state which of these it is and show the working: a transcribed spec-sheet row (name the table column and the footnote that governs dense vs sparse), a derived row (show the arithmetic with the inputs substituted, e.g. '12.48 PFLOPS server / 8 NPUs = 1560'), a second-hand or aggregator row (say so, and set vendor_claim false), or an unverified row with source_id null. If a figure cannot be found in any cited source, say that in the row itself rather than in a record-level note where it competes with forty other claims. When a precision does not fit the enum, add the enum member rather than filing it under 'other' - and before adding one, check whether an existing member already covers it, because the FP64 case would have added a member while the real answer for two of the 19 rows was 'this is a duplicate' and for one it was 'the aggregator never stated a precision'.
notes
Fixed 2026-10-04 (w3f basis backfill). Changes to schemas/accelerator.schema.json: flops[] items now require a third field, `basis`, an enum of vendor-spec-sheet / vendor-whitepaper / vendor-docs / secondary-aggregator / derived / unverified, plus an optional `basis_detail` carrying the derivation in words with the numbers substituted; `unit` was redefined with a stated purpose (distinguishing two hardware paths at one precision, e.g. A100's 9.7 vector vs 19.5 tensor-core FP64, which must never be summed). Backfill: all 386 rows across 80 records now carry `basis`, distributed vendor-spec-sheet 270, vendor-whitepaper 73, secondary-aggregator 21, vendor-docs 15, derived 5, unverified 2 - no row carries an empty value. `precision` gained 'fp64' (17 rows, all previously 'other') and 'int16' (1 row, Cambricon); 'other' now has ZERO rows in the repo. An 'mlperf-submission' basis value was drafted and then deliberately REMOVED unused: MLPerf Inference submissions report achieved throughput under a named scenario, never vendor peak tensor-core TFLOPS, so the repo's 32 MLPerf sources of kind 'benchmark' cannot back a flops row and adding the member would have reproduced this repo's documented failure mode of enum values recognised by zero records. The huawei-ascend-950dt duplicate row was removed rather than relabelled. THE PARALLEL FINDING WORTH CARRYING FORWARD: requiring a new field broke four tests in tools/test_tools.py whose fixture accelerator omitted it, which is the correct behaviour - a required field SHOULD break fixtures that omit it - but it means the blast radius of a schema change includes test fixtures, and those fixtures are the repo's own statement of what a minimal valid record looks like. Related: [[structured-2-4-sparsity-mma-sp]], [[amd-fp8-ocp-vs-mxfp8-precision]], [[bw-fp8-fine-grained-scaling-hopper-vs-blackwell]], [[block-scaled-tensor-core-fp4-fp8]].