inference/research
verifiedhigh · 0.90majorupdated 2026-10-10

A record can cite a source_id that exists but is absent from its own sources[], and no check in the repo can see it

[major] toolchain

Fields

id
gotchas/w10r-cited-source-id-is-not-a-declared-source
Class
toolchain
Severity
major
Concepts
provenancecross-referencingschema-validationflopssourcesvalidated-invariant
Symptom
AGENTS.md rule 2 says every claim must have at least one sources entry and every entry must resolve, and tools/validate.py implements exactly that: it resolves each sources[] entry against data/sources/ and errors on a dangling one. The invariant is one-directional. A record whose notes cite a source BY ID - 'Sources: arch-deepseek-v2-mla, arch-nsa, ...' - while that id is missing from sources[] passes validate.py clean, because there is nothing for the validator to resolve. The cited source is a real, existing, read file; only the declaration is missing. On 2026-10-10, scanning all 156 data/flops/ records for any source id appearing anywhere in a record but absent from its own sources[] found exactly two (attn-low-rank-kv-compression-status citing arch-deepseek-v2-mla, and xpu-gaudi-static-shapes citing xpu-vllm-gaudi-plugin-system). Neither is a typo and neither points at a nonexistent source: both are genuine citations whose record-level provenance list was never backfilled. The failure is silent in both directions - a reader cannot tell that the notes' citation is undeclared, and a query joining on sources[] under-reports the record's actual evidence.
Root cause
The schema defines sources as an array of source ids and validate.py checks it in one direction only: for each entry in sources[], does a file exist. There is no reverse check - 'for each source id this record mentions anywhere, is it in sources[]?' - because the validator never parses notes for ids in the first place. The same asymmetry made the original wave-8 finding invisible on 121/121 passing tests. It is the mirror image of the dangling-reference budget in tools/ref_budget.json: that budget catches references that resolve to NOTHING, while this defect's references resolve to something real. A second reason the shape survives: citing a source in prose reads as extra evidence, so nothing in the record looks wrong to a human reviewer, and prose is where most citations live in this repo because notes carry the reasoning.
Workaround
Treat 'the record must declare every source it cites' as the missing half of rule 2, and check it mechanically rather than by eye: load the source ids from data/sources/, walk every string field of every record, and match each id as a BOUNDED token. Bounded matters - a naive substring match reports arch-deepseek-v3 as undeclared inside the declared arch-deepseek-v32-dsa, which produces a false positive that trains the reader to ignore the check. Longest-id-first alternation with lookarounds on slug characters ([A-Za-z0-9_-]) gets both right. A fix is one line: add the missing id to sources[]. The check is worth adding to validate.py, not just to an ad-hoc script, but that is a tools/ edit outside this wave's scope - record it rather than doing it.
notes
FOUND AND FIXED 2026-10-10 (w10-xref-consistency) over data/flops/, which owns data/flops/ and data/gotchas/ but not tools/. Both records were fixed by ADDING the cited id to sources[] - never by editing the citation text or the number it cites, which would have hidden the defect instead of repairing it. Each repair is annotated in the record's own notes with a dated '=== SOURCE DECLARED ===' section, matching the annotation convention wave-8 used for the same defect class on data/accelerators/. What was measured, so the negative result is reproducible and not a claim: - 156 flops records, 609 affected_by_hardware values, 2018 source ids, 116 accelerator ids. - Undeclared source ids: 2 before the fix, 0 after (re-run of the audit script). - Literal '%s' printf placeholders in data/flops/: 0. The wave-3 corruption (commit 9968769) that wave-8 repaired on 34 accelerator records did not propagate into flops, and no other format artifact was found either (checked for %X, %d, {0}, ${...}). - hardware_parameters[].values_for holds bare accelerator ids per its schema description ('bare accelerator ids, only where the record commits to named parts'): 446 values across 65 records, 1 unresolved (gfx950 on amd-fp4-mxfp4-support) - the same ROCm LLVM target-name class as the affected_by_hardware debt, not a new defect. TWO NEAR-MISSES WORTH RECORDING, because both would have produced a WRONG fix rather than a missed one: 1. The substring false positive above (arch-deepseek-v3 inside arch-deepseek-v32-dsa) reported 8 phantom undeclared ids on the first run. A reviewer trusting the count would have 'declared' sources that were already declared, inflating sources[] and wasting a reviewer's attention on 8 records that were fine. 2. attn-low-rank-kv-compression-status now declares BOTH attn-deepseek-v2-mla-paper and arch-deepseek-v2-mla, which are two source records for the SAME arXiv paper (2405.04434) under different ids and different publisher strings ('arXiv' vs 'DeepSeek-AI (arXiv)'). They were deliberately NOT merged: data/sources/ is outside this wave's write scope, and consolidating sources is a rename that must be verified to have zero inbound references first. The duplication is recorded in the record's notes so a future sources pass consolidates it instead of rediscovering it. THE DANGLING affected_by_hardware FINDING, recorded here because it shares this wave's scope: 9 unresolved of 609 values in data/flops/. Eight are ROCm LLVM target names - gfx950 (4), gfx942 (3), gfx1151 (1) - which name real silicon but have no accelerator record to point at, so they stay as budgeted debt and are NOT typos. The ninth is amd-ryzen-strix-point on data/flops/embedding, the only product-shaped dangling value in the directory. It was re-examined and deliberately left dangling: repointing it to amd-radeon-8060s-ryzen-ai-max would assert Strix HALO silicon (RDNA 3.5 iGPU, gfx1151) where the record means Strix POINT (gfx1150, the laptop APU), and deleting it would drop the only pointer to 'AMD APU designs' in the record's unified-memory argument. The repair it needs is an accelerator record for Strix Point, which is a data/accelerators/ write. The record's 2026-10-04 note already reached this conclusion and the 2026-10-10 re-examination confirmed it, which is the pattern the ledger (docs/26-known-defects.md) exists to stop being re-derived. WHY A NEGATIVE RESULT IS THE RIGHT OUTPUT FOR TWO OF THE THREE CHECKS: an audit whose job is to confirm that a defect has not propagated into a second directory gets its value from being re-runnable, not from finding something new. The '%s' pass is the clearest case - 34 accelerator records were corrupted and repaired in wave 8, so the obvious next question is whether flops carried the same corruption, and the answer is no. Recording the measurement (0 records, checked for every format token) is what lets the next wave skip it instead of re-deriving it.

Notes

FOUND AND FIXED 2026-10-10 (w10-xref-consistency) over data/flops/, which owns data/flops/ and data/gotchas/ but not tools/. Both records were fixed by ADDING the cited id to sources[] - never by editing the citation text or the number it cites, which would have hidden the defect instead of repairing it. Each repair is annotated in the record's own notes with a dated '=== SOURCE DECLARED ===' section, matching the annotation convention wave-8 used for the same defect class on data/accelerators/.

What was measured, so the negative result is reproducible and not a claim: - 156 flops records, 609 affected_by_hardware values, 2018 source ids, 116 accelerator ids. - Undeclared source ids: 2 before the fix, 0 after (re-run of the audit script). - Literal '%s' printf placeholders in data/flops/: 0. The wave-3 corruption (commit 9968769) that wave-8 repaired on 34 accelerator records did not propagate into flops, and no other format artifact was found either (checked for %X, %d, {0}, ${...}). - hardware_parameters[].values_for holds bare accelerator ids per its schema description ('bare accelerator ids, only where the record commits to named parts'): 446 values across 65 records, 1 unresolved (gfx950 on amd-fp4-mxfp4-support) - the same ROCm LLVM target-name class as the affected_by_hardware debt, not a new defect.

TWO NEAR-MISSES WORTH RECORDING, because both would have produced a WRONG fix rather than a missed one: 1. The substring false positive above (arch-deepseek-v3 inside arch-deepseek-v32-dsa) reported 8 phantom undeclared ids on the first run. A reviewer trusting the count would have 'declared' sources that were already declared, inflating sources[] and wasting a reviewer's attention on 8 records that were fine. 2. attn-low-rank-kv-compression-status now declares BOTH attn-deepseek-v2-mla-paper and arch-deepseek-v2-mla, which are two source records for the SAME arXiv paper (2405.04434) under different ids and different publisher strings ('arXiv' vs 'DeepSeek-AI (arXiv)'). They were deliberately NOT merged: data/sources/ is outside this wave's write scope, and consolidating sources is a rename that must be verified to have zero inbound references first. The duplication is recorded in the record's notes so a future sources pass consolidates it instead of rediscovering it.

THE DANGLING affected_by_hardware FINDING, recorded here because it shares this wave's scope: 9 unresolved of 609 values in data/flops/. Eight are ROCm LLVM target names - gfx950 (4), gfx942 (3), gfx1151 (1) - which name real silicon but have no accelerator record to point at, so they stay as budgeted debt and are NOT typos. The ninth is amd-ryzen-strix-point on data/flops/embedding, the only product-shaped dangling value in the directory. It was re-examined and deliberately left dangling: repointing it to amd-radeon-8060s-ryzen-ai-max would assert Strix HALO silicon (RDNA 3.5 iGPU, gfx1151) where the record means Strix POINT (gfx1150, the laptop APU), and deleting it would drop the only pointer to 'AMD APU designs' in the record's unified-memory argument. The repair it needs is an accelerator record for Strix Point, which is a data/accelerators/ write. The record's 2026-10-04 note already reached this conclusion and the 2026-10-10 re-examination confirmed it, which is the pattern the ledger (docs/26-known-defects.md) exists to stop being re-derived.

WHY A NEGATIVE RESULT IS THE RIGHT OUTPUT FOR TWO OF THE THREE CHECKS: an audit whose job is to confirm that a defect has not propagated into a second directory gets its value from being re-runnable, not from finding something new. The '%s' pass is the clearest case - 34 accelerator records were corrupted and repaired in wave 8, so the obvious next question is whether flops carried the same corruption, and the answer is no. Recording the measurement (0 records, checked for every format token) is what lets the next wave skip it instead of re-deriving it.

Cross-references