inference/research
verifiedhigh · 0.95minorupdated 2026-10-10

Contested paper audit: cross-index verification resolves venue disputes but leaves preprints and schema gaps contested

[minor] measurement

Fields

id
gotchas/w11-contested-paper-audit-resolutions-and-unresolved-gaps
Class
measurement
Severity
minor
Concepts
papervenueprovenanceauditcontestedpmlrarxiviclrcrossref
Symptom
30 papers were marked status=contested after wave-9/10 venue and hardware sweeps. An audit across arXiv API metadata, PMLR, Crossref, and conference indexes resolved only two records to verified (deepspeed-moe at ICML 2022, whisper at ICML 2023), while 24 records remain contested because their cited preprint sources lack or contradict the published venue attribution, or their true venues are absent from the schema enum.
Root cause
The repo's provenance rule requires that status may change away from contested only when the finding names one venue that the paper's own sources agree on AND that venue is in schemas/paper.schema.json's venue enum. Most papers in data/papers/ cite arXiv preprint source records (pap-arch-*, pap-eff-*) whose abstracts and metadata describe preprints rather than conference proceedings, causing sources to disagree with conference index findings. Furthermore, venues like ACM FAccT, EACL, and demo tracks represent schema gaps absent from the venue enum.
Workaround
Record dated findings citing specific proceedings URLs and quotes in the paper's notes field while preserving both values. Resolve status to verified only where all declared sources and proceedings deposits agree (such as PMLR proceedings deposits matching the author comment). Report enum gaps (FAccT, EACL) for parent schema extension rather than forcing status changes.
notes
Audited 30 contested papers: 4 excluded as parent-fixed hardware citations (eagle-2, leviathan, s1, prm800k), 1 skipped as not-recorded in scratch scripts (speechgpt), 2 resolved to verified (deepspeed-moe, whisper), and 23 verified in notes but retained as contested due to source disagreements or schema enum gaps.

Notes

Audited 30 contested papers: 4 excluded as parent-fixed hardware citations (eagle-2, leviathan, s1, prm800k), 1 skipped as not-recorded in scratch scripts (speechgpt), 2 resolved to verified (deepspeed-moe, whisper), and 23 verified in notes but retained as contested due to source disagreements or schema enum gaps.

Cross-references

Affects (25)