RTX 5070: Llama 3.1 8B Q4_K_M decode tokens/sec

llama.cpp CUDA backend, batch size 1, 512 token prompt, 128 token output, Q4_K_M quantization, 12GB VRAM

RTX 5070 with 12GB GDDR7 at 672 GB/s. An 8B model at Q4_K_M requires ~5GB of VRAM, fitting comfortably. The 672 GB/s bandwidth yields high decode throughput for small models. The RTX 5070 is best suited for models up to ~13B at Q4_K_M or ~7B at Q8_0. LOAD-CONDITION AUDIT (bq- 2026-10-04) - STATUS: DECOMMISSIONED. THIS IS NOT A MEASUREMENT. WHY THIS RECORD IS POISONOUS RATHER THAN MERELY INCOMPLETE: it looks fully condition-complete. It states batch size 1, a 512-token prompt, a 128-token output and the precision - the exact four things the repo's iron rule demands - and it is still not a benchmark. Those four values are the generic defaults shared verbatim by all 11 records in this family; they describe no run because no run happened. WHAT THE SOURCES ACTUALLY CONTAIN: local-nvidia-rtx-5070-product-page and consumer-nvidia-rtx-blackwell-gpu-architecture-whitepaper (a product page and an architecture/spec document) - specification/datasheet documents containing NO decode-throughput measurement of any kind. The recorded figure (120 tok/s) appears nowhere in any cited source. Three of these notes state the reasoning explicitly as bandwidth-derived ('the N GB/s bandwidth yields ~M tokens/sec'), which is an ESTIMATE ARITHMETICALLY PROJECTED from a spec sheet, not a measurement of a model on a part. WHY THE FIGURES ARE ALSO PHYSICALLY IMPLAUSIBLE AS SET: local-rtx-5070-13b-q4-decode records 120 tok/s on a card with 672 GB/s of bandwidth, which is faster than the same family's own 85 tok/s figure for a 1792 GB/s RTX 5090 - a 2.7x bandwidth advantage yielding a LOWER number, and 120 tok/s exceeds what any 8B 4-bit model can pull at that bandwidth. MEASURED_BY: left as 'third_party' ONLY because schemas/benchmark.schema.json forbids null for measured_by and the audit was scoped not to edit schemas/. It is a placeholder, not a fact: nobody third-party measured this. Same for value, which is typed number and cannot be nulled here. Both should become null the moment the schema permits it; until then, status=deprecated plus this note is what stands between this record and a citation. CONFIDENCE 0.9 -> 0.2 and reproducible true -> false: reproducible was unsupportable for a figure with no run behind it. TO RECOVER: an actual llama.cpp or MLX-LM run on the named part, recording the engine build string, model revision, exact prompt/output token counts, whether prompts were VARIED or REPEATED (repeated prompts read 9.5-16.9x high - see ev-llamacpp-spec-bench-replay-inflates), the warmup policy, achieved clocks, and the raw tok/s counter. Until then this record must not be cited as a throughput number. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (local-nvidia-rtx-5070). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface. RE-AUDIT 2026-10-04, independent pass (benchmark throughput + latency audit): THE DEPRECATION IS CONFIRMED, on the primary basis rather than on the earlier internal-consistency argument. TWO INDEPENDENT NEGATIVE METHODS, both required before asserting a source lacks a measurement: (1) CORPUS-INTERNAL. Each recorded value was searched for in its own cited source record. 0 occurrences for all 12 citations across the eleven records. The cited records are 7 citations of consumer-nvidia-rtx-blackwell-gpu-architecture-whitepaper (kind: whitepaper) and 5 citations of spec-sheet records (consumer-nvidia-rtx-pro-6000-blackwell-datasheet, consumer-apple-macbook-pro-14-m4-max-tech-specs, local-apple-macbook-pro-m5-tech-specs, local-apple-mac-studio-m5-tech-specs, local-nvidia-rtx-5070-product-page). No benchmark kind is cited anywhere in this family. (2) EXTERNAL, the primary documents themselves, retrieved and keyword-searched: the NVIDIA RTX Blackwell GPU Architecture whitepaper PDF (V1.1) returned 0 occurrences of 'token', 'throughput', 'tok/s', 'decode' and 'inference'; the Apple MacBook Pro M4 Max tech-specs page and the Apple Mac Studio tech-specs page each returned 0 occurrences of 'token', 'throughput', 'tok/s' and 'inference' (their 6 'decode' hits are video-codec rows - 'Video decode engine', 'ProRes encode and decode engine' - not LLM decode). A spec sheet that cannot say 'token' cannot contain a decode-throughput measurement. THE EARLIER 'INVERSION' ARGUMENT WAS WITHDRAWN AS INVALID. It read local-rtx-5070-13b-q4-decode (120 tok/s on 672 GB/s) against local-rtx-5090-32b-q4-decode (85 tok/s on 1792 GB/s) and called the 2.7x bandwidth advantage producing a lower number proof of error. It is not: those two records run DIFFERENT MODELS. The 5070 record loads Llama 3.1 8B at Q4_K_M (~5 GB); the 5090 record loads Qwen3 32B at Q4_K_M (~18 GB). Measured against their own bandwidth roofs they are 89% and 85% respectively - mutually consistent. A slower-card/higher-tok/s comparison is only evidence when the two records hold the model, the quantization and the workload constant, and here nothing does. THE REPLACEMENT TEST IS THE HARD ROOFLINE, which does hold those constant: at batch 1 every decode step must read the model weights at least once, so tok/s cannot exceed bandwidth / weight bytes. Five of these eleven figures exceed their OWN roof - local-rtx-pro-6000-70b-q8-decode 55 tok/s against a 25.6 roof (215%), local-m5-max-70b-q4-decode 22 against 17.5 (125%), local-rtx-5080-32b-q4-decode 65 against 53.3 (122%), local-m4-max-70b-q4-decode 18 against 15.6 (115%), local-m5-ultra-70b-q4-decode 35 against 34.3 (102%). These are dense models, so no MoE correction applies. A Q8_0 70B cannot out-run a Q4_K_M 70B on two parts of identical bandwidth, because Q8_0 moves twice the bytes per token. INSTRUMENT VALIDATED BEFORE USE: the same roofline test was run against six known-good, reproducible third-party and vendor batch-1 decode records in this repo and passes 6 of 6 at 34-86% of roof - after correcting it for mixture-of-experts models, whose divisor is ACTIVE parameters rather than total. Uncorrected, it falsely flags Qwen3-30B-A3B and Qwen3.5-35B-A3B records at 1328% and 336% of roof; anyone reusing this test must apply that correction. THE REMAINING SINGLE-USER/MULTI-USER GAP IS REAL BUT MOOT: these eleven records state batch size 1 and never say per-user vs aggregate. At batch 1 the two coincide, so the omission would not make the number ambiguous - which is precisely the point that makes these records dangerous. The four load conditions are present, mutually consistent, and describe no run. VERDICT: status=deprecated is JUSTIFIED. The value, measured_by and the four load-condition fields cannot be nulled (schema requires a number and forbids null for measured_by), so the record is retained with status=deprecated, the unit prefixed UNSOURCED, DO NOT CITE, confidence 0.2 and reproducible false, plus this note.