inference/research
verifiedhigh · 0.85updated 2026-10-08

llama.cpp speculative decoding: a repeated-prompt benchmark protocol reads 9.5x-16.9x HIGH tok/s versus varied-prompt traffic on identical settings

13.4 value 13.4 with metric speedup_ratio: an INFLATION FACTOR (how many times too fast a benchmark reads when it replays one prompt instead of varying them; reported range 9.5x-16.9x), NOT 13.4 output tokens/s. BASELINE: the same llama.cpp b8400 build, same settings, 5 DISTINCT prompts of similar length instead of the same prompt 5 times. This was the most dangerous of the three, because 13.4 reads as a respectable per-GPU tok/s and would have been quoted as one.

Fields

id
benchmarks/ev-llamacpp-spec-bench-replay-inflates
Metric
speedup_ratio
Value
13.4
Unit
value 13.4 with metric speedup_ratio: an INFLATION FACTOR (how many times too fast a benchmark reads when it replays one prompt instead of varying them; reported range 9.5x-16.9x), NOT 13.4 output tokens/s. BASELINE: the same llama.cpp b8400 build, same settings, 5 DISTINCT prompts of similar length instead of the same prompt 5 times. This was the most dangerous of the three, because 13.4 reads as a respectable per-GPU tok/s and would have been quoted as one.
Model
Qwen3-8B Q4_K_M + Qwen3-0.6B Q8_0 draft model
Methodology
THIS RECORD'S VALUE IS THE FACTOR, NOT A THROUGHPUT NUMBER, and the value field records the inflation factor deliberately so it cannot be mistaken for a rate. Conditions: llama.cpp release b8400 (commit cf23ee2), llama-server on Windows official release binaries, command `llama-server -m Qwen3-8B-Q4_K_M.gguf -md Qwen3-0.6B-Q8_0.gguf -ngl 99 -ngld 99 -fa on --draft-max 48 -c 16384 -ub 2048 -b 2048 --parallel 1 --metrics`, Vulkan backend. A repeated-prompt protocol (same ~15+ token request sent 5 times, max_tokens 128-512) reads 9.5x-16.9x higher tok/s than 5 DISTINCT prompts of similar length at identical settings, with draft acceptance climbing toward 1.0 by run 3-5. The true steady-state acceptance for this model pair is ~0.6 and throughput is ~10x lower than the repeated-prompt reading. Mechanism, source-verified: common/speculative.cpp's draft() keeps a persistent drafter context (prompt_dft) across requests on a slot; when the cached token after the matched region equals id_last it returns the remainder of the cached text as the draft, up to n_max, WITHOUT running the draft model and WITHOUT applying params.p_min, because 'the confidence check exists only in the model-sampling loop further down'.
Measured by
third_party
Reproducible
yes
notes
The single most useful thing in this record is the author's one-line diagnosis, which is the repo's own evidence standard stated by a project maintainer-facing bug report: 'Any benchmark that warms up and times with the same prompt (a very common pattern) measures the replay path, not speculative decoding.' So the number a reader would most easily find published for llama.cpp speculative decoding is, on this code path, a measurement artifact. The production face of the same bug is worse than the benchmark face: with chat traffic sharing long prefixes, acceptance dropped 0.60 -> 0.12, drafted tokens per 512-token reply exploded ~440 -> ~1470, and end-to-end latency went ~3x (3.5 s -> 6.7 s) on b8407 with HIP verify + Vulkan draft, Qwen3-4B Q4_K_M + Qwen3-0.6B Q8_0, reproduced 2/2. The identical request on a fresh slot drafts normally, and 'plain or streaming traffic mixes that do NOT share long prefixes never trigger it, which makes the degradation intermittent and hard to attribute in production.' The reporter also documents the feature surface incidentally: --draft-max, --draft-p-min and --draft-p-split map to llama-server's --spec-draft-n-max / --spec-draft-p-min / --spec-draft-p-split. Reported on releases b8400/b8407 and stated present on current master at that time; the issue is CLOSED but this record does not claim a fix landed. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: MLPerf-style server scenario. ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - hardware: unrecoverable from ev-llamacpp-issue-26100-spec-benchmark-inflates-10x - no accelerator model is named anywhere in the source - input length: unrecoverable from ev-llamacpp-issue-26100-spec-benchmark-inflates-10x - would need the run's ISL or the prepared dataset's input-length profile None of these were filled in. A plausible value is worse than an absent one: it would be trusted. UNIT/METRIC DIRECTION CORRECTION 2026-10-04 (benchmark throughput + latency audit). The stored value is UNCHANGED; what was wrong is the `metric` enum, which labelled this record as a raw tokens/s measurement when the number is a dimensionless RATIO. Stored value is 13.4 with unit "x MULTIPLE of the true steady-state tok/s": the figure is how many times too fast a benchmark reads when it replays ONE prompt instead of varying them (reported range 9.5x-16.9x, 13.4 recorded as the midpoint). It is an inflation FACTOR, not a throughput. This is the error docs/07-benchmarking-methodology.md section 8 warns about from the other side: a reader filtering data/benchmarks/ by `metric: decode_tok_s` expects tokens per second, so a speedup_ratio filed under that key is not a wrong number, it is a number of the wrong KIND in the one field a capacity planner is most likely to filter on. CORRECT ENUM MEMBER: none of decode_tok_s / prefill_tok_s / tps_aggregate / tok_s_per_user describes a ratio, and schemas/benchmark.schema.json is out of scope for this audit so the record is left with an explicit `unit` and this note rather than silently reclassified. Anyone selecting benchmarks by metric MUST additionally check `unit` until a speedup_ratio member exists in the enum. METRIC CORRECTION 2026-10-05 (schema authority pass, agent w5-schema). The stored VALUE is UNCHANGED. What changed is `metric`, which labelled a DIMENSIONLESS RATIO as a dimensioned rate. schemas/benchmark.schema.json now has a speedup_ratio member, so the number no longer has to borrow a rate key it does not belong under, and every filter on data/benchmarks/ that selects tok/s no longer returns a ratio. Value and baseline unchanged; only the kind of quantity is now stated by the field that means it. FORMAT_ID BACKFILL 2026-10-08 (w10-bench-formatid): format_id was null and is now 'gguf-q4-k-m'. EVIDENCE, quoted from this record's own text: methodology command line: "llama-server -m Qwen3-8B-Q4_K_M.gguf -md Qwen3-0.6B-Q8_0.gguf". NO NUMBER CHANGED - value, metric and unit are untouched; this adds metadata, it does not re-measure. The value was NOT taken from a model name, from a sibling record, or from outside knowledge. Left null everywhere the record's own unit/methodology does not name a format for this run: 63 of the 150 records in data/benchmarks/ are still null and each one has its reason, because a null with a reason is correct and a guessed value is not.

Notes

The single most useful thing in this record is the author's one-line diagnosis, which is the repo's own evidence standard stated by a project maintainer-facing bug report: 'Any benchmark that warms up and times with the same prompt (a very common pattern) measures the replay path, not speculative decoding.' So the number a reader would most easily find published for llama.cpp speculative decoding is, on this code path, a measurement artifact. The production face of the same bug is worse than the benchmark face: with chat traffic sharing long prefixes, acceptance dropped 0.60 -> 0.12, drafted tokens per 512-token reply exploded ~440 -> ~1470, and end-to-end latency went ~3x (3.5 s -> 6.7 s) on b8407 with HIP verify + Vulkan draft, Qwen3-4B Q4_K_M + Qwen3-0.6B Q8_0, reproduced 2/2. The identical request on a fresh slot drafts normally, and 'plain or streaming traffic mixes that do NOT share long prefixes never trigger it, which makes the degradation intermittent and hard to attribute in production.' The reporter also documents the feature surface incidentally: --draft-max, --draft-p-min and --draft-p-split map to llama-server's --spec-draft-n-max / --spec-draft-p-min / --spec-draft-p-split. Reported on releases b8400/b8407 and stated present on current master at that time; the issue is CLOSED but this record does not claim a fix landed.

LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: MLPerf-style server scenario. ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - hardware: unrecoverable from ev-llamacpp-issue-26100-spec-benchmark-inflates-10x - no accelerator model is named anywhere in the source - input length: unrecoverable from ev-llamacpp-issue-26100-spec-benchmark-inflates-10x - would need the run's ISL or the prepared dataset's input-length profile None of these were filled in. A plausible value is worse than an absent one: it would be trusted.

UNIT/METRIC DIRECTION CORRECTION 2026-10-04 (benchmark throughput + latency audit). The stored value is UNCHANGED; what was wrong is the `metric` enum, which labelled this record as a raw tokens/s measurement when the number is a dimensionless RATIO. Stored value is 13.4 with unit "x MULTIPLE of the true steady-state tok/s": the figure is how many times too fast a benchmark reads when it replays ONE prompt instead of varying them (reported range 9.5x-16.9x, 13.4 recorded as the midpoint). It is an inflation FACTOR, not a throughput. This is the error docs/07-benchmarking-methodology.md section 8 warns about from the other side: a reader filtering data/benchmarks/ by `metric: decode_tok_s` expects tokens per second, so a speedup_ratio filed under that key is not a wrong number, it is a number of the wrong KIND in the one field a capacity planner is most likely to filter on. CORRECT ENUM MEMBER: none of decode_tok_s / prefill_tok_s / tps_aggregate / tok_s_per_user describes a ratio, and schemas/benchmark.schema.json is out of scope for this audit so the record is left with an explicit `unit` and this note rather than silently reclassified. Anyone selecting benchmarks by metric MUST additionally check `unit` until a speedup_ratio member exists in the enum.

METRIC CORRECTION 2026-10-05 (schema authority pass, agent w5-schema). The stored VALUE is UNCHANGED. What changed is `metric`, which labelled a DIMENSIONLESS RATIO as a dimensioned rate. schemas/benchmark.schema.json now has a speedup_ratio member, so the number no longer has to borrow a rate key it does not belong under, and every filter on data/benchmarks/ that selects tok/s no longer returns a ratio. Value and baseline unchanged; only the kind of quantity is now stated by the field that means it.

FORMAT_ID BACKFILL 2026-10-08 (w10-bench-formatid): format_id was null and is now 'gguf-q4-k-m'. EVIDENCE, quoted from this record's own text: methodology command line: "llama-server -m Qwen3-8B-Q4_K_M.gguf -md Qwen3-0.6B-Q8_0.gguf". NO NUMBER CHANGED - value, metric and unit are untouched; this adds metadata, it does not re-measure. The value was NOT taken from a model name, from a sibling record, or from outside knowledge. Left null everywhere the record's own unit/methodology does not name a format for this run: 63 of the 150 records in data/benchmarks/ are still null and each one has its reason, because a null with a reason is correct and a guessed value is not.

Cross-references

Engine (1)

Format (1)