vLLM: MTP speculative decoding makes latency 76.5% WORSE on Qwen3-Next-80B-A3B-Instruct-FP8 at TP=4 (4x H100)

PER-REQUEST latency, measured with vLLM's OWN harness (`vllm bench latency`), so the number is produced by the engine under test rather than by an external timer. Fully pinned conditions, which is why this is a benchmark record and not an anecdote: vLLM commit ee59a7c61574485cf4ddbc6037ba557941be5c56 on main, model Qwen/Qwen3-Next-80B-A3B-Instruct-FP8, TP=4, max-model-len 262144, num_speculative_tokens 2, hardware 4x NVIDIA H100 80GB HBM3, driver 580.105.08, PyTorch 2.10.0+cu129, CUDA 12.9.86, Python 3.12.11, Intel Xeon Sapphire Rapids host. THREE ARMS, all reported with percentiles: baseline (no MTP, no prefix caching) avg 0.894 s / P50 0.895 / P90 0.901 / P99 0.907; with --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' avg 1.578 s (+76.5%) / P50 1.573 / P90 1.621 / P99 1.670; with MTP PLUS --enable-prefix-caching avg 1.629 s (+82.2%) / P50 1.628 / P90 1.673 / P99 1.691. No prompt length, output length or concurrency level is given in the source beyond max-model-len, so this is a latency-at-the-harness-default shape measurement.

Recorded deliberately as the counterweight to every 'speculative decoding: supported' feature flag in this repo, because support is not benefit. Two things a reader should take from it. (1) On this model and configuration speculation made things worse, and combining it with prefix caching - usually pitched as complementary - made it worse still (+82.2% vs +76.5%). (2) The regression is NOT yet root-caused: the reporter suspects 'copying num_accepted_tokens back to CPU before we do the mamba_postprocess' and the issue is still OPEN as of 2026-10-04, so this is a symptom measurement, not an explanation. The test protocol - same server, same model, same hardware, three arms, engine's own harness, full environment pasted - is the model for how this repo would want a latency claim recorded. No equivalent regression is recorded for other engines here, which reflects what was published, not an absence of the effect elsewhere. ACCELERATOR ID NOTE: the reporter wrote 'NVIDIA H100 80GB HBM3' via nvidia-smi, which is the SXM form factor; nvidia-h100-sxm is used, with nvidia-h100-nvl and nvidia-h100-pcie as the alternates the reporter's string does not distinguish between. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: per-request. ABSENT CONDITIONS (1 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - warmup: unrecoverable from ev-vllm-issue-35387-mtp-latency-regression - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-h100-sxm). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface. FORMAT_ID BACKFILL 2026-10-08 (w10-bench-formatid): format_id was null and is now 'fp8-e4m3'. EVIDENCE, quoted from this record's own text: methodology, in the list of fully pinned conditions: "model Qwen/Qwen3-Next-80B-A3B-Instruct-FP8". NO NUMBER CHANGED - value, metric and unit are untouched; this adds metadata, it does not re-measure. The value was NOT taken from a model name, from a sibling record, or from outside knowledge. Left null everywhere the record's own unit/methodology does not name a format for this run: 63 of the 150 records in data/benchmarks/ are still null and each one has its reason, because a null with a reason is correct and a guessed value is not.