2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MXFP4/FP8/GPTQ MTP-3 decode at depth 0
100.304 tok/s (per-request decode after first token, concurrency 1, depth 0, MTP-3)
Fields
id
benchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-mtp3-d0-decodeMetric
decode_tok_s
Value
100.304
Unit
tok/s (per-request decode after first token, concurrency 1, depth 0, MTP-3)
Model
Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger)
Methodology
Donato Capitella, local-llm-benchmarks.dev r9700-episode.csv, protocol deterministic-prime-extend-completions-v1, measured at the API endpoint of an OpenAI-completions server. Follow-up prompt ~2048 tokens, 256 generated tokens, temperature 0, mean of 2 repeats. Hardware: two AMD Radeon AI PRO R9700 (32 GB each), Ryzen 9900X3D, 64 GB host RAM. pp_tps is effective follow-up prefill (uncached tokens / client-observed time to first output, so recomputed-history time is included). tg_tps is generation after the first output token, so TTFT is excluded. Per request, not aggregate across users. Run run-4d514b0d7034191a, measured 2026-10-07. Engine tcclaviger vLLM 0.29.0.dev0+g2bdbbc8080, PyTorch 2.11.0+rocm10.0, ROCm runtime 7.15.26333, image digest c1f5f9da18b8. Model tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ revision 6fe3e7464c18 (120.33 GB): MXFP4 experts, FP8 attention and MTP, INT6 PLE. Mode tcclaviger-mtp-3: tensor_parallel_size 2, --enable-expert-offload --expert-offload-mem 36, --ple-nvme-offload --ple-cache-gb 4, --max-model-len 131072, --max-num-seqs 16, --max-num-batched-tokens 4096, chunked prefill, gpu_memory_utilization 0.95, kv fp8, prefix caching, speculative MTP with 3 draft tokens, cudagraph capture size 4. Concurrency 1. Target depth 0: cached_depth_mean 0, computed_prompt_mean 2048, recomputed_history_mean 0. DECODE of one request, not aggregate. stddev 2.783.
Measured by
third_party
Reproducible
yes
notes
Two physical cards; accelerator_ids lists the card twice. PLE rows live on NVMe (derived PLE file 41.60 GB), so part of the working set is not in VRAM. Not comparable with the single-R9700 llama.cpp rows already filed for this card: different engine, different model, speculative decoding on, and decode excludes TTFT.
Notes
Two physical cards; accelerator_ids lists the card twice. PLE rows live on NVMe (derived PLE file 41.60 GB), so part of the working set is not in VRAM. Not comparable with the single-R9700 llama.cpp rows already filed for this card: different engine, different model, speculative decoding on, and decode excludes TTFT.