inference/research
verifiedmedium · 0.65updated 2026-10-17

2x R9700 vLLM 0.27.1 Qwen3.8-27B native FP8 prefill at depth 8192, concurrency 8

2291.275 tok/s (effective follow-up prefill, concurrency 8, depth 8192)

Fields

id
benchmarks/llmb-r9700-2x-vllm0271-qwen38-27b-fp8-c8-d8k-prefill
Metric
prefill_tok_s
Value
2,291.275
Unit
tok/s (effective follow-up prefill, concurrency 8, depth 8192)
Model
Qwen3.8-27B native FP8
Methodology
Donato Capitella, local-llm-benchmarks.dev r9700-episode.csv, protocol deterministic-prime-extend-completions-v1, measured at the API endpoint of an OpenAI-completions server. Follow-up prompt ~2048 tokens, 256 generated tokens, temperature 0, mean of 2 repeats. Hardware: two AMD Radeon AI PRO R9700 (32 GB each), Ryzen 9900X3D, 64 GB host RAM. pp_tps is effective follow-up prefill (uncached tokens / client-observed time to first output, so recomputed-history time is included). tg_tps is generation after the first output token, so TTFT is excluded. Per request, not aggregate across users. Run run-8d7ca1e3bb280883. Engine vLLM 0.27.1, PyTorch 2.11.0+rocm7.14.0, ROCm 7.14, image digest f32d39b23389. Model qwen38-27b-native-fp8-017b9c7a, quant FP8, mode baseline (no speculative decoding). Concurrency 8, two GPUs. Target depth 8192: cached_depth_mean 7840, computed_prompt_mean 2400, recomputed_history_mean 352. PREFILL. stddev 3.342.
Measured by
third_party
Reproducible
yes
notes
Concurrency is 8, so these tok/s figures are the per-stream rate reported by the episode CSV, not a sum across the 8 streams. Deeper rows of this same run (24k and above) show cached_depth_mean collapsing to 0 and recomputed_history equal to the whole prefix; those are omitted because the prefix cache failed and the number is not a depth measurement.

Notes

Concurrency is 8, so these tok/s figures are the per-stream rate reported by the episode CSV, not a sum across the 8 streams. Deeper rows of this same run (24k and above) show cached_depth_mean collapsing to 0 and recomputed_history equal to the whole prefix; those are omitted because the prefix cache failed and the number is not a depth measurement.

Cross-references