inference/research

Benchmarks (4,667)

Measurements with their methodology attached. Check measured_by and reproducible before quoting a number — a vendor tok/s and a reproduced tok/s are not the same claim.

contested 68draft 72verified 4,516deprecated 11→ sources (2562)
GGUF Q5_0 GSM8K 'delta' vs f16 - quantization scoring ABOVE baseline
1.45 accuracy_delta_pp (GSM8K percentage points, quantized MINUS f16 - i.e. scored ABOVE baseline)Llama-3.1-8B-Instruct (Q5_0 GGUF) on ? via lm-eval-harnessbenchmarks/gguf-q5-0-gsm8k-accuracy-delta-llama3-1-8b
contestedlow · 0.60
GPT-5.6 Sol Ultrafast on Cerebras CS-4: 750 output tokens/sec
750 output tokens per secondGPT-5.6 Sol (Ultrafast mode) on cerebras-wse-3 via alt-cerebras-ultrafastbenchmarks/alt-cerebras-gpt56-ultrafast-tps
contestedlow · 0.60
Humanity's Last Exam: Cerebras GPT-5.6 Sol Ultrafast vs Claude Fable 5
7.0 x speedup in WALL-CLOCK TIME to complete the same 2500 Humanity's Last Exam questions. BASELINE: Claude Fable 5 with Claude Code at 78h27m; Cerebras GPT-5.6 Sol Ultrafast with Codex at 11h11m. This is a time-to-completion ratio between two DIFFERENT models on DIFFERENT hardware, which is why the record is status: contested and confidence 0.55 - a speedup_ratio is the honest key but it does not make the comparison sound.GPT-5.6 Sol Ultrafast vs Claude Fable 5 on cerebras-wse-3 via alt-cerebras-ultrafastbenchmarks/alt-cerebras-hle-benchmark
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 128k (Hardware Corner per-card page)
61.5 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-tg-128k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 16k (Hardware Corner per-card page)
137.6 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-tg-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 32k (Hardware Corner per-card page)
119.7 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-tg-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 4k (Hardware Corner per-card page)
160.3 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-tg-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 64k (Hardware Corner per-card page)
94.9 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-tg-64k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 128k (Hardware Corner per-card page)
936.4 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-pp-128k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 16k (Hardware Corner per-card page)
3440.1 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-pp-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 32k (Hardware Corner per-card page)
2521.3 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-pp-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 4k (Hardware Corner per-card page)
4805.4 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-pp-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 64k (Hardware Corner per-card page)
1666.5 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-gptoss-20b-pp-64k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B decode at 16k (Hardware Corner per-card page)
56.9 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-tg-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B decode at 32k (Hardware Corner per-card page)
42.1 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-tg-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B decode at 4k (Hardware Corner per-card page)
76.2 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-tg-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B decode at 64k (Hardware Corner per-card page)
28.2 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-tg-64k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B prefill at 16k (Hardware Corner per-card page)
1914.8 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-pp-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B prefill at 32k (Hardware Corner per-card page)
1232.1 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-pp-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B prefill at 4k (Hardware Corner per-card page)
2817.1 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-pp-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 14B prefill at 64k (Hardware Corner per-card page)
828.5 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-14b-pp-64k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 30B-A3B (MoE) decode at 16k (Hardware Corner per-card page)
121.9 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-30ba3b-tg-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 30B-A3B (MoE) decode at 32k (Hardware Corner per-card page)
92.0 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-30ba3b-tg-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 30B-A3B (MoE) decode at 4k (Hardware Corner per-card page)
166.9 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-30ba3b-tg-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 30B-A3B (MoE) prefill at 16k (Hardware Corner per-card page)
2205.6 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-30ba3b-pp-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 30B-A3B (MoE) prefill at 32k (Hardware Corner per-card page)
1483.9 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-30ba3b-pp-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 30B-A3B (MoE) prefill at 4k (Hardware Corner per-card page)
3441.0 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-30ba3b-pp-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 32B decode at 16k (Hardware Corner per-card page)
32.6 tok/s (token generation, per-request, batch 1)Qwen3 32B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-32b-tg-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 32B decode at 4k (Hardware Corner per-card page)
38.0 tok/s (token generation, per-request, batch 1)Qwen3 32B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-32b-tg-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 32B prefill at 16k (Hardware Corner per-card page)
862.7 tok/s (prompt processing, per-request, batch 1)Qwen3 32B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-32b-pp-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 32B prefill at 4k (Hardware Corner per-card page)
1238.8 tok/s (prompt processing, per-request, batch 1)Qwen3 32B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-32b-pp-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B decode at 128k (Hardware Corner per-card page)
29.0 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-tg-128k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B decode at 16k (Hardware Corner per-card page)
93.6 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-tg-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B decode at 32k (Hardware Corner per-card page)
71.8 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-tg-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B decode at 4k (Hardware Corner per-card page)
123.7 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-tg-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B decode at 64k (Hardware Corner per-card page)
49.0 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-tg-64k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B prefill at 128k (Hardware Corner per-card page)
612.4 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-pp-128k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B prefill at 16k (Hardware Corner per-card page)
2834.2 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-pp-16k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B prefill at 32k (Hardware Corner per-card page)
1868.0 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-pp-32k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B prefill at 4k (Hardware Corner per-card page)
4467.9 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-pp-4k
contestedlow · 0.55
RTX 3090 Ti llama.cpp: Qwen3 8B prefill at 64k (Hardware Corner per-card page)
1111.1 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc26-3090ti-qwen3-8b-pp-64k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 128k (Hardware Corner per-card page)
66.6 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-tg-128k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 16k (Hardware Corner per-card page)
133.1 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-tg-16k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 32k (Hardware Corner per-card page)
115.9 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-tg-32k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 4k (Hardware Corner per-card page)
156.0 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-tg-4k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) decode at 64k (Hardware Corner per-card page)
91.3 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-tg-64k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 128k (Hardware Corner per-card page)
1363.7 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-pp-128k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 16k (Hardware Corner per-card page)
4940.7 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-pp-16k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 32k (Hardware Corner per-card page)
3903.5 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-pp-32k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 4k (Hardware Corner per-card page)
6178.3 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-pp-4k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: gpt-oss 20B (MXFP4) prefill at 64k (Hardware Corner per-card page)
2644.9 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-gptoss-20b-pp-64k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 14B decode at 16k (Hardware Corner per-card page)
58.0 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-14b-tg-16k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 14B decode at 32k (Hardware Corner per-card page)
45.5 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-14b-tg-32k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 14B decode at 4k (Hardware Corner per-card page)
74.3 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-14b-tg-4k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 14B prefill at 16k (Hardware Corner per-card page)
2303.9 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-14b-pp-16k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 14B prefill at 32k (Hardware Corner per-card page)
1658.2 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-14b-pp-32k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 14B prefill at 4k (Hardware Corner per-card page)
3264.7 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-14b-pp-4k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B decode at 16k (Hardware Corner per-card page)
87.5 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-tg-16k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B decode at 32k (Hardware Corner per-card page)
63.3 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-tg-32k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B decode at 4k (Hardware Corner per-card page)
120.5 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-tg-4k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B decode at 64k (Hardware Corner per-card page)
40.4 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-tg-64k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B prefill at 16k (Hardware Corner per-card page)
3653.8 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-pp-16k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B prefill at 32k (Hardware Corner per-card page)
2269.0 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-pp-32k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B prefill at 4k (Hardware Corner per-card page)
5557.3 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-pp-4k
contestedlow · 0.55
RTX 5070 Ti llama.cpp: Qwen3 8B prefill at 64k (Hardware Corner per-card page)
1078.7 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc26-5070ti-qwen3-8b-pp-64k
contestedlow · 0.55
SmoothQuant W8A8 7-task average accuracy delta vs fp16, BLOOM-176B
-1.6 accuracy_delta_pp (average zero-shot accuracy percentage points, quantized minus FP16)BLOOM-176B on ? via smoothquant-referencebenchmarks/smoothquant-w8a8-zeroshot-accuracy-delta-bloom-176b
contestedmedium · 0.70
Trainium3 vs Trainium2 on Amazon Bedrock: output tokens per megawatt ratio
5.0 x higher output tokens per megawatt at similar latency per user (vendor claim)unspecified (vendor claim, workload not stated) on aws-trainium3 via alt-neuron-compilerbenchmarks/aws-trn3-vs-trn2-bedrock-tokens-per-mw-ratio
contestedlow · 0.50
Trainium3 vs Trainium2 vendor-claimed performance ratio (Trn3 vs Trn2 UltraServer)
4.4 x faster versus Trainium2 UltraServer baseline (vendor claim, workload unstated)unspecified (vendor claim, workload not stated) on aws-trainium3 via alt-neuron-compilerbenchmarks/aws-trn3-vs-trn2-perf-ratio-vendor
contestedlow · 0.55
4x Furiosa RNGD: Qwen3-32B about 1100 tok/s aggregate at batch 32 (SDK 2026-03-13)
1100 tok/s aggregate across the batch, 4-card serverQwen3-32B on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-4xrngd-qwen3-32b-b32-tps-aggregate
draftlow · 0.55
4x Furiosa RNGD: Qwen3-32B about 1500 tok/s aggregate at batch 64 (SDK 2026-03-13)
1500 tok/s aggregate across the batch, 4-card serverQwen3-32B on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-4xrngd-qwen3-32b-b64-tps-aggregate
draftlow · 0.55
4x Furiosa RNGD: Qwen3-32B TTFT 1.1-2.1 s at a 30 tok/s per user SLO
1600 ms, midpoint of a 1100-2100 ms vendor range, 4-card serverQwen3-32B on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-4xrngd-qwen3-32b-ttft-ms-slo30
draftlow · 0.50
8x V100 32GB (DGX-1) on 70B Q4-class, TP8 NVLink (reported range)
40.0 tok/s (reported range 30-50; midpoint recorded)70B Q4-class (GPTQ/AWQ via vLLM/TGI) on nvidia-v100-sxm2-32gb via vllmbenchmarks/comm-v100-x8-70b-q4-tp8-decode-est
draftlow · 0.50
A100 40GB SXM4 Llama-3-70B Q4_K_M decode (sibling-derived estimate)
18.0 tok/s (ESTIMATE - not measured)Llama-3-70B Q4_K_M (GGUF) on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/comm-a100-40gb-sxm4-est-llama3-70b-q4-decode
draftlow · 0.50
Adding MI50s over PCIe makes decode WORSE, not better: gpt-oss-120B MXFP4 tg8192 falls 71.2 -> 33.9 -> 21.3 -> 14.3 tok/s from 2 to 4 to 6 to 8 cards under -sm tensor
14.32 tok/sgpt-oss-120b MXFP4 MoE (59.02 GiB) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/comm-mi50-8x-gpt-oss-120b-mxfp4-decode-collapse
draftlow · 0.55
Community-reported decode throughput for a 70B Q4_K_M served by llama.cpp RPC across three separate machines holding 56GB of VRAM in total
4.38 tok/sLlama 3.3 70B (Q4_K_M GGUF) on ? via llama-cppbenchmarks/comm-4090-llamacpp-rpc-3-host-decode-tok-s
draftlow · 0.35
Community-reported vLLM decode throughput on RTX 5090 (sm_120) under WSL2 with CUDA graphs working, Qwen3-14B AWQ
140 tok/sQwen/Qwen3-14B-AWQ (awq_marlin quantization) on nvidia-rtx-5090 via vllmbenchmarks/comm-5090-wsl2-qwen3-14b-awq-decode-tok-s
draftlow · 0.40
DeepSeek_Simulator H800: DeepSeek-V3 decode 2585 tok/s/device (d=48 b_mla=256, single-batch overlap)
2585.0 tok/s per device (simulated decode aggregate, EP degree d=48, per-rank batch 256)deepseek-ai/DeepSeek-V3 on nvidia-h800-80gb via sglangbenchmarks/dsim-h800-deepseekv3-d48-b256-single-tps
draftlow · 0.40
DeepSeek_Simulator H800: DeepSeek-V3 decode 3121 tok/s/device (d=32 b_mla=128, two-microbatch overlap)
3121.0 tok/s per device (simulated decode aggregate, EP degree d=32, per-rank batch 128)deepseek-ai/DeepSeek-V3 on nvidia-h800-80gb via sglangbenchmarks/dsim-h800-deepseekv3-d32-b128-2mb-tps
draftlow · 0.40
DeepSeek_Simulator H800: DeepSeek-V3 decode 3121 tok/s/device (d=48 b_mla=128, two-microbatch overlap)
3121.0 tok/s per device (simulated decode aggregate, EP degree d=48, per-rank batch 128)deepseek-ai/DeepSeek-V3 on nvidia-h800-80gb via sglangbenchmarks/dsim-h800-deepseekv3-d48-b128-2mb-tps
draftlow · 0.40
Fully-populated EPYC Genoa 12ch on a 405B Q4 model (derived feasibility number)
1.75 tok/s (ESTIMATE - derived)Llama-3.1-405B-class Q4 (~230GB weights) on amd-epyc-9554-genoa via ik-llama-cppbenchmarks/comm-epyc-genoa-405b-q4-decode-est
draftlow · 0.45
Furiosa RNGD single card: about-10B model 2000-3000 tok/s aggregate (Hot Chips 2024 range)
2500 tok/s aggregate, midpoint of a 2000-3000 vendor rangeabout-10B-parameter LLM (checkpoint not named; Llama 3.1 named separately as runnable on one card) on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-rngd-10b-tps-aggregate-hotchips-2024
draftlow · 0.45
Furiosa RNGD single card: GPT-J 6B about 12 queries/s (Hot Chips 2024, pre-optimization)
12 queries/s (MLPerf-style, single card, not tokens/s)GPT-J 6B on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-rngd-gptj-6b-12qps-hotchips-2024
draftlow · 0.50
HOSTKEY: RTX PRO 5000 72GB Blackwell, deepseek-r1:14b, Ollama, avg decode
67.71 tok/sDeepSeek R1 14B on nvidia-rtx-pro-5000-72gb-blackwell via ollamabenchmarks/hostkey-rtxpro5000-72gb-deepseek-r1-14b-ollama-decode
draftlow · 0.55
HOSTKEY: RTX PRO 5000 72GB Blackwell, deepseek-r1:32b, Ollama, avg decode
32.89 tok/sDeepSeek R1 32B on nvidia-rtx-pro-5000-72gb-blackwell via ollamabenchmarks/hostkey-rtxpro5000-72gb-deepseek-r1-32b-ollama-decode
draftlow · 0.55
HOSTKEY: RTX PRO 5000 72GB Blackwell, gpt-oss:20b, Ollama, avg decode
149.38 tok/sgpt-oss 20B on nvidia-rtx-pro-5000-72gb-blackwell via ollamabenchmarks/hostkey-rtxpro5000-72gb-gpt-oss-20b-ollama-decode
draftlow · 0.55
HOSTKEY: RTX PRO 5000 72GB Blackwell, qwen3-coder-next, Ollama, avg decode
74.34 tok/sQwen3-Coder-Next on nvidia-rtx-pro-5000-72gb-blackwell via ollamabenchmarks/hostkey-rtxpro5000-72gb-qwen3-coder-next-ollama-decode
draftlow · 0.55
Intel Arc B580: DeepSeek-R1-Distill-Qwen 14B Q4_K_M decode tok/s, batch 1 (llama.cpp Vulkan)
35.6 tok/sDeepSeek-R1-Distill-Qwen 14B on intel-arc-b580 via llama-cppbenchmarks/local-arc-b580-r1-qwen-14b-q4-decode-b1
draftlow · 0.60
Intel Arc B580: DeepSeek-R1-Distill-Qwen 7B Q4_K_M decode tok/s, batch 1 (llama.cpp Vulkan)
67.2 tok/sDeepSeek-R1-Distill-Qwen 7B on intel-arc-b580 via llama-cppbenchmarks/local-arc-b580-r1-qwen-7b-q4-decode-b1
draftlow · 0.60
Level1Techs: 4x Intel Arc Pro B70, vLLM, Qwen3.5-27B, 50 concurrent reqs ~1024 tok ctx: ~369 tok/s aggregate output
369.0 tok/s output, aggregate over 50 concurrent requestsQwen3.5-27B on intel-arc-pro-b70 via vllmbenchmarks/l1t-4x-arc-pro-b70-qwen35-27b-vllm-50req-tps
draftlow · 0.50
Level1Techs: 4x Intel Arc Pro B70, vLLM, Qwen3.5-27B, 50 concurrent: avg TTFT ~11.4 s
11400.0 ms mean time-to-first-token under 50-request loadQwen3.5-27B on intel-arc-pro-b70 via vllmbenchmarks/l1t-4x-arc-pro-b70-qwen35-27b-vllm-50req-ttft
draftlow · 0.50
Level1Techs: 4x RTX 3090, vLLM, Qwen3.5-27B, 50 concurrent reqs ~1024 tok ctx: ~348 tok/s aggregate output
348.0 tok/s output, aggregate over 50 concurrent requestsQwen3.5-27B on nvidia-rtx-3090 via vllmbenchmarks/l1t-4x-rtx-3090-qwen35-27b-vllm-50req-tps
draftlow · 0.50
llama.cpp CUDA on NVIDIA A10G (A10 AWS variant): Llama-2-7B Q4_0 FA-on tg128 tok/s
111.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a10 via llama-cppbenchmarks/llamacpp-cuda-a10g-llama27b-q40-fa-tg128
draftlow · 0.60
llama.cpp CUDA on NVIDIA A10G: Llama-2-7B Q4_0 FA-on pp512 tok/s
2190.43 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a10 via llama-cppbenchmarks/llamacpp-cuda-a10g-llama27b-q40-fa-pp512
draftlow · 0.60
llama.cpp decode on 2x MI50 with -sm tensor: 71.2 tok/s for gpt-oss-120B MXFP4 (and layer split is FASTER at 81.7)
71.22 tok/sgpt-oss-120b MXFP4 MoE (59.02 GiB) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/comm-mi50-2x-gpt-oss-120b-mxfp4-tensor-split-decode
draftlow · 0.50
llama.cpp on Radeon AI PRO R9700 (gfx1201, RDNA4): Bonsai-27B Q1_0 decode throughput, batch 1
61.66 tok/s (tg64, mean of two interleaved runs: 61.97 and 61.34)Bonsai-27B (Q1_0, 3.53 GiB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-radeon-r9700-gfx1201-q1-0-decode-tok-s
draftlow · 0.50
llama.cpp under ROCm on 2x RX 7900 XTX (gfx1100): Qwen3.8-27B Q4_K_XL batch-1 tensor-split decode throughput
47.06 tok/s (batch-1 tensor-split decode across 2 cards)Qwen3.8-27B (Q4_K_XL, 17.545 GB) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rx7900xtx-gfx1100-qwen3-27b-q4-batch1-decode-tok-s
draftlow · 0.50
MLX-LM on MacBook Pro 14-inch M3 Pro 18GB: Qwen3-4B-Instruct-2507 4-bit generation throughput
44.62 tok/s (generation of 128 tokens, per-request, batch 1)Qwen3-4B-Instruct-2507 (MLX 4-bit, mlx-community) on apple-m3-max via mlx-lmbenchmarks/mlx-m3-pro-qwen3-4b-4bit-decode-tok-s
draftlow · 0.45
NVIDIA A100, llama.cpp Vulkan (2257758, coopmat2): Llama-2-7B Q4_0 FA tg128 t/s
170.56 tok/s per-request tg128 (FA); ±0.02Llama-2-7B Q4_0 GGUF on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/a100-vulkan-llama27b-q40-fa-tg128
draftlow · 0.55
NVIDIA A100, llama.cpp Vulkan (2257758, coopmat2): Llama-2-7B Q4_0 tg128 t/s
160.78 tok/s per-request tg128 (no FA); ±0.16Llama-2-7B Q4_0 GGUF on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/a100-vulkan-llama27b-q40-nofa-tg128
draftlow · 0.55
NVIDIA DGX-1 (8x V100 32GB) Llama-3.3-70B Q4 decode (estimated)
40.0 tok/s decode range 30-50 (est), tensor-parallel over NVLink planeLlama-3.3-70B Q4-class on nvidia-dgx-1-8x-v100, nvidia-v100-sxm2-32gb via vllmbenchmarks/comm-dgx1-8x-v100-llama33-70b-q4-decode-est
draftlow · 0.40
Qualcomm Cloud AI 100 Ultra (16-card server): LLaMA 70B ISL=128 OSL=128 batch-64 throughput tok/s (vendor)
5700 tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)LLaMA 70B on qualcomm-ai200 via vllmbenchmarks/qaic100ultra-llama70b-128-server-tps
draftlow · 0.45
Qualcomm Cloud AI 100 Ultra (16-card server): LLaMA 70B ISL=2048 OSL=2048 batch-64 throughput tok/s (vendor)
3500 tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)LLaMA 70B on qualcomm-ai200 via vllmbenchmarks/qaic100ultra-llama70b-2048-server-tps
draftlow · 0.45
Qualcomm Cloud AI 100 Ultra (16-card server): LLaMA 7B ISL=128 OSL=128 batch-64 throughput tok/s (vendor)
46000 tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)LLaMA 7B on qualcomm-ai200 via vllmbenchmarks/qaic100ultra-llama7b-128-server-tps
draftlow · 0.45
Qualcomm Cloud AI 100 Ultra (16-card server): LLaMA 7B ISL=2048 OSL=2048 batch-64 throughput tok/s (vendor)
7300 tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)LLaMA 7B on qualcomm-ai200 via vllmbenchmarks/qaic100ultra-llama7b-2048-server-tps
draftlow · 0.45
REPORTED (unverified, Reddit): 16x MI50 32GB serving DeepSeek-V3.2 AWQ 4-bit at 10 tok/s decode and 2000 tok/s prefill at 69k context
10.0 tok/sDeepSeek-V3.2 AWQ 4-bit on amd-radeon-instinct-mi50x via vllmbenchmarks/comm-mi50-16x-deepseek-v32-awq-reported
draftlow · 0.30
rocm-inference-bench: Llama-3-8B BF16 on MI250X, 42.3 tok/s (README table)
42.3 tok/s (batch-1, methodology undocumented)meta-llama/Llama-3-8B on amd-instinct-mi250x via hf-optimumbenchmarks/kuhaku-mi250x-llama3-8b-bf16-decode
draftlow · 0.35
rocm-inference-bench: Llama-3-8B FP8 on MI300X, 68.7 tok/s (README table)
68.7 tok/s (batch-1, methodology undocumented)meta-llama/Llama-3-8B on amd-instinct-mi300x via hf-optimumbenchmarks/kuhaku-mi300x-llama3-8b-fp8-decode
draftlow · 0.35
rocm-inference-bench: Mistral-7B BF16 on MI250X, 38.9 tok/s (README table)
38.9 tok/s (batch-1, methodology undocumented)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250x via hf-optimumbenchmarks/kuhaku-mi250x-mistral-7b-bf16-decode
draftlow · 0.35
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Llama 3.1 8B Instruct decode_tok_s
12.9 blended generation tok/s across LocalScore's 9 tasksLlama 3.1 8B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-llama31-8b-q4km-gen
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Llama 3.1 8B Instruct prefill_tok_s
99.0 blended prompt tok/s across LocalScore's 9 tasksLlama 3.1 8B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-llama31-8b-q4km-pp
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Llama 3.1 8B Instruct ttft_ms
13610.0 blended time-to-first-token, millisecondsLlama 3.1 8B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-llama31-8b-q4km-ttft
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Llama 3.2 1B Instruct decode_tok_s
67.0 blended generation tok/s across LocalScore's 9 tasksLlama 3.2 1B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-llama32-1b-q4km-gen
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Llama 3.2 1B Instruct prefill_tok_s
551.0 blended prompt tok/s across LocalScore's 9 tasksLlama 3.2 1B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-llama32-1b-q4km-pp
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Llama 3.2 1B Instruct ttft_ms
2530.0 blended time-to-first-token, millisecondsLlama 3.2 1B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-llama32-1b-q4km-ttft
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Qwen2.5 14B Instruct decode_tok_s
7.1 blended generation tok/s across LocalScore's 9 tasksQwen2.5 14B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-qwen25-14b-q4km-gen
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Qwen2.5 14B Instruct prefill_tok_s
51.0 blended prompt tok/s across LocalScore's 9 tasksQwen2.5 14B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-qwen25-14b-q4km-pp
draftlow · 0.40
Ryzen AI 9 HX 370 (Radeon 890M) LocalScore: Qwen2.5 14B Instruct ttft_ms
26300.0 blended time-to-first-token, millisecondsQwen2.5 14B Instruct on amd-ryzen-ai-9-hx-370 via llamafilebenchmarks/localscore-hx370-qwen25-14b-q4km-ttft
draftlow · 0.40
Ryzen AI 9 HX 370 dual-channel Qwen2.5-32B Q4 dense decode - the desktop trap
3.5 tok/sQwen2.5-32B Q4 (dense) on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/comm-ryzen-ai-hx370-qwen25-32b-q4-decode
draftlow · 0.55
SGLang DSPARK speculative decoding A/B on DeepSeek-V4.1-Flash, 8x RTX PRO 6000: 76.5-78.9 vs 36.4-36.5 tok/s on prose at block_size=2
77.7 tok/s decode, single stream, DSPARK block_size=2, prose prompts, mid-point of the 76.5-78.9 range the source reportsDeepSeek-V4.1-Flash on nvidia-rtx-pro-6000-blackwell via sglangbenchmarks/ev-sglang-dspark-spec-decoding-2x-decode-ab
draftmedium · 0.80
SGLang on 8x RTX PRO 6000: prefix-cache hit on a 112K-token shared prefix serves in 0.34 s, ~39x the cold prefill rate
325000.0 tok/s prefill equivalent on a cache HIT (112,000-token prefix served from cache in 0.34 s)DeepSeek-V4.1-Flash on nvidia-rtx-pro-6000-blackwell via sglangbenchmarks/ev-sglang-dsv41-prefix-cache-hit-prefill-speedup
draftmedium · 0.70
SGLang serving DeepSeek-V4.1-Flash in production on 8x RTX PRO 6000: 128K cold prefill throughput at chunked_prefill_size=2048
6853.0 tok/s prefill (single stream, tokens counted from the response `usage` field, not SSE chunk counts)DeepSeek-V4.1-Flash (V4.1 vision path, MXFP4 experts) on nvidia-rtx-pro-6000-blackwell via sglangbenchmarks/ev-sglang-dsv41-production-128k-prefill-tps
draftmedium · 0.80
SGLang serving DeepSeek-V4.1-Flash in production on 8x RTX PRO 6000: single-stream decode throughput, prose vs structured output
93.9 tok/s decode, single stream (one user at a time)DeepSeek-V4.1-Flash on nvidia-rtx-pro-6000-blackwell via sglangbenchmarks/ev-sglang-dsv41-production-decode-tok-s-per-user
draftmedium · 0.75
Sharding model weights across cards instead of keeping them host-resident more than doubles both prefill and decode on 4x MI50, at ~7 GB extra VRAM per card
501.2 tok/sQwen3.8-Flash-Next UD-Q4_K_XL on amd-radeon-instinct-mi50x via llama-cppbenchmarks/comm-mi50-4x-qwen38-pleshard-shard-vs-host
draftlow · 0.60
SiliconScore community row: Apple M3 Ultra, MLX, GLM-4.5-Air 4bit MLX, decode 54.0 t/s
54.0 tok/s avg decode (single-request)GLM-4.5-Air (4bit MLX) on apple-m3-ultra via apple-mlxbenchmarks/ss-m3-ultra-glm45-air-4bit-decode
draftlow · 0.40
SiliconScore community row: Apple M3 Ultra, MLX, GLM-5 4bit MLX, decode 16.7 t/s
16.7 tok/s avg decode (single-request)GLM-5 (4bit MLX) on apple-m3-ultra via apple-mlxbenchmarks/ss-m3-ultra-glm5-4bit-decode
draftlow · 0.40
SiliconScore community row: Apple M3 Ultra, MLX, Qwen3-Coder-Next 4bit MLX, decode 74.0 t/s
74.0 tok/s avg decode (single-request)Qwen3-Coder-Next (4bit MLX) on apple-m3-ultra via apple-mlxbenchmarks/ss-m3-ultra-qwen3-coder-next-4bit-decode
draftlow · 0.40
SiliconScore community row: Apple M3 Ultra, MLX, Qwen3.5-122B-A10B 8bit MLX, decode 43.0 t/s
43.0 tok/s avg decode (single-request)Qwen3.5-122B-A10B (8bit MLX) on apple-m3-ultra via apple-mlxbenchmarks/ss-m3-ultra-qwen35-122b-a10b-8bit-decode
draftlow · 0.40
SiliconScore community row: Apple M3 Ultra, MLX, Qwen3.5-122B-A10B MXFP4 MLX, decode 57.0 t/s
57.0 tok/s avg decode (single-request)Qwen3.5-122B-A10B (MXFP4 MLX) on apple-m3-ultra via apple-mlxbenchmarks/ss-m3-ultra-qwen35-122b-a10b-mxfp4-decode
draftlow · 0.40
SiliconScore community row: Apple M3 Ultra, MLX, Qwen3.5-397B-A17B q4.1bit MLX, decode 40.2 t/s
40.2 tok/s avg decode (single-request)Qwen3.5-397B-A17B (q4.1bit MLX) on apple-m3-ultra via apple-mlxbenchmarks/ss-m3-ultra-qwen35-397b-a17b-41bit-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, DeepSeek R1 Distill Llama 70B Q4_K_M, MLX, avg decode
16.0 tok/sDeepSeek R1 Distill Llama 70B on apple-m4-ultra via mlx-lmbenchmarks/siliconscore-m4ultra-deepseek-r1-distill-llama-70b-q4km-mlx-lm-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, gpt-oss 120B Q4_K_M, MLX, avg decode
10.0 tok/sgpt-oss 120B on apple-m4-ultra via mlx-lmbenchmarks/siliconscore-m4ultra-gpt-oss-120b-q4km-mlx-lm-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, Llama 3.3 70B Q4_K_M, MLX, avg decode
18.0 tok/sLlama 3.3 70B on apple-m4-ultra via mlx-lmbenchmarks/siliconscore-m4ultra-llama-33-70b-q4km-mlx-lm-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, Llama 4 Scout 17B-16E Q4_K_M, MLX, avg decode
30.0 tok/sLlama 4 Scout 17B-16E on apple-m4-ultra via mlx-lmbenchmarks/siliconscore-m4ultra-llama-4-scout-17b-16e-q4km-mlx-lm-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, Mistral Small 4 119B Q4_K_M, MLX, avg decode
45.0 tok/sMistral Small 4 119B on apple-m4-ultra via mlx-lmbenchmarks/siliconscore-m4ultra-mistral-small-4-119b-q4km-mlx-lm-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, Qwen 2.5 72B Q4_K_M, Ollama, avg decode
15.0 tok/sQwen 2.5 72B on apple-m4-ultra via ollamabenchmarks/siliconscore-m4ultra-qwen-25-72b-q4km-ollama-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, Qwen 3 235B-A22B Q4_K_M, MLX, avg decode
22.0 tok/sQwen 3 235B-A22B on apple-m4-ultra via mlx-lmbenchmarks/siliconscore-m4ultra-qwen-3-235b-a22b-q4km-mlx-lm-decode
draftlow · 0.40
siliconscore: M4 Ultra 192GB, Qwen 3 32B Q4_K_M, Ollama, avg decode
32.0 tok/sQwen 3 32B on apple-m4-ultra via ollamabenchmarks/siliconscore-m4ultra-qwen-3-32b-q4km-ollama-decode
draftlow · 0.40
Tesla P40 Llama-3.1-8B Q4_K_M decode (modelled)
35.0 tok/s decode (modelled)Llama-3.1-8B on nvidia-tesla-p40 via llama-cppbenchmarks/comm-p40-llama31-8b-q4-tg-modelled
draftlow · 0.50
vLLM bring-up of DeepSeek-V4.1-Flash on 8x RTX PRO 6000: 4584 tok/s prefill on the pure-torch eager fallback path at 1M context
4584.0 tok/s prefill, eager mode, pure-torch logits path (NOT the optimized CUDA-graph production path)DeepSeek-V4.1-Flash (769B total, MXFP4 experts, Engram memory) on nvidia-rtx-pro-6000-blackwell via vllmbenchmarks/ev-vllm-dsv41-production-128k-prefill-tps-eager
draftmedium · 0.75
W8A8 int8 quantization on MI50 is 4-6x SLOWER than fp16/GPTQ, not faster, because it falls to a TritonScaledMMLinearKernel path and never engages the hardware INT8 path
10.81 tok/s (total token throughput)Qwen3-0.6B.w8a8 on amd-radeon-instinct-mi50x via vllmbenchmarks/comm-gfx906-w8a8-int8-slower-than-gptq
draftlow · 0.60
2x AMD Instinct MI100, llama.cpp: Llama-2-70B Q6_K decode tok/s (community-measured)
8.32 decode tok/s (single-stream, 2x MI100 combined)Llama-2-70B Q6_K (GGUF) on amd-instinct-mi100 via llama-cppbenchmarks/comm-mi100-2x-llama2-70b-q6k-decode
verifiedmedium · 0.80
2x MI100 Llama-2-70B Q6_K token-gen
8.32 tok/s decode across 2 cardsLlama-2-70B on amd-instinct-mi100 via llama-cppbenchmarks/comm-mi100-2x-llama2-70b-q6k-tg
verifiedmedium · 0.75
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MTP-3 aggregate decode, c1-distinct
100.75 tok/s (aggregate across 1 concurrent requests, short prompt, 2048 output)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-sustained-c1-distinct-tg
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MTP-3 aggregate decode, c2-distinct
81.057 tok/s (aggregate across 2 concurrent requests, short prompt, 2048 output)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-sustained-c2-distinct-tg
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MTP-3 aggregate decode, c2-shared
99.263 tok/s (aggregate across 2 concurrent requests, short prompt, 2048 output)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-sustained-c2-shared-tg
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MTP-3 aggregate decode, c4-distinct
94.933 tok/s (aggregate across 4 concurrent requests, short prompt, 2048 output)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-sustained-c4-distinct-tg
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MTP-3 aggregate decode, c4-shared
121.014 tok/s (aggregate across 4 concurrent requests, short prompt, 2048 output)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-sustained-c4-shared-tg
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MXFP4/FP8/GPTQ MTP-3 decode at depth 0
100.304 tok/s (per-request decode after first token, concurrency 1, depth 0, MTP-3)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-mtp3-d0-decode
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MXFP4/FP8/GPTQ MTP-3 decode at depth 65536
106.684 tok/s (per-request decode after first token, concurrency 1, depth 65536, MTP-3)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-mtp3-d64k-decode
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MXFP4/FP8/GPTQ MTP-3 prefill at depth 0
2676.443 tok/s (effective follow-up prefill, concurrency 1, depth 0)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-mtp3-d0-prefill
verifiedmedium · 0.65
2x R9700 tcclaviger vLLM Qwen3.8-Flash-Next MXFP4/FP8/GPTQ MTP-3 prefill at depth 65536
1116.002 tok/s (effective follow-up prefill, concurrency 1, depth 65536)Qwen3.8-Flash-Next MXFP4-FP8-GPTQ (tcclaviger) on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-tcclaviger-qwen38-flash-next-mtp3-d64k-prefill
verifiedmedium · 0.65
2x R9700 vLLM 0.27.1 Qwen3.8-27B native FP8 decode at depth 0, concurrency 8
135.853 tok/s (per-stream decode after first token, concurrency 8, depth 0)Qwen3.8-27B native FP8 on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-vllm0271-qwen38-27b-fp8-c8-d0-decode
verifiedmedium · 0.65
2x R9700 vLLM 0.27.1 Qwen3.8-27B native FP8 decode at depth 8192, concurrency 8
116.822 tok/s (per-stream decode after first token, concurrency 8, depth 8192)Qwen3.8-27B native FP8 on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-vllm0271-qwen38-27b-fp8-c8-d8k-decode
verifiedmedium · 0.65
2x R9700 vLLM 0.27.1 Qwen3.8-27B native FP8 prefill at depth 0, concurrency 8
3411.146 tok/s (effective follow-up prefill, concurrency 8, depth 0)Qwen3.8-27B native FP8 on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-vllm0271-qwen38-27b-fp8-c8-d0-prefill
verifiedmedium · 0.65
2x R9700 vLLM 0.27.1 Qwen3.8-27B native FP8 prefill at depth 8192, concurrency 8
2291.275 tok/s (effective follow-up prefill, concurrency 8, depth 8192)Qwen3.8-27B native FP8 on amd-radeon-ai-pro-r9700, amd-radeon-ai-pro-r9700 via vllmbenchmarks/llmb-r9700-2x-vllm0271-qwen38-27b-fp8-c8-d8k-prefill
verifiedmedium · 0.65
2x RX 7900 XTX llama.cpp ROCm 7.1.1: Llama 3.1 70B Q4_K_M FA-on tg128
13.37 tok/s (tg128, per-request, batch 1, split across 2 GPUs)Llama 3.1 70B Q4_K_M (39.59 GiB, 70.55B) on amd-radeon-rx-7900-xtx, amd-radeon-rx-7900-xtx via llama-cppbenchmarks/rocm711-2x-rx7900xtx-llama31-70b-q4km-fa-tg128
verifiedlow · 0.60
4x Furiosa RNGD: EXAONE 3.5 32B, batch 1, 50 tok/s at 32K context
50 tok/s, per-request, batch 1, 4-card tensor parallelEXAONE 3.5 32B on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-4xrngd-exaone35-32b-32k-b1-decode
verifiedmedium · 0.70
4x Furiosa RNGD: EXAONE 3.5 32B, batch 1, 60 tok/s at 4K context
60 tok/s, per-request, batch 1, 4-card tensor parallelEXAONE 3.5 32B on furiosaai-rngd-renegade via vllmbenchmarks/furiosa-4xrngd-exaone35-32b-4k-b1-decode
verifiedmedium · 0.70
4x RTX 3090 vLLM Qwen3.5-27B 50-request aggregate (Level1Techs reference run)
348.0 tok/s aggregate across 50 concurrent requestsQwen3.5-27B on nvidia-rtx-3090 via vllmbenchmarks/comm-4x-rtx-3090-vllm-qwen35-27b-agg
verifiedmedium · 0.75
8x H100 node under real AI training draws only 76% of its 10.2 kW TDP rating; calibrated model reaches 11.4% MAPE versus 27-37% for TDP-based estimates
76.0 percent of rated 10.2 kW node TDP actually drawn under compute-intensive training load? on nvidia-h100-sxm via vllmbenchmarks/h100-8gpu-node-loaded-76-percent-of-tdp-measured
verifiedhigh · 0.88
8x Intel Gaudi 2 HL-225B, MLPerf v4.0 offline: Llama-2-70B FP8 aggregate output tok/s
8034.85 output tok/s AGGREGATE across 8x Gaudi 2 (batch-1024 class; ~1004 tok/s per module)Llama-2-70B FP8 on intel-gaudi2 via vllm-gaudibenchmarks/comm-gaudi2-8x-llama2-70b-fp8-mlperf
verifiedmedium · 0.80
8x NVIDIA V100 32GB SXM2 (DGX-1 NVLink plane): 70B Q4-class decode tok/s (bandwidth-scaled estimate)
40.0 decode tok/s (8-way tensor parallel, NVLink, estimated)Llama-3.3-70B-class (dense) Q4 on nvidia-v100-sxm2-32gb via vllmbenchmarks/comm-dgx1-8x-v100-llama-70b-q4-decode-est
verifiedlow · 0.55
8xH200 SXM-141GB (MaxQ), MLPerf v4.1 llama2-70b Offline RANGING run: 0 tok/s at 2411.2 W mean node draw - idle floor is 41.7% of the 5806.9 W loaded plateau
0 output tokens/s = ZERO, at 2411.2 W mean whole-node draw (first 20 samples 2411.18 W, last 20 samples 2455.93 W of the 3801-sample ranging trace) - deliberately a zero-value record: the entire point is that watts are non-zero when throughput is zero, and this is what breaks naive cost-per-token arithmeticLlama 2 70B (MLPerf 'llama2-70b-99.9') - model named for provenance only; at idle the engine emits nothing on nvidia-h200-sxm via tensorrt-llmbenchmarks/h200-8x-mlperf-v4-1-maxq-llama2-70b-offline-idle-node-draw
verifiedmedium · 0.82
8xH200 SXM-141GB (MaxQ), MLPerf v4.1 llama2-70b Offline: 25262.1 output tok/s at 5806.9 W mean node draw = 0.2299 J/output token
25262.1 output tokens/s (MLPerf 'Tokens per second', aggregate) AT 5806.9 W MEAN WHOLE-NODE DRAW - i.e. 0.2299 J/output token; metric enum has no energy metric so the watts travel in the unit, see notesLlama 2 70B (MLPerf 'llama2-70b-99.9') on nvidia-h200-sxm via tensorrt-llmbenchmarks/h200-8x-mlperf-v4-1-maxq-llama2-70b-offline-energy-per-token
verifiedhigh · 0.93
8xH200 SXM-141GB (MaxQ), MLPerf v4.1 llama2-70b Server: 23113.14 completed tok/s at 5767.2 W mean = 0.2495 J/output token
23113.14 completed tokens/s (MLPerf 'Completed tokens per second') AT 5767.2 W MEAN WHOLE-NODE DRAW - i.e. 0.2495 J/output token; watts in the unit because the metric enum has no energy metricLlama 2 70B (MLPerf 'llama2-70b-99.9') on nvidia-h200-sxm via tensorrt-llmbenchmarks/h200-8x-mlperf-v4-1-maxq-llama2-70b-server-energy-per-token
verifiedhigh · 0.93
8xH200 SXM-141GB (MaxQ), MLPerf v4.1 mixtral-8x7b Offline: 48987.9 output tok/s at 5854.3 W mean = 0.1195 J/output token
48987.9 output tokens/s (MLPerf 'Tokens per second', aggregate) AT 5854.3 W MEAN WHOLE-NODE DRAW - i.e. 0.1195 J/output token; watts in the unit because the metric enum has no energy metricMixtral 8x7B (MLPerf 'mixtral-8x7b') on nvidia-h200-sxm via tensorrt-llmbenchmarks/h200-8x-mlperf-v4-1-maxq-mixtral-8x7b-offline-energy-per-token
verifiedhigh · 0.92
8xH200 SXM-141GB, MLPerf v4.1 llama2-70b Offline throughput under MaxP (no power measurement) vs MaxQ: 31302.7 vs 25262.1 tok/s
31302.7 output tokens/s (MLPerf 'Tokens per second', aggregate), non-power MaxP submission, NO POWER FIGURE PUBLISHED FOR THIS RUN - the comparison value is 25262.1 tok/s at 5806.9 W for the MaxQ submission of the same benchmarkLlama 2 70B (MLPerf 'llama2-70b-99.9') on nvidia-h200-sxm via tensorrt-llmbenchmarks/h200-8x-mlperf-v4-1-maxp-llama2-70b-offline-vs-maxq
verifiedhigh · 0.85
A100 40GB PCIe gemma-4-12B-it UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
86.4 tok/s decode (tg128, per-request batch-1); top prefill 2930 tok/sgemma-4-12B-it on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-12b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe gemma-4-26B-A4B-it (MoE, ~4B active) UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
128 tok/s decode (tg128, per-request batch-1); top prefill 4115 tok/sgemma-4-26B-A4B-it (MoE, ~4B active) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-26b-a4b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe gemma-4-31B-it UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
41.2 tok/s decode (tg128, per-request batch-1); top prefill 1197 tok/sgemma-4-31B-it on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-31b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe gpt-oss-20b (MoE) F16 decode (llamabench, llama.cpp b9935)
187 tok/s decode (tg128, per-request batch-1); top prefill 5568 tok/sgpt-oss-20b (MoE) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gpt-oss-20b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe llama.cpp b9935: gemma-4-12B-it UD-Q4_K_XL decode
86.4 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 2930 t/sgemma-4-12B-it UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-12b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gemma-4-12B-it UD-Q4_K_XL prefill
2930 tok/s prefill (ctx 0.5k, FA on); decode 86.4 t/s tg128gemma-4-12B-it UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-12b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gemma-4-26B-A4B-it UD-Q4_K_XL (MoE) decode
128 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 4115 t/sgemma-4-26B-A4B-it UD-Q4_K_XL (MoE) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-26b-a4b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gemma-4-26B-A4B-it UD-Q4_K_XL (MoE) prefill
4115 tok/s prefill (ctx 0.5k, FA on); decode 128 t/s tg128gemma-4-26B-A4B-it UD-Q4_K_XL (MoE) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-26b-a4b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gemma-4-31B-it UD-Q4_K_XL decode
41.2 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 1197 t/sgemma-4-31B-it UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-31b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gemma-4-31B-it UD-Q4_K_XL prefill
1197 tok/s prefill (ctx 0.5k, FA on); decode 41.2 t/s tg128gemma-4-31B-it UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gemma4-31b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gpt-oss-20B F16 decode
187 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 5568 t/sgpt-oss-20B F16 on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gpt-oss-20b-f16-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: gpt-oss-20B F16 prefill
5568 tok/s prefill (ctx 0.5k, FA on); decode 187 t/s tg128gpt-oss-20B F16 on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-gpt-oss-20b-f16-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Ornith-1.0-35B Q4_K_M decode
148 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 3321 t/sOrnith-1.0-35B Q4_K_M on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-ornith10-35b-q4km-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Ornith-1.0-35B Q4_K_M prefill
3321 tok/s prefill (ctx 0.5k, FA on); decode 148 t/s tg128Ornith-1.0-35B Q4_K_M on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-ornith10-35b-q4km-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.5-4B UD-Q4_K_XL decode
166 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 5306 t/sQwen3.5-4B UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen35-4b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.5-4B UD-Q4_K_XL prefill
5306 tok/s prefill (ctx 0.5k, FA on); decode 166 t/s tg128Qwen3.5-4B UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen35-4b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.5-9B UD-Q4_K_XL decode
125 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 4009 t/sQwen3.5-9B UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen35-9b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.5-9B UD-Q4_K_XL prefill
4009 tok/s prefill (ctx 0.5k, FA on); decode 125 t/s tg128Qwen3.5-9B UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen35-9b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.6-27B UD-Q4_K_XL decode
43.9 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 1262 t/sQwen3.6-27B UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen36-27b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.6-27B UD-Q4_K_XL prefill
1262 tok/s prefill (ctx 0.5k, FA on); decode 43.9 t/s tg128Qwen3.6-27B UD-Q4_K_XL on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen36-27b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.6-35B-A3B UD-Q4_K_XL (MoE) decode
136 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 3413 t/sQwen3.6-35B-A3B UD-Q4_K_XL (MoE) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen36-35b-a3b-q4kxl-tg128
verifiedlow · 0.55
A100 40GB PCIe llama.cpp b9935: Qwen3.6-35B-A3B UD-Q4_K_XL (MoE) prefill
3413 tok/s prefill (ctx 0.5k, FA on); decode 136 t/s tg128Qwen3.6-35B-A3B UD-Q4_K_XL (MoE) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen36-35b-a3b-q4kxl-pp
verifiedlow · 0.55
A100 40GB PCIe MiniCPM5-1B F16 decode (llamabench, llama.cpp b9935)
361 tok/s decode (tg128, per-request batch-1); top prefill 24868 tok/sMiniCPM5-1B on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-minicpm5-1b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe ornith-1.0-35b Q4_K_M decode (llamabench, llama.cpp b9935)
148 tok/s decode (tg128, per-request batch-1); top prefill 3321 tok/sornith-1.0-35b on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-ornith-1-0-35b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe Qwen3.5-4B UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
166 tok/s decode (tg128, per-request batch-1); top prefill 5306 tok/sQwen3.5-4B on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen35-4b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe Qwen3.5-9B UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
125 tok/s decode (tg128, per-request batch-1); top prefill 4009 tok/sQwen3.5-9B on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen35-9b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe Qwen3.6-27B UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
43.9 tok/s decode (tg128, per-request batch-1); top prefill 1262 tok/sQwen3.6-27B on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen36-27b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe Qwen3.6-35B-A3B (MoE, ~3.5B active) UD-Q4_K_XL decode (llamabench, llama.cpp b9935)
136 tok/s decode (tg128, per-request batch-1); top prefill 3413 tok/sQwen3.6-35B-A3B (MoE, ~3.5B active) on nvidia-a100-40gb-pcie via llama-cppbenchmarks/llamabench-a100pcie40-qwen36-35b-a3b-decode-tg128
verifiedmedium · 0.65
A100 40GB PCIe vLLM: google/gemma-3-12b-it aggregate output throughput, 50 concurrent
2091.15 tok/s (aggregate output tokens, 50 concurrent)gemma-3-12b-it on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-gemma-3-12b-it-c50-out
verifiedlow · 0.60
A100 40GB PCIe vLLM: google/gemma-3-12b-it median TTFT, 50 concurrent
574.88 ms (median time to first token, 50 concurrent)gemma-3-12b-it on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-gemma-3-12b-it-c50-ttft
verifiedlow · 0.60
A100 40GB PCIe vLLM: google/gemma-3-4b-it aggregate output throughput, 50 concurrent
3385.72 tok/s (aggregate output tokens, 50 concurrent)gemma-3-4b-it on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-gemma-3-4b-it-c50-out
verifiedlow · 0.60
A100 40GB PCIe vLLM: google/gemma-3-4b-it median TTFT, 50 concurrent
401.91 ms (median time to first token, 50 concurrent)gemma-3-4b-it on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-gemma-3-4b-it-c50-ttft
verifiedlow · 0.60
A100 40GB PCIe vLLM: Qwen/Qwen2.5-14B-Instruct aggregate output throughput, 50 concurrent
420.31 tok/s (aggregate output tokens, 50 concurrent)qwen2.5-14b-instruct on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-qwen25-14b-instruct-c50-out
verifiedlow · 0.60
A100 40GB PCIe vLLM: Qwen/Qwen2.5-14B-Instruct median TTFT, 50 concurrent
333.83 ms (median time to first token, 50 concurrent)qwen2.5-14b-instruct on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-qwen25-14b-instruct-c50-ttft
verifiedlow · 0.60
A100 40GB PCIe vLLM: Qwen/Qwen2.5-7B-Instruct aggregate output throughput, 50 concurrent
2225.26 tok/s (aggregate output tokens, 50 concurrent)qwen2.5-7b-instruct on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-qwen25-7b-instruct-c50-out
verifiedlow · 0.60
A100 40GB PCIe vLLM: Qwen/Qwen2.5-7B-Instruct median TTFT, 50 concurrent
615.61 ms (median time to first token, 50 concurrent)qwen2.5-7b-instruct on nvidia-a100-40gb-pcie via vllmbenchmarks/dm-a100-40gb-vllm-qwen25-7b-instruct-c50-ttft
verifiedlow · 0.60
A100 40GB SXM4 llama.cpp b10808: Ornith-1.5-35B-A3B Q4_K_M (MoE, 3B active) decode
148 tok/s decode (tg128, batch-1, ctx 0.5k, FA on, kv q8_0); prefill 3291 t/sOrnith-1.5-35B-A3B Q4_K_M (MoE, 3B active) on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/llamabench-a100sxm40-ornith15-35b-a3b-q4km-tg128
verifiedlow · 0.55
A100 40GB SXM4 llama.cpp b10808: Ornith-1.5-9B Q4_K_M decode
129 tok/s decode (tg128, batch-1, ctx 0.5k, FA on, kv q8_0); prefill 4052 t/sOrnith-1.5-9B Q4_K_M on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/llamabench-a100sxm40-ornith15-9b-q4km-tg128
verifiedlow · 0.55
A100-80GB SXM Llama-3-70B Q4_K_M decode (RunPod measured)
23.0 tok/s, single-GPU llama.cpp GGUFMeta-Llama-3-70B on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/a100-80gb-sxm-llama-3-70b-q4-k-m-decode
verifiedmedium · 0.75
A100-80GB SXM Llama-3-8B Q4_K_M decode (RunPod measured)
133.0 tok/s, single-GPU llama.cpp GGUFMeta-Llama-3-8B on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/a100-80gb-sxm-llama-3-8b-q4-k-m-decode
verifiedmedium · 0.75
A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Llama-8B aggregate output throughput, 100 concurrent
1333.21 tok/s (aggregate output tokens, 100 concurrent)deepseek-r1-distill-llama-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-deepseek-r1-distill-llama-8b-c100-out
verifiedlow · 0.60
A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Llama-8B aggregate output throughput, 50 concurrent
1155.8 tok/s (aggregate output tokens, 50 concurrent)deepseek-r1-distill-llama-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-deepseek-r1-distill-llama-8b-c50-out
verifiedlow · 0.60
A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Llama-8B median TTFT, 100 concurrent
820.62 ms (median time to first token, 100 concurrent)deepseek-r1-distill-llama-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-deepseek-r1-distill-llama-8b-c100-ttft
verifiedlow · 0.60
A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Llama-8B median TTFT, 50 concurrent
417.54 ms (median time to first token, 50 concurrent)deepseek-r1-distill-llama-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-deepseek-r1-distill-llama-8b-c50-ttft
verifiedlow · 0.60
A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B aggregate output throughput, 50 concurrent
446.73 tok/s (aggregate output tokens, 50 concurrent)deepseek-r1-distill-qwen-14b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-deepseek-r1-distill-qwen-14b-c50-out
verifiedlow · 0.60
A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B median TTFT, 50 concurrent
408.12 ms (median time to first token, 50 concurrent)deepseek-r1-distill-qwen-14b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-deepseek-r1-distill-qwen-14b-c50-ttft
verifiedlow · 0.60
A40 vLLM: google/gemma-3-12b-it aggregate output throughput, 50 concurrent
546.93 tok/s (aggregate output tokens, 50 concurrent)gemma-3-12b-it on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-gemma-3-12b-it-c50-out
verifiedlow · 0.60
A40 vLLM: google/gemma-3-12b-it median TTFT, 50 concurrent
1865.75 ms (median time to first token, 50 concurrent)gemma-3-12b-it on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-gemma-3-12b-it-c50-ttft
verifiedlow · 0.60
A40 vLLM: google/gemma-3-4b-it aggregate output throughput, 100 concurrent
401.22 tok/s (aggregate output tokens, 100 concurrent)gemma-3-4b-it on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-gemma-3-4b-it-c100-out
verifiedlow · 0.60
A40 vLLM: google/gemma-3-4b-it aggregate output throughput, 50 concurrent
756.72 tok/s (aggregate output tokens, 50 concurrent)gemma-3-4b-it on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-gemma-3-4b-it-c50-out
verifiedlow · 0.60
A40 vLLM: google/gemma-3-4b-it median TTFT, 100 concurrent
671.18 ms (median time to first token, 100 concurrent)gemma-3-4b-it on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-gemma-3-4b-it-c100-ttft
verifiedlow · 0.60
A40 vLLM: google/gemma-3-4b-it median TTFT, 50 concurrent
1153.98 ms (median time to first token, 50 concurrent)gemma-3-4b-it on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-gemma-3-4b-it-c50-ttft
verifiedlow · 0.60
A40 vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 100 concurrent
1705.49 tok/s (aggregate output tokens, 100 concurrent)llama-3-1-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-llama-3-1-8b-c100-out
verifiedlow · 0.60
A40 vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 50 concurrent
1016.6 tok/s (aggregate output tokens, 50 concurrent)llama-3-1-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-llama-3-1-8b-c50-out
verifiedlow · 0.60
A40 vLLM: meta-llama/Llama-3.1-8B-Instruct median TTFT, 100 concurrent
487.81 ms (median time to first token, 100 concurrent)llama-3-1-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-llama-3-1-8b-c100-ttft
verifiedlow · 0.60
A40 vLLM: meta-llama/Llama-3.1-8B-Instruct median TTFT, 50 concurrent
219.88 ms (median time to first token, 50 concurrent)llama-3-1-8b on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-llama-3-1-8b-c50-ttft
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-14B-Instruct aggregate output throughput, 100 concurrent
584.14 tok/s (aggregate output tokens, 100 concurrent)qwen2.5-14b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-14b-instruct-c100-out
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-14B-Instruct aggregate output throughput, 50 concurrent
529.88 tok/s (aggregate output tokens, 50 concurrent)qwen2.5-14b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-14b-instruct-c50-out
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-14B-Instruct median TTFT, 100 concurrent
1426.07 ms (median time to first token, 100 concurrent)qwen2.5-14b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-14b-instruct-c100-ttft
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-14B-Instruct median TTFT, 50 concurrent
1434.29 ms (median time to first token, 50 concurrent)qwen2.5-14b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-14b-instruct-c50-ttft
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-7B-Instruct aggregate output throughput, 100 concurrent
1123.99 tok/s (aggregate output tokens, 100 concurrent)qwen2.5-7b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-7b-instruct-c100-out
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-7B-Instruct aggregate output throughput, 50 concurrent
1094.54 tok/s (aggregate output tokens, 50 concurrent)qwen2.5-7b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-7b-instruct-c50-out
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-7B-Instruct median TTFT, 100 concurrent
1093.55 ms (median time to first token, 100 concurrent)qwen2.5-7b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-7b-instruct-c100-ttft
verifiedlow · 0.60
A40 vLLM: Qwen/Qwen2.5-7B-Instruct median TTFT, 50 concurrent
761.11 ms (median time to first token, 50 concurrent)qwen2.5-7b-instruct on nvidia-a40 via vllmbenchmarks/dm-a40-vllm-qwen25-7b-instruct-c50-ttft
verifiedlow · 0.60
Aggregate output throughput, MPT-7B static batch 64, 1x A100
800.0 output tokens/s aggregate across all in-flight requests (64 concurrent)MPT-7B (FP16) on ? via fastertransformerbenchmarks/mpt7b-a100-bs64-aggregate-output-tps
verifiedmedium · 0.80
AMD 8xMI355X, MLPerf Inference v6.1 INTERACTIVE: Llama 2 70B aggregate output throughput while holding ttft<=450ms and tpot<=40ms
73191.51 output tokens/s (aggregate, harness 'Completed tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-amd-mi355x-llama2-70b-interactive
verifiedhigh · 0.95
AMD 8xMI355X, MLPerf Inference v6.1 Offline: Llama 2 70B on AMD's own node, aggregate output throughput
106517 output tokens/s (aggregate, harness 'Tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-amd-mi355x-llama2-70b-offline
verifiedhigh · 0.95
AMD 8xMI355X, MLPerf Inference v6.1 Server: Llama 2 70B aggregate output throughput under the Server latency constraint
103274.69 output tokens/s (aggregate, harness 'Completed tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-amd-mi355x-llama2-70b-server
verifiedhigh · 0.95
AMD EPYC 9554 (64c Genoa, 1P, 12x DDR5-5600), llama.cpp: Llama-3.3-70B Q4_K_M decode tok/s
7.12 decode tok/s (single-socket, 12 channels populated)Llama-3.3-70B (dense) Q4_K_M on ? via llama-cppbenchmarks/comm-epyc-9554-llama33-70b-q4-decode
verifiedmedium · 0.80
AMD MI300X: measured power-limit sweep finds efficiency peak hardware- and workload-specific, and power limits below 400 W are not respected
400.0 W - the power-limit floor below which the MI300X did not honour the requested cap in this study? on amd-instinct-mi300x via vllmbenchmarks/mi300x-power-limit-floor-400w-not-respected
verifiedmedium · 0.82
AMD Radeon RX 7900 XTX 24GB, llama.cpp ROCm: Llama-2-7B Q4_0 decode tok/s (community measured)
118.9 decode tok/s (TG, single-stream; PP 2424)Llama-2-7B Q4_0 (GGUF) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/comm-rx7900xtx-llama2-7b-q4-decode
verifiedmedium · 0.78
AMD RX 7900 XTX 24GB, llama.cpp ROCm: Yi-34B Q4_0 decode tok/s (community measured)
32.5 decode tok/s (single-stream, tg128-class llama-bench run)Yi-34B Q4_0 (GGUF) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/comm-rx7900xtx-yi34b-q40-decode
verifiedmedium · 0.75
AMD Ryzen AI Max+ Radeon 8060S iGPU (RDNA 3.5) under Ollama: Qwen3.5 35B-A3B MoE generation throughput, 100% GPU offload
42.04 tok/s generation throughput (per-request, 100% GPU offload)Qwen3.5 35B-A3B (MoE, 3B active) on amd-radeon-8060s-ryzen-ai-max via ollamabenchmarks/ryzen-ai-max-8060s-qwen35-35b-a3b-decode-tok-s
verifiedlow · 0.60
AMD Strix Halo (Ryzen AI Max+ 395, Radeon 8060S) 128GB, llama.cpp Vulkan: Llama-3.3-70B Q4_K_M decode tok/s
5.0 decode tok/s (single-stream, ~41 GB weights)Llama-3.3-70B (dense) Q4_K_M (GGUF) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/comm-strix-halo-llama33-70b-q4-decode
verifiedmedium · 0.82
AMD Strix Halo (Ryzen AI Max+ 395) 128GB, llama.cpp Vulkan: gpt-oss-120b MXFP4 MoE decode tok/s
33.7 decode tok/s (single-stream; pp512 386.8)gpt-oss-120b MXFP4 (MoE, 117B params / 5.1B active) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/comm-strix-halo-gptoss120b-mxfp4-decode
verifiedmedium · 0.82
Apple M1 Max 32-core GPU 64GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M pp1024 prefill tok/s
33.01 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 70B Q4_K_M on apple-m1-max via llama-cppbenchmarks/xd-apple-m1-max-70b-q4km-pp1024
verifiedmedium · 0.70
Apple M1 Max 32-core GPU 64GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
4.09 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 70B Q4_K_M on apple-m1-max via llama-cppbenchmarks/xd-apple-m1-max-70b-q4km-tg1024
verifiedmedium · 0.70
Apple M1 Max 32-core GPU 64GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
418.77 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B F16 on apple-m1-max via llama-cppbenchmarks/xd-apple-m1-max-8b-f16-pp1024
verifiedmedium · 0.70
Apple M1 Max 32-core GPU 64GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
18.43 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B F16 on apple-m1-max via llama-cppbenchmarks/xd-apple-m1-max-8b-f16-tg1024
verifiedmedium · 0.70
Apple M1 Max 32-core GPU 64GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
355.45 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on apple-m1-max via llama-cppbenchmarks/xd-apple-m1-max-8b-q4km-pp1024
verifiedmedium · 0.70
Apple M1 Max 32-core GPU 64GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
34.49 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on apple-m1-max via llama-cppbenchmarks/xd-apple-m1-max-8b-q4km-tg1024
verifiedmedium · 0.70
Apple M1 Ultra (64GB), llama.cpp/LM Studio: Llama-3.3-70B 4-bit decode tok/s (Silicon Score measured)
12.6 decode tok/s (single-stream, 4-bit, 4K ctx, LM Studio)Llama-3.3-70B (dense) 4-bit on apple-m1-ultra via llama-cppbenchmarks/comm-apple-m1-ultra-llama33-70b-4bit-decode
verifiedmedium · 0.80
Apple M1 Ultra 64GB Llama-3.3-70B 4bit token-gen
12.6 tok/s 4bit @4K ctx (LM Studio)Llama-3.3-70B on apple-m1-ultra via llama-cppbenchmarks/comm-apple-m1-ultra-llama33-70b-4bit-tg
verifiedmedium · 0.75
Apple M2 Max (30-core GPU), llama.cpp Metal: Llama-2-7B F16 pp512
600.46 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-llama27b-f16-pp512-8e672ef
verifiedhigh · 0.85
Apple M2 Max (30-core GPU), llama.cpp Metal: Llama-2-7B F16 tg128
24.16 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-llama27b-f16-tg128-8e672ef
verifiedhigh · 0.85
Apple M2 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 pp512
537.6 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-llama27b-q40-pp512-8e672ef
verifiedhigh · 0.85
Apple M2 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 tg128
60.99 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-llama27b-q40-tg128-8e672ef
verifiedhigh · 0.85
Apple M2 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 pp512
540.15 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-llama27b-q80-pp512-8e672ef
verifiedhigh · 0.85
Apple M2 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 tg128
39.97 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-llama27b-q80-tg128-8e672ef
verifiedhigh · 0.85
Apple M2 Max (30c, 32GB) llamafile: Llama-3.1-8B Q4_K_M decode
31.19 tok/s avg generation (batch-1); prefill 325.0 t/sllama-3-1-8b-instruct Q4_K_M on apple-m2-max via llamafilebenchmarks/localscore-m2-max-30c-32gb-llama31-8b-q4km-decode
verifiedmedium · 0.70
Apple M2 Max (38-core GPU), llama.cpp Metal: Llama-2-7B F16 pp512
755.67 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-38core-llama27b-f16-pp512-8e672ef
verifiedhigh · 0.85
Apple M2 Max (38-core GPU), llama.cpp Metal: Llama-2-7B F16 tg128
24.65 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-38core-llama27b-f16-tg128-8e672ef
verifiedhigh · 0.85
Apple M2 Max (38-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 pp512
671.31 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-38core-llama27b-q40-pp512-8e672ef
verifiedhigh · 0.85
Apple M2 Max (38-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 tg128
65.95 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-38core-llama27b-q40-tg128-8e672ef
verifiedhigh · 0.85
Apple M2 Max (38-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 pp512
677.91 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-38core-llama27b-q80-pp512-8e672ef
verifiedhigh · 0.85
Apple M2 Max (38-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 tg128
41.83 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-38core-llama27b-q80-tg128-8e672ef
verifiedhigh · 0.85
Apple M2 Max (38c, 96GB) llamafile: Llama-3.1-8B Q4_K_M decode+prefill
46.41 tok/s avg generation (batch-1); prefill 484.06 t/sllama-3-1-8b-instruct Q4_K_M on apple-m2-max via llamafilebenchmarks/localscore-m2-max-38c-96gb-llama31-8b-q4km-decode
verifiedmedium · 0.70
Apple M2 Max (38c, 96GB) llamafile: Llama-3.1-8B Q4_K_M prompt-eval
484.06 tok/s prompt-eval (batch); decode 46.41 t/sllama-3-1-8b-instruct Q4_K_M on apple-m2-max via llamafilebenchmarks/localscore-m2-max-38c-96gb-llama31-8b-q4km-pp
verifiedmedium · 0.70
Apple M2 Max (38c, 96GB) llamafile: Qwen-2.5-14B Q4_K_M decode
25.23 tok/s avg generation (batch-1); prefill 252.47 t/sqwen-2-5-14b-instruct Q4_K_M on apple-m2-max via llamafilebenchmarks/localscore-m2-max-38c-96gb-qwen25-14b-q4km-decode
verifiedmedium · 0.70
Apple M2 Max (38c, 96GB) MLX: Qwen3.5-27B 8-bit decode
12.0 tok/s (MLX, 8-bit, per-request)Qwen3.5-27B 8bit on apple-m2-max via mlx-lmbenchmarks/comm-m2-max-38c-96gb-qwen35-27b-8bit-mlx-decode
verifiedlow · 0.60
Apple M2 Max 38-core (64GB), llama.cpp/LM Studio: Llama-3.3-70B 4-bit decode tok/s (Silicon Score measured)
8.8 decode tok/s (single-stream, 4-bit, 4K ctx, LM Studio)Llama-3.3-70B (dense) 4-bit on apple-m2-max via llama-cppbenchmarks/comm-apple-m2-max-llama33-70b-4bit-decode
verifiedmedium · 0.80
Apple M2 Max 64GB Llama-3.3-70B 4bit token-gen
8.8 tok/s 4bit (LM Studio)Llama-3.3-70B on apple-m2-max via llama-cppbenchmarks/comm-apple-m2-max-llama33-70b-4bit-tg
verifiedmedium · 0.70
Apple M2 Ultra (192GB) Llama-3.3-70B 4-bit decode (bandwidth-scaled estimate)
10.0 tok/s ESTIMATE - no direct measured rowLlama-3.3-70B 4-bit on apple-m2-ultra via llama-cppbenchmarks/comm-apple-m2-ultra-llama33-70b-4bit-decode-est
verifiedmedium · 0.75
Apple M3 Max (30-core GPU), llama.cpp Metal: Llama-2-7B F16 pp512
589.41 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-30core-llama27b-f16-pp512-8e672ef
verifiedhigh · 0.85
Apple M3 Max (30-core GPU), llama.cpp Metal: Llama-2-7B F16 tg128
19.54 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-30core-llama27b-f16-tg128-8e672ef
verifiedhigh · 0.85
Apple M3 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 pp512
567.59 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-30core-llama27b-q40-pp512-8e672ef
verifiedhigh · 0.85
Apple M3 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 tg128
56.58 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-30core-llama27b-q40-tg128-8e672ef
verifiedhigh · 0.85
Apple M3 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 pp512
566.4 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-30core-llama27b-q80-pp512-8e672ef
verifiedhigh · 0.85
Apple M3 Max (30-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 tg128
34.3 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-30core-llama27b-q80-tg128-8e672ef
verifiedhigh · 0.85
Apple M3 Max (40-core GPU), llama.cpp Metal: Llama-2-7B F16 pp512
779.17 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-llama27b-f16-pp512-8e672ef
verifiedhigh · 0.85
Apple M3 Max (40-core GPU), llama.cpp Metal: Llama-2-7B F16 tg128
25.09 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.F16.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-llama27b-f16-tg128-8e672ef
verifiedhigh · 0.85
Apple M3 Max (40-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 pp512
759.7 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-llama27b-q40-pp512-8e672ef
verifiedhigh · 0.85
Apple M3 Max (40-core GPU), llama.cpp Metal: Llama-2-7B Q4_0 tg128
66.31 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q4_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-llama27b-q40-tg128-8e672ef
verifiedhigh · 0.85
Apple M3 Max (40-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 pp512
757.64 tok/s (pp512, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-llama27b-q80-pp512-8e672ef
verifiedhigh · 0.85
Apple M3 Max (40-core GPU), llama.cpp Metal: Llama-2-7B Q8_0 tg128
42.75 tok/s (tg128, Metal, -ngl 99)Llama-2-7B (GGUF, TheBloke llama-2-7b.Q8_0.gguf) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-llama27b-q80-tg128-8e672ef
verifiedhigh · 0.85
Apple M3 Max (64GB) Llama-3.3-70B Q4_K_M decode
8.2 tok/s decode at 258 ctx, falling to 6.1 at 32k ctxLlama-3.3-70B (dense) Q4_K_M on apple-m3-max via llama-cppbenchmarks/comm-apple-m3-max-llama33-70b-4bit-decode
verifiedmedium · 0.75
Apple M3 Max (64GB) llama.cpp: Llama-3.3-70B Q4_K_M decode @258 ctx
8.15 tok/s single-stream @ 258 ctx; falls to 6.13 @ 32k ctxLlama-3.3-70B Q4_K_M on apple-m3-max via llama-cppbenchmarks/comm-apple-m3-max-llama33-70b-q4km-decode-258ctx
verifiedmedium · 0.70
Apple M3 Max (64GB) llama.cpp: Llama-3.3-70B Q4_K_M decode @32k ctx
6.13 tok/s single-stream @ 32,170 ctxLlama-3.3-70B Q4_K_M on apple-m3-max via llama-cppbenchmarks/comm-apple-m3-max-llama33-70b-q4km-decode-32kctx
verifiedmedium · 0.70
Apple M3 Max (64GB), llama.cpp: Llama-3.3-70B Q4_K_M decode tok/s, context-scaled (Silicon Score measured)
6.1 decode tok/s (single-stream, Q4_K_M, 32K ctx)Llama-3.3-70B (dense) Q4_K_M on apple-m3-max via llama-cppbenchmarks/comm-apple-m3-max-llama33-70b-q4km-decode
verifiedmedium · 0.75
Apple M3 Ultra (512GB) Llama-3.3-70B 4-bit decode
15.5 tok/s (4-bit, 8K ctx)Llama-3.3-70B 4-bit on apple-m3-ultra via llama-cppbenchmarks/comm-apple-m3-ultra-llama33-70b-4bit-decode
verifiedmedium · 0.75
Apple M3 Ultra (512GB), llama.cpp/MLX: Llama-3.3-70B 8-bit decode tok/s (Silicon Score measured)
8.5 decode tok/s (single-stream, 8-bit)Llama-3.3-70B (dense) 8-bit on apple-m3-ultra via llama-cppbenchmarks/comm-apple-m3-ultra-llama33-70b-8bit-decode
verifiedmedium · 0.75
Apple M4 Max (128GB, 40-core) Llama-3.3-70B 4-bit decode
11.8 tok/s (4-bit); 8-bit 6.5Llama-3.3-70B 4-bit on apple-m4-max via llama-cppbenchmarks/comm-apple-m4-max-llama33-70b-4bit-decode
verifiedmedium · 0.75
Apple M4 Pro via Ollama, five 3-9B models: 0.09-0.25 J/output token system-level, 4.4-11.6 mJ/token package-level, spanning a 2.6x range by model size alone
59.0 output tokens/s (Qwen 2.5 7B); ENERGY PER TOKEN, SYSTEM-LEVEL: Llama 3.2 3B 0.09, Phi-3 3.8B 0.12, Qwen 2.5 7B 0.20, Mistral 7B 0.22, Gemma 2 9B 0.25 J/tok. PACKAGE-LEVEL mJ/tok: 4.4, 5.4, 9.4, 10.4, 11.6 respectively. Energy metric absent from the enum; joules carried in the unit stringLlama 3.2 3B / Phi-3 3.8B / Qwen 2.5 7B / Mistral 7B / Gemma 2 9B (Q4_K_M) on apple-m4-max via ollamabenchmarks/apple-m4-pro-5-model-system-joules-per-token-range
verifiedmedium · 0.76
Apple M4 Pro, Llama 3.2 3B via Ollama: 0.09 J/output token system-level (10 W system estimate), 4.4 mJ/token at the CPU+GPU package
115.2 output tokens/s; ENERGY: 0.09 J/output token SYSTEM-level and 0.0044 J/token (4.4 mJ) CPU+GPU PACKAGE-level. The metric enum has no energy member so throughput carries the value and the joules travel in the unit string - see the H200 power records for the same conventionLlama 3.2 3B (Q4_K_M) on apple-m4-max via ollamabenchmarks/apple-m4-pro-llama3-2-3b-ollama-system-joules-per-token
verifiedmedium · 0.78
Arc Pro B60 24GB Qwen3.5-35B-A3B Q8 token-gen, SYCL (Level1Techs)
8.3 tok/sQwen3.5-35B-A3B (MoE) on intel-arc-pro-b60 via llama-cppbenchmarks/comm-arc-pro-b60-qwen35-35b-a3b-q8-sycl-tg
verifiedmedium · 0.75
Arc Pro B60 24GB Qwen3.5-9B Q4_K_XL token-gen, Vulkan backend (Level1Techs)
17.8 tok/sQwen3.5-9B on intel-arc-pro-b60 via llama-cppbenchmarks/comm-arc-pro-b60-qwen35-9b-q4-vulkan-tg
verifiedmedium · 0.75
Arc Pro B60 Qwen3.5-9B Q4_K_XL token-gen (SYCL)
23.0 tok/s tg128 SYCLQwen3.5-9B on intel-arc-pro-b60 via llama-cppbenchmarks/comm-arc-pro-b60-qwen35-9b-q4-sycl-tg
verifiedmedium · 0.75
Arc Pro B70 Qwen3.5-27B token-gen (vLLM, single-req)
13.4 tok/s single-request floor, dynamic FP8Qwen3.5-27B on intel-arc-pro-b70 via vllmbenchmarks/comm-arc-pro-b70-qwen35-27b-vllm-fp8-tg
verifiedmedium · 0.70
Argonne A100-SXM-40GB x1 Deepspeed-MII meta-llama/Llama-2-7b-hf b1
93.82 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 Deepspeed-MII meta-llama/Llama-2-7b-hf b64
4259.07 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 Deepspeed-MII meta-llama/Meta-Llama-3-8B b1
106.95 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama3-8b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 Deepspeed-MII meta-llama/Meta-Llama-3-8B b64
3590.96 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama3-8b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 Deepspeed-MII mistralai/Mistral-7B-v0.1 b1
116.31 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mistral-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 Deepspeed-MII mistralai/Mistral-7B-v0.1 b64
3855.99 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mistral-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 llama.cpp meta-llama/Llama-2-7b-hf b1
58.92 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama2-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 llama.cpp meta-llama/Llama-2-7b-hf b64
114.45 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama2-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 llama.cpp meta-llama/Meta-Llama-3-8B b1
53.13 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama3-8b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 llama.cpp meta-llama/Meta-Llama-3-8B b64
103.4 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama3-8b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 llama.cpp mistralai/Mistral-7B-v0.1 b1
54.95 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-mistral-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 llama.cpp mistralai/Mistral-7B-v0.1 b64
106.99 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-mistral-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 TensorRT-LLM meta-llama/Llama-2-7b-hf b1
168.96 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 TensorRT-LLM meta-llama/Llama-2-7b-hf b64
6115.68 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 TensorRT-LLM meta-llama/Meta-Llama-3-8B b1
143.73 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-8b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 TensorRT-LLM meta-llama/Meta-Llama-3-8B b64
5999.61 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-8b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 TensorRT-LLM mistralai/Mistral-7B-v0.1 b1
153.08 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mistral-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 TensorRT-LLM mistralai/Mistral-7B-v0.1 b64
6372.97 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mistral-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 vLLM meta-llama/Llama-2-7b-hf b1
154.87 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 vLLM meta-llama/Llama-2-7b-hf b64
4919.51 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 vLLM meta-llama/Meta-Llama-3-8B b1
140.18 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama3-8b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 vLLM meta-llama/Meta-Llama-3-8B b64
4981.92 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama3-8b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 vLLM mistralai/Mistral-7B-v0.1 b1
148.53 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x1 vLLM mistralai/Mistral-7B-v0.1 b64
4968.35 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 Deepspeed-MII meta-llama/Llama-2-7b-hf b1
151.24 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 Deepspeed-MII meta-llama/Llama-2-7b-hf b64
5163.51 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 Deepspeed-MII meta-llama/Meta-Llama-3-8B b1
135.75 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama3-8b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 Deepspeed-MII meta-llama/Meta-Llama-3-8B b64
4562.56 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama3-8b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 Deepspeed-MII mistralai/Mistral-7B-v0.1 b1
143.69 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mistral-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 Deepspeed-MII mistralai/Mistral-7B-v0.1 b64
5009.78 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mistral-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp meta-llama/Llama-2-7b-hf b1
72.83 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp meta-llama/Llama-2-7b-hf b64
108.93 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp meta-llama/Meta-Llama-3-8B b1
66.77 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama3-8b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp meta-llama/Meta-Llama-3-8B b64
98.33 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama3-8b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp mistralai/Mistral-7B-v0.1 b1
68.09 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-mistral-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp mistralai/Mistral-7B-v0.1 b64
101.95 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-mistral-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 llama.cpp Qwen/Qwen2-7B b1
68.6 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-qwen2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM meta-llama/Llama-2-7b-hf b1
265.37 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM meta-llama/Llama-2-7b-hf b64
9136.6 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM meta-llama/Meta-Llama-3-8B b1
226.08 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-8b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM meta-llama/Meta-Llama-3-8B b64
8824.68 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-8b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM mistralai/Mistral-7B-v0.1 b1
238.43 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mistral-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM mistralai/Mistral-7B-v0.1 b64
9454.07 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mistral-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM Qwen/Qwen2-7B b1
237.76 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-qwen2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 TensorRT-LLM Qwen/Qwen2-7B b64
9383.39 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-qwen2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM meta-llama/Llama-2-7b-hf b1
234.63 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM meta-llama/Llama-2-7b-hf b64
6802.08 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM meta-llama/Meta-Llama-3-8B b1
203.74 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama3-8b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM meta-llama/Meta-Llama-3-8B b64
6813.25 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama3-8b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM mistralai/Mistral-7B-v0.1 b1
213.64 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM mistralai/Mistral-7B-v0.1 b64
6810.19 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM Qwen/Qwen2-7B b1
213.65 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-qwen2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x2 vLLM Qwen/Qwen2-7B b64
7166.66 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-qwen2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII meta-llama/Llama-2-70b-hf b1
48.55 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII meta-llama/Llama-2-70b-hf b64
1804.24 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII meta-llama/Llama-2-7b-hf b1
140.36 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII meta-llama/Llama-2-7b-hf b64
6664.0 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII meta-llama/Meta-Llama-3-8B b1
139.77 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama3-8b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII meta-llama/Meta-Llama-3-8B b64
5891.91 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-llama3-8b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII mistralai/Mistral-7B-v0.1 b1
142.3 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mistral-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII mistralai/Mistral-7B-v0.1 b64
6361.23 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mistral-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII mistralai/Mixtral-8x7B-v0.1 b1
97.81 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mixtral-8x7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 Deepspeed-MII mistralai/Mixtral-8x7B-v0.1 b64
4722.67 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-niche-deepspeed-mii-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp meta-llama/Llama-2-7b-hf b1
84.3 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp meta-llama/Llama-2-7b-hf b64
96.94 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp meta-llama/Meta-Llama-3-8B b1
76.46 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama3-8b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp meta-llama/Meta-Llama-3-8B b64
87.96 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-llama3-8b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp mistralai/Mistral-7B-v0.1 b1
78.66 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-mistral-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp mistralai/Mistral-7B-v0.1 b64
90.64 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-mistral-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 llama.cpp Qwen/Qwen2-7B b1
78.8 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llama-cpp-qwen2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Llama-2-70b-hf b1
60.38 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Llama-2-70b-hf b64
2525.97 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Llama-2-7b-hf b1
345.77 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Llama-2-7b-hf b64
13447.59 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Meta-Llama-3-70B b1
59.56 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-70B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Meta-Llama-3-8B b1
327.92 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-8b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM meta-llama/Meta-Llama-3-8B b64
12889.04 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-llama3-8b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM mistralai/Mistral-7B-v0.1 b1
348.85 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mistral-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM mistralai/Mistral-7B-v0.1 b64
13829.43 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mistral-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM mistralai/Mixtral-8x7B-v0.1 b1
233.91 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mixtral-8x7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM mistralai/Mixtral-8x7B-v0.1 b64
8306.96 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM Qwen/Qwen2-7B b1
344.56 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-qwen2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 TensorRT-LLM Qwen/Qwen2-7B b64
13648.51 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrt-llm-qwen2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM meta-llama/Llama-2-70b-hf b1
59.74 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM meta-llama/Llama-2-7b-hf b1
319.19 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM meta-llama/Llama-2-7b-hf b64
8986.41 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM mistralai/Mistral-7B-v0.1 b1
305.09 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM mistralai/Mistral-7B-v0.1 b64
9333.92 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM mistralai/Mixtral-8x7B-v0.1 b1
193.67 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mixtral-8x7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM mistralai/Mixtral-8x7B-v0.1 b64
4222.76 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne A100-SXM-40GB x4 vLLM Qwen/Qwen2-7B b1
300.78 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-qwen2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne Gaudi2 x8 DS-MII Llama-3-8B b64
18187.0 tok/s aggregatemeta-llama/Meta-Llama-3-8B on intel-gaudi2 via niche-deepspeed-miibenchmarks/argonne-gaudi2-deepspeed-llama3-8b-bf16-8x-b64-128io-tps
verifiedmedium · 0.75
Argonne Gaudi2 x8 DS-MII Mistral-7B b64
14502.3 tok/s aggregatemistralai/Mistral-7B-v0.1 on intel-gaudi2 via niche-deepspeed-miibenchmarks/argonne-gaudi2-deepspeed-mistral-7b-bf16-8x-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 llama.cpp Llama-2-7B b1
124.2 tok/s aggregatemeta-llama/Llama-2-7b-hf on nvidia-gh200 via llama-cppbenchmarks/argonne-gh200-llamacpp-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 llama.cpp Llama-2-7B b64
232.7 tok/s aggregatemeta-llama/Llama-2-7b-hf on nvidia-gh200 via llama-cppbenchmarks/argonne-gh200-llamacpp-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 TRT-LLM Llama-2-7B b64
14255.8 tok/s aggregatemeta-llama/Llama-2-7b-hf on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 TRT-LLM Llama-3-8B b64
14385.7 tok/s aggregatemeta-llama/Meta-Llama-3-8B on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 TRT-LLM Mistral-7B b1
371.9 tok/s aggregatemistralai/Mistral-7B-v0.1 on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 TRT-LLM Mistral-7B b64
15306.2 tok/s aggregatemistralai/Mistral-7B-v0.1 on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-mistral-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 vLLM Llama-2-7B b1
297.4 tok/s aggregatemeta-llama/Llama-2-7b-hf on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 vLLM Llama-2-7B b64
8576.1 tok/s aggregatemeta-llama/Llama-2-7b-hf on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 vLLM Llama-3-8B b64
9800.6 tok/s aggregatemeta-llama/Meta-Llama-3-8B on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne GH200 x1 vLLM Qwen2-7B b64
9685.2 tok/s aggregateQwen/Qwen2-7B on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-qwen2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: A100 40GB SXM x1, Deepspeed-MII, Llama-2-7b-hf, 128in/128out, batch 1
93.82 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100-nichedeepspeedmii-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, llama.cpp, Llama-2-7b-hf, 128in/128out, batch 1
58.92 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100-llamacpp-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, llama.cpp, Meta-Llama-3-8B, 128in/128out, batch 1
53.13 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100-llamacpp-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, TensorRT-LLM, Llama-2-7b-hf, 128in/128out, batch 1
168.96 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, TensorRT-LLM, Llama-2-7b-hf, 128in/128out, batch 64
6115.68 tok/s aggregate across batch 64 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, TensorRT-LLM, Meta-Llama-3-8B, 128in/128out, batch 1
143.73 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Llama-2-7b-hf, 1024in/1024out, batch 1
146.53 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama2-7b-fp16-b1-1024io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Llama-2-7b-hf, 128in/128out, batch 1
154.87 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Llama-2-7b-hf, 128in/128out, batch 64
4919.51 tok/s aggregate across batch 64 (1x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Meta-Llama-3-8B, 128in/128out, batch 1
140.18 tok/s aggregate across batch 1 (1x A100-40GB-SXM)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Meta-Llama-3-8B, 128in/128out, batch 64
4981.92 tok/s aggregate across batch 64 (1x A100-40GB-SXM)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Mistral-7B-v0.1, 128in/128out, batch 1
148.53 tok/s aggregate across batch 1 (1x A100-40GB-SXM)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, Deepspeed-MII, Llama-2-70b-hf, 128in/128out, batch 1
48.55 tok/s aggregate across batch 1 (4x A100-40GB-SXM)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100-nichedeepspeedmii-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, Deepspeed-MII, Llama-2-70b-hf, 128in/128out, batch 64
1804.24 tok/s aggregate across batch 64 (4x A100-40GB-SXM)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100-nichedeepspeedmii-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, Deepspeed-MII, Llama-2-7b-hf, 128in/128out, batch 64
6664.0 tok/s aggregate across batch 64 (4x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100-nichedeepspeedmii-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, Deepspeed-MII, Mixtral-8x7B-v0.1, 128in/128out, batch 64
4722.67 tok/s aggregate across batch 64 (4x A100-40GB-SXM)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100-nichedeepspeedmii-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, llama.cpp, Llama-2-7b-hf, 128in/128out, batch 1
84.3 tok/s aggregate across batch 1 (4x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100-llamacpp-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, llama.cpp, Llama-2-7b-hf, 128in/128out, batch 64
96.94 tok/s aggregate across batch 64 (4x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100-llamacpp-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, TensorRT-LLM, Llama-2-70b-hf, 128in/128out, batch 1
60.38 tok/s aggregate across batch 1 (4x A100-40GB-SXM)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, TensorRT-LLM, Llama-2-70b-hf, 128in/128out, batch 64
2525.97 tok/s aggregate across batch 64 (4x A100-40GB-SXM)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, TensorRT-LLM, Meta-Llama-3-8B, 128in/128out, batch 64
12889.04 tok/s aggregate across batch 64 (4x A100-40GB-SXM)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-llama3-8b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, TensorRT-LLM, Mixtral-8x7B-v0.1, 128in/128out, batch 64
8306.96 tok/s aggregate across batch 64 (4x A100-40GB-SXM)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100-tensorrtllm-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Llama-2-70b-hf, 128in/128out, batch 1
59.74 tok/s aggregate across batch 1 (4x A100-40GB-SXM)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Llama-2-7b-hf, 128in/128out, batch 1
319.19 tok/s aggregate across batch 1 (4x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Llama-2-7b-hf, 128in/128out, batch 64
8986.41 tok/s aggregate across batch 64 (4x A100-40GB-SXM)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Mistral-7B-v0.1, 128in/128out, batch 64
9446.83 tok/s aggregate across batch 64 (4x A100-40GB-SXM)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-mistral-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Mixtral-8x7B-v0.1, 128in/128out, batch 1
195.84 tok/s aggregate across batch 1 (4x A100-40GB-SXM)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-mixtral-8x7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: A100 40GB SXM x4, vLLM, Mixtral-8x7B-v0.1, 128in/128out, batch 64
4222.76 tok/s aggregate across batch 64 (4x A100-40GB-SXM)mistralai/Mixtral-8x7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100-vllm-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: Gaudi2 x1, DeepSpeed, Llama-2-7B BF16, 128in/128out, batch 1
291.7 tok/s aggregate (batch 1 = single request)meta-llama/Llama-2-7b-hf on intel-gaudi2 via niche-deepspeed-miibenchmarks/argonne-gaudi2-deepspeed-llama2-7b-bf16-b1-128io-tps
verifiedmedium · 0.70
Argonne LLM-Inference-Bench: Gaudi2 x8, DeepSpeed, Llama-2-70B BF16, 128in/128out, batch 64
4764.8 tok/s aggregate across 64 batched requests on 8x Gaudi2meta-llama/Llama-2-70b-hf on intel-gaudi2 via niche-deepspeed-miibenchmarks/argonne-gaudi2-deepspeed-llama2-70b-bf16-8x-b64-128io-tps
verifiedmedium · 0.70
Argonne LLM-Inference-Bench: Intel Max 1550 x1, vLLM 0.5.2-XPU, Llama-2-7B FP16, 128in/128out, batch 1
127.5 tok/s aggregate (batch 1 = single request)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: Intel Max 1550 x1, vLLM 0.5.2-XPU, Llama-2-7B FP16, 128in/128out, batch 64
2100.9 tok/s aggregate across 64 batched requestsmeta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: Intel Max 1550 x1, vLLM 0.5.2-XPU, Llama-3-8B FP16, 128in/128out, batch 1
130.7 tok/s aggregate (batch 1 = single request)meta-llama/Meta-Llama-3-8B on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: Intel Max 1550 x1, vLLM 0.5.2-XPU, Llama-3-8B FP16, 128in/128out, batch 64
2123.6 tok/s aggregate across 64 batched requestsmeta-llama/Meta-Llama-3-8B on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: Intel Max 1550 x1, vLLM 0.5.2-XPU, Mistral-7B FP16, 128in/128out, batch 1
126.4 tok/s aggregate (batch 1 = single request)mistralai/Mistral-7B-v0.1 on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: Intel Max 1550 x1, vLLM 0.5.2-XPU, Mistral-7B FP16, 128in/128out, batch 64
2245.0 tok/s aggregate across 64 batched requestsmistralai/Mistral-7B-v0.1 on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-mistral-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: MI250 x1, llama.cpp (ROCm/HIP), Llama-2-7B, 128in/128out, batch 1
44.4 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne LLM-Inference-Bench: MI250 x1, vLLM 0.6.1, Llama-2-7B FP16, 128in/128out, batch 1
139.9 tok/s aggregate (batch 1 = single request; in+out tokens / latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x1, vLLM 0.6.1, Llama-2-7B FP16, 128in/128out, batch 64
4477.1 tok/s aggregate across 64 batched requestsmeta-llama/Llama-2-7b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x1, vLLM 0.6.1, Llama-3-8B FP16, 128in/128out, batch 1
130.2 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x1, vLLM 0.6.1, Llama-3-8B FP16, 128in/128out, batch 64
3751.2 tok/s aggregate across 64 batched requestsmeta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x1, vLLM 0.6.1, Mixtral-8x7B FP16, 128in/128out, batch 1
83.1 tok/s aggregate (batch 1, single request)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-mixtral-8x7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x1, vLLM 0.6.1, Mixtral-8x7B FP16, 128in/128out, batch 64
1218.6 tok/s aggregate across 64 batched requestsmistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-mixtral-8x7b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x4 (TP), vLLM 0.6.1, Llama-2-70B FP16, 128in/128out, batch 1
47.7 tok/s aggregate (batch 1, single request over 4x MI250 TP)meta-llama/Llama-2-70b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x4 (TP), vLLM 0.6.1, Llama-2-70B FP16, 128in/128out, batch 64
1773.4 tok/s aggregate across 64 batched requests on 4x MI250meta-llama/Llama-2-70b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x4 (TP), vLLM 0.6.1, Qwen2-72B FP16, 128in/128out, batch 1
46.0 tok/s aggregate (batch 1, single request over 4x MI250 TP)Qwen/Qwen2-72B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-qwen2-72b-fp16-4x-b1-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: MI250 x4 (TP), vLLM 0.6.1, Qwen2-72B FP16, 128in/128out, batch 64
1708.6 tok/s aggregate across 64 batched requests on 4x MI250Qwen/Qwen2-72B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-qwen2-72b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne LLM-Inference-Bench: SambaNova SN40L x8, SambaFlow, Llama-3-70B BF16, 128in/128out, batch 16
1050.0 tok/s aggregate across 16 batched requests on 8x SN40Lmeta-llama/Meta-Llama-3-70B on sambanova-sn40l via sambaflow-sambanova-rdu-runtimebenchmarks/argonne-sn40l-sambaflow-llama3-70b-8x-b16-128io-tps
verifiedmedium · 0.70
Argonne LLM-Inference-Bench: SambaNova SN40L x8, SambaFlow, Llama-3-8B BF16, 128in/128out, batch 1
1165.7 tok/s aggregate (batch 1 single request on 8-RDU node)meta-llama/Meta-Llama-3-8B on sambanova-sn40l via sambaflow-sambanova-rdu-runtimebenchmarks/argonne-sn40l-sambaflow-llama3-8b-8x-b1-128io-tps
verifiedmedium · 0.70
Argonne LLM-Inference-Bench: SambaNova SN40L x8, SambaFlow, Mistral-7B BF16, 128in/128out, batch 1
1226.4 tok/s aggregate (batch 1 single request on 8-RDU node)mistralai/Mistral-7B-v0.1 on sambanova-sn40l via sambaflow-sambanova-rdu-runtimebenchmarks/argonne-sn40l-sambaflow-mistral-7b-8x-b1-128io-tps
verifiedmedium · 0.70
Argonne LLM-Inference-Bench: SambaNova SN40L x8, SambaFlow, Mistral-7B BF16, 128in/128out, batch 16
6967.6 tok/s aggregate across 16 batched requests on 8x SN40Lmistralai/Mistral-7B-v0.1 on sambanova-sn40l via sambaflow-sambanova-rdu-runtimebenchmarks/argonne-sn40l-sambaflow-mistral-7b-8x-b16-128io-tps
verifiedmedium · 0.70
Argonne MI250 llama.cpp Llama-2-7B b64
65.0 tok/s aggregate (batch*(in+out)/lat)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 llama.cpp Mistral-7B b1
41.4 tok/s aggregatemistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 llama.cpp Mixtral-8x7B b1
17.2 tok/s aggregate (batch*(in+out)/lat)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 llama.cpp Qwen2-72B b1
4.6 tok/s aggregate (batch*(in+out)/lat)Qwen/Qwen2-72B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-72b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 vLLM Llama-3-70B 4x b1
33.0 tok/s aggregate (batch*(in+out)/lat)meta-llama/Meta-Llama-3-70B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama3-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 vLLM Llama-3-70B 4x b64
1060.0 tok/s aggregate (batch*(in+out)/lat)meta-llama/Meta-Llama-3-70B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama3-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 vLLM Mistral-7B b1
135.7 tok/s aggregatemistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 vLLM Mistral-7B b64
3903.4 tok/s aggregatemistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-mistral-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 vLLM Qwen2-7B b1
136.0 tok/s aggregate (batch*(in+out)/lat)Qwen/Qwen2-7B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-qwen2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 vLLM Qwen2-7B b64
4091.8 tok/s aggregate (batch*(in+out)/lat)Qwen/Qwen2-7B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-qwen2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp meta-llama/Llama-2-7b-hf FP16 b1 128io
44.43 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp meta-llama/Llama-2-7b-hf FP16 b64 128io
65.04 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp meta-llama/Meta-Llama-3-8B FP16 b1 128io
39.87 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-8b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp meta-llama/Meta-Llama-3-8B FP16 b64 128io
57.72 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-8b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp mistralai/Mistral-7B-v0.1 FP16 b1 128io
41.36 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp mistralai/Mistral-7B-v0.1 FP16 b64 128io
59.75 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp mistralai/Mixtral-8x7B-v0.1 FP16 b1 128io
17.19 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp mistralai/Mixtral-8x7B-v0.1 FP16 b64 128io
28.92 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp Qwen/Qwen2-7B FP16 b1 128io
40.67 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-7b-fp16-1x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x1 llama.cpp Qwen/Qwen2-7B FP16 b64 128io
59.7 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-7b-fp16-1x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp meta-llama/Llama-2-7b-hf FP16 b1 128io
43.66 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp meta-llama/Llama-2-7b-hf FP16 b64 128io
64.15 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp meta-llama/Meta-Llama-3-8B FP16 b1 128io
39.29 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-8b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp meta-llama/Meta-Llama-3-8B FP16 b64 128io
56.63 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-8b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp mistralai/Mistral-7B-v0.1 FP16 b1 128io
40.68 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp mistralai/Mistral-7B-v0.1 FP16 b64 128io
58.87 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp mistralai/Mixtral-8x7B-v0.1 FP16 b1 128io
16.87 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp mistralai/Mixtral-8x7B-v0.1 FP16 b64 128io
28.54 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp Qwen/Qwen2-7B FP16 b1 128io
39.61 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-7b-fp16-2x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x2 llama.cpp Qwen/Qwen2-7B FP16 b64 128io
58.09 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Llama-2-70b-hf FP16 b1 128io
4.78 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Llama-2-70b-hf FP16 b64 128io
6.99 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Llama-2-7b-hf FP16 b1 128io
42.49 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Llama-2-7b-hf FP16 b64 128io
62.54 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Meta-Llama-3-70B FP16 b1 128io
4.73 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-70B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Meta-Llama-3-70B FP16 b64 128io
6.92 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-70B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Meta-Llama-3-8B FP16 b1 128io
38.36 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-8b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp meta-llama/Meta-Llama-3-8B FP16 b64 128io
55.6 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-llama3-8b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp mistralai/Mistral-7B-v0.1 FP16 b1 128io
39.68 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp mistralai/Mistral-7B-v0.1 FP16 b64 128io
57.27 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mistral-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp mistralai/Mixtral-8x7B-v0.1 FP16 b1 128io
16.31 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp mistralai/Mixtral-8x7B-v0.1 FP16 b64 128io
27.64 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-mixtral-8x7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp Qwen/Qwen2-72B FP16 b1 128io
4.58 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-72B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-72b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp Qwen/Qwen2-72B FP16 b64 128io
6.72 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-72B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-72b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp Qwen/Qwen2-7B FP16 b1 128io
38.78 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-7b-fp16-4x-b1-128io-tps
verifiedmedium · 0.75
Argonne MI250 x4 llama.cpp Qwen/Qwen2-7B FP16 b64 128io
56.18 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-qwen2-7b-fp16-4x-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Llama-2-70B b1
15.5 tok/s aggregate (batch*(in+out)/lat)meta-llama/Llama-2-70b-hf on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-llama2-70b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Llama-2-7B b1
71.5 tok/s aggregate (batch*(in+out)/lat)meta-llama/Llama-2-7b-hf on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Llama-2-7B b64
133.6 tok/s aggregate (batch*(in+out)/lat)meta-llama/Llama-2-7b-hf on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Llama-3-70B b1
15.4 tok/s aggregate (batch*(in+out)/lat)meta-llama/Meta-Llama-3-70B on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-llama3-70b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Llama-3-8B b1
63.8 tok/s aggregatemeta-llama/Meta-Llama-3-8B on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Mistral-7B b1
65.3 tok/s aggregate (batch*(in+out)/lat)mistralai/Mistral-7B-v0.1 on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Mixtral-8x7B b1
33.0 tok/s aggregatemistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-mixtral-8x7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X llama.cpp Qwen2-7B b1
70.3 tok/s aggregate (batch*(in+out)/lat)Qwen/Qwen2-7B on amd-instinct-mi300x via llama-cppbenchmarks/argonne-mi300x-llamacpp-qwen2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-2-70B b1
45.7 tok/s aggregatemeta-llama/Llama-2-70b-hf on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama2-70b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-2-70B b64
928.5 tok/s aggregatemeta-llama/Llama-2-70b-hf on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama2-70b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-2-7B b1
113.5 tok/s aggregate (batch*(in+out)/lat)meta-llama/Llama-2-7b-hf on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-2-7B b64
5926.2 tok/s aggregate (batch*(in+out)/lat)meta-llama/Llama-2-7b-hf on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-3-70B b1
51.4 tok/s aggregate (batch*(in+out)/lat)meta-llama/Meta-Llama-3-70B on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama3-70b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-3-70B b64
876.5 tok/s aggregate (batch*(in+out)/lat)meta-llama/Meta-Llama-3-70B on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama3-70b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-3-8B b1
114.0 tok/s aggregatemeta-llama/Meta-Llama-3-8B on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Llama-3-8B b64
5244.1 tok/s aggregatemeta-llama/Meta-Llama-3-8B on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Mistral-7B b1
256.4 tok/s aggregate (batch*(in+out)/lat)mistralai/Mistral-7B-v0.1 on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Mistral-7B b64
4348.4 tok/s aggregate (batch*(in+out)/lat)mistralai/Mistral-7B-v0.1 on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-mistral-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Mixtral-8x7B b1
171.6 tok/s aggregatemistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-mixtral-8x7b-fp16-b1-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Mixtral-8x7B b64
3715.4 tok/s aggregatemistralai/Mixtral-8x7B-v0.1 on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-mixtral-8x7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne MI300X vLLM Qwen2-7B b64
5771.7 tok/s aggregate (batch*(in+out)/lat)Qwen/Qwen2-7B on amd-instinct-mi300x via vllmbenchmarks/argonne-mi300x-vllm-qwen2-7b-fp16-b64-128io-tps
verifiedmedium · 0.75
Argonne SN40L x8 SambaFlow Llama-3-8B b16
7381.3 tok/s aggregatemeta-llama/Meta-Llama-3-8B on sambanova-sn40l via sambaflow-sambanova-rdu-runtimebenchmarks/argonne-sn40l-sambaflow-llama3-8b-8x-b16-128io-tps
verifiedmedium · 0.75
Argonne: A100 40GB SXM x1, llama.cpp CUDA, Llama-2-7B FP16, 128io, b1
58.9 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llamacpp-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x1, TensorRT-LLM 0.11.0, Llama-2-7B FP16, 128io, b1
169.0 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-trtllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x1, vLLM 0.6.1, Llama-2-7B FP16, 128io, b1
154.9 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x1, vLLM 0.6.1, Llama-2-7B FP16, 128io, b64
4919.5 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x1, vLLM 0.6.1, Llama-3-8B FP16, 128io, b1
140.2 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x1, vLLM 0.6.1, Llama-3-8B FP16, 128io, b64
4981.9 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama3-8b-fp16-b64-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x1, vLLM 0.6.1, Mistral-7B FP16, 128io, b1
148.5 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral-7b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x4 TP, DeepSpeed-MII, Llama-2-70B FP16, 128io, b64
1804.2 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-dsmii-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne: A100 40GB SXM x4 TP, TensorRT-LLM 0.11.0, Llama-2-70B FP16, 128io, b64
2526.0 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-trtllm-llama2-70b-fp16-4x-b64-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, DeepSpeed-MII, Meta-Llama-3-8B FP16, 128io, batch 1
107.0 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-nichedeepspeedmii-metallama38b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, DeepSpeed-MII, Mistral-7B-v0.1 FP16, 128io, batch 1
116.3 tok/s aggregate (batch 1, single request)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via niche-deepspeed-miibenchmarks/argonne-a100sxm40-nichedeepspeedmii-mistral7bv01-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, llama.cpp, Llama-2-7b-hf FP16, 128io, batch 1
58.9 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llamacpp-llama27bhf-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, llama.cpp, Meta-Llama-3-8B FP16, 128io, batch 1
53.1 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via llama-cppbenchmarks/argonne-a100sxm40-llamacpp-metallama38b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, TensorRT-LLM 0.11.0, Llama-2-7b-hf FP16, 128io, batch 1
169.0 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrtllm-llama27bhf-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, TensorRT-LLM 0.11.0, Meta-Llama-3-8B FP16, 128io, batch 1
143.7 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrtllm-metallama38b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, TensorRT-LLM 0.11.0, Mistral-7B-v0.1 FP16, 128io, batch 1
153.1 tok/s aggregate (batch 1, single request)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via tensorrt-llmbenchmarks/argonne-a100sxm40-tensorrtllm-mistral7bv01-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, vLLM 0.6.1, Llama-2-7b-hf FP16, 128io, batch 1
154.9 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-llama27bhf-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, vLLM 0.6.1, Meta-Llama-3-8B FP16, 128io, batch 1
140.2 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-metallama38b-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: A100-40GB SXM x1, vLLM 0.6.1, Mistral-7B-v0.1 FP16, 128io, batch 1
148.5 tok/s aggregate (batch 1, single request)mistralai/Mistral-7B-v0.1 on nvidia-a100-40gb-sxm4 via vllmbenchmarks/argonne-a100sxm40-vllm-mistral7bv01-fp16-b1-128io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, llama.cpp, Llama-2-7b-hf FP16, 2048io, batch 1
116.4 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on nvidia-gh200 via llama-cppbenchmarks/argonne-gh200-llamacpp-llama27bhf-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, llama.cpp, Meta-Llama-3-8B FP16, 2048io, batch 1
100.5 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-gh200 via llama-cppbenchmarks/argonne-gh200-llamacpp-metallama38b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, llama.cpp, Mistral-7B-v0.1 FP16, 2048io, batch 1
103.8 tok/s aggregate (batch 1, single request)mistralai/Mistral-7B-v0.1 on nvidia-gh200 via llama-cppbenchmarks/argonne-gh200-llamacpp-mistral7bv01-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, llama.cpp, Qwen2-7B FP16, 2048io, batch 1
104.6 tok/s aggregate (batch 1, single request)Qwen/Qwen2-7B on nvidia-gh200 via llama-cppbenchmarks/argonne-gh200-llamacpp-qwen27b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, TensorRT-LLM 0.11.0, Llama-2-7b-hf FP16, 2048io, batch 1
334.6 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-llama27bhf-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, TensorRT-LLM 0.11.0, Meta-Llama-3-8B FP16, 2048io, batch 1
327.8 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-metallama38b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, TensorRT-LLM 0.11.0, Mistral-7B-v0.1 FP16, 2048io, batch 1
348.8 tok/s aggregate (batch 1, single request)mistralai/Mistral-7B-v0.1 on nvidia-gh200 via tensorrt-llmbenchmarks/argonne-gh200-trtllm-mistral7bv01-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, vLLM 0.6.1, DeciLM-7B FP16, 2048io, batch 1
277.5 tok/s aggregate (batch 1, single request)Deci/DeciLM-7B on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-decilm7b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, vLLM 0.6.1, Llama-2-7b-hf FP16, 2048io, batch 1
278.9 tok/s aggregate (batch 1, single request)meta-llama/Llama-2-7b-hf on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-llama27bhf-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, vLLM 0.6.1, Meta-Llama-3-8B FP16, 2048io, batch 1
267.8 tok/s aggregate (batch 1, single request)meta-llama/Meta-Llama-3-8B on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-metallama38b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, vLLM 0.6.1, Qwen1.5-7B FP16, 2048io, batch 1
249.5 tok/s aggregate (batch 1, single request)Qwen/Qwen1.5-7B on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-qwen157b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: GH200 x1, vLLM 0.6.1, Qwen2-7B FP16, 2048io, batch 1
262.8 tok/s aggregate (batch 1, single request)Qwen/Qwen2-7B on nvidia-gh200 via vllmbenchmarks/argonne-gh200-vllm-qwen27b-fp16-b1-2048io-tps
verifiedmedium · 0.80
Argonne: PVC x1 llama.cpp Llama-2-7B FP16 128io b1
22.18 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via llama-cppbenchmarks/argonne-pvc-llamacpp-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 llama.cpp Llama-2-7B FP16 128io b64
43.1 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via llama-cppbenchmarks/argonne-pvc-llamacpp-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 llama.cpp Mixtral-8x7B FP16 128io b1
7.0 tok/s aggregate (batch*(in+out)/latency)mistralai/Mixtral-8x7B-v0.1 on xpu-intel-dc-gpu-max-1550 via llama-cppbenchmarks/argonne-pvc-llamacpp-mixtral-8x7b-fp16-b1-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 vLLM Llama-2-7B FP16 128io b1
13.66 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-llama2-7b-fp16-b1-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 vLLM Llama-2-7B FP16 128io b64
441.4 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-llama2-7b-fp16-b64-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 vLLM Llama-3-8B FP16 128io b1
13.28 tok/s aggregate (batch*(in+out)/latency)meta-llama/Meta-Llama-3-8B on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-llama3-8b-fp16-b1-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 vLLM Mistral-7B FP16 128io b64
428.4 tok/s aggregate (batch*(in+out)/latency)mistralai/Mistral-7B-v0.1 on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-mistral-7b-fp16-b64-128io-tps
verifiedmedium · 0.70
Argonne: PVC x1 vLLM Qwen2-7B FP16 128io b64
477.8 tok/s aggregate (batch*(in+out)/latency)Qwen/Qwen2-7B on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-qwen2-7b-fp16-b64-128io-tps
verifiedmedium · 0.70
Argonne: PVC x2 vLLM Llama-2-7B FP16 128io b64
1894.5 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-llama2-7b-fp16-2x-b64-128io-tps
verifiedmedium · 0.70
Argonne: PVC x4 llama.cpp Llama-2-70B FP16 128io b1
0.29 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-70b-hf on xpu-intel-dc-gpu-max-1550 via llama-cppbenchmarks/argonne-pvc-llamacpp-llama2-70b-fp16-4x-b1-128io-tps
verifiedmedium · 0.70
Argonne: PVC x4 vLLM Llama-2-7B FP16 128io b128
1918.0 tok/s aggregate (batch*(in+out)/latency)meta-llama/Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-pvc-vllm-llama2-7b-fp16-4x-b128-128io-tps
verifiedmedium · 0.70
AWQ INT3-g128 WikiText-2 perplexity delta vs fp16, OPT-6.7B
0.53 perplexity_delta (WikiText-2 ppl points, quantized minus FP16)OPT-6.7B on ? via tinychat-awqbenchmarks/awq-int3g128-wikitext2-ppl-delta-opt-6-7b
verifiedhigh · 0.90
AWQ INT4-g128 WikiText-2 perplexity delta vs fp16, Llama-2 13B
0.09 perplexity_delta (WikiText-2 ppl points, quantized minus FP16)Llama-2 13B on ? via tinychat-awqbenchmarks/awq-int4g128-wikitext2-ppl-delta-llama2-13b
verifiedhigh · 0.85
AWQ INT4-g128 WikiText-2 perplexity delta vs fp16, Llama-2 7B
0.13 perplexity_delta (WikiText-2 ppl points, quantized minus FP16)Llama-2 7B on ? via tinychat-awqbenchmarks/awq-int4g128-wikitext2-ppl-delta-llama2-7b
verifiedhigh · 0.90
AWQ INT4-g128 WikiText-2 perplexity delta vs fp16, Mistral-7B
0.16 perplexity_delta (WikiText-2 ppl points, quantized minus FP16)Mistral-7B-Instruct-v0.2 on ? via tinychat-awqbenchmarks/awq-int4g128-wikitext2-ppl-delta-mistral-7b
verifiedhigh · 0.85
AWS Inf2.48xlarge: 2280 FP16 TFLOPS (12-chip instance)
2280 TFLOPS FP16N/A (peak compute specification) on aws-inferentia2 via alt-neuron-compilerbenchmarks/alt-aws-inf2-fp16-tps
verifiedhigh · 0.85
AWS Neuron docs: bert-base-cased on inf1.xlarge batch, 1095 inf/s (p50 58ms)
1095.0 inferences/s (encoder, batch 8, seq_len 128, fp32)bert-base-cased on aws-inferentia1 via alt-neuron-compilerbenchmarks/aws-neuron-inf1-bert-base-batch-inf-s
verifiedmedium · 0.70
AWS Neuron docs: bert-base-uncased on trn1.2xlarge, real-time, 2085 inf/s (p50 0.94ms)
2085.45 inferences/s (encoder fwd pass, batch-1, latency-optimized)bert-base-uncased on aws-trainium1 via alt-neuron-compilerbenchmarks/aws-neuron-trn1-bert-base-rt-inf-s
verifiedmedium · 0.70
AWS Neuron docs: google/flan-t5-xl on trn1.32xlarge TP=8, 122.52 tok/s (in+out)
122.52 tok/s (in+out tokens, batch-1, TP=8 over 16-chip trn1.32xlarge)google/flan-t5-xl on aws-trainium1 via alt-neuron-compilerbenchmarks/aws-neuron-trn1-google-flan-t5-xl-tp8-128io-tps
verifiedmedium · 0.70
AWS Neuron docs: ResNet-50 on inf1.xlarge batch, 2207 img/s (p50 18ms)
2207.0 images/s (batch 10, fp16)ResNet-50 on aws-inferentia1 via alt-neuron-compilerbenchmarks/aws-neuron-inf1-resnet50-batch-inf-s
verifiedmedium · 0.70
AWS Neuron docs: t5-3b on trn1.32xlarge TP=8, 116.29 tok/s (in+out)
116.29 tok/s (in+out tokens, batch-1, TP=8 over 16-chip trn1.32xlarge)t5-3b on aws-trainium1 via alt-neuron-compilerbenchmarks/aws-neuron-trn1-t5-3b-tp8-128io-tps
verifiedmedium · 0.70
AWS Neuron docs: YOLOv4 on inf1.2xlarge real-time, 180 img/s (p50 40ms)
180.0 images/s (batch 1, fp32)YOLOv4 on aws-inferentia1 via alt-neuron-compilerbenchmarks/aws-neuron-inf1-yolov4-rt-inf-s
verifiedmedium · 0.70
AWS Neuron Inf1: BERT-base-uncased fp32, inf1.xlarge, batch 3, latency-optimized
285.0 inferences/s (latency-optimized, seq len 128)bert-base-uncased on aws-inferentia1 via alt-neuron-compilerbenchmarks/aws-inf1-bert-base-lat-opt-tput
verifiedmedium · 0.80
AWS Neuron Inf1: BERT-base-uncased fp32, inf1.xlarge, batch 6, throughput-optimized
1181.0 inferences/s (batch-throughput-optimized, seq len 128)bert-base-uncased on aws-inferentia1 via alt-neuron-compilerbenchmarks/aws-inf1-bert-base-batch128-tput
verifiedmedium · 0.80
AWS Neuron Inf2: bert-large-uncased fp32, inf2.xlarge, batch 4, throughput-optimized
950.05 inferences/s (batch-throughput-optimized, seq len 128)bert-large-uncased on aws-inferentia2 via alt-neuron-compilerbenchmarks/aws-inf2-bert-large-batch4-tput
verifiedmedium · 0.80
AWS Neuron Inf2: flan-t5-xl, inf2.24xlarge TP8, batch 1, 128in/84out, throughput-optimized
117.61 tokens/s (input+output counted, TP8)google/flan-t5-xl on aws-inferentia2 via alt-neuron-compilerbenchmarks/aws-inf2-flan-t5-xl-tp8-b1-128io-tput
verifiedmedium · 0.80
AWS Neuron Trn1: bert-large-uncased fp32, trn1.2xlarge, batch 4, throughput-optimized
1104.44 inferences/s (batch-throughput-optimized, seq len 128)bert-large-uncased on aws-trainium1 via alt-neuron-compilerbenchmarks/aws-trn1-bert-large-batch4-tput
verifiedmedium · 0.80
AWS Neuron: trn1.32xlarge (Trainium1), NeuronX Distributed, google/flan-t5-xl, TP=8, batch 1, throughput-opt
122.52 tok/s (counts input+output tokens; TP=8 across 16 Trainium chips)google/flan-t5-xl on aws-trainium1 via alt-neuron-compilerbenchmarks/aws-trn1-neuronxd-flant5xl-fp32-tp8-b1-tps
verifiedmedium · 0.70
AWS Neuron: trn1.32xlarge (Trainium1), NeuronX Distributed, t5-3b, TP=8, batch 1, throughput-opt
116.29 tok/s (counts input+output tokens; TP=8 across 16 Trainium chips)t5-3b on aws-trainium1 via alt-neuron-compilerbenchmarks/aws-trn1-neuronxd-t53b-fp32-tp8-b1-tps
verifiedmedium · 0.70
AWS Trn2.48xlarge: 20.8 FP8 PFLOPS (16-chip instance)
20.8 PFLOPS FP8N/A (peak compute specification) on aws-trainium2 via alt-neuron-compilerbenchmarks/alt-aws-trn2-fp8-tps
verifiedhigh · 0.85
BigDL-LLM self-speculative decoding on Intel GPU Max 1550 (1 tile): ~1.35x FP16 next-token-latency speedup
1.35 x vs non-speculative FP16 baseline (mean of Llama-2-7B/13B, Mistral-7B rows)Llama-2-7B/13B + Mistral-7B (avg) on xpu-intel-dc-gpu-max-1550 via hf-optimumbenchmarks/intel-bigdl-max1550-fp16-selfspec-speedup
verifiedlow · 0.55
Blackhole p150 tt-metal: Whisper distil-large-v3 batch-1 tok/s per user
263.4 tok/s per user at batch 1 (equals aggregate at batch 1); vendor marks Blackhole numbers as still-optimizingWhisper distil-large-v3 on tenstorrent-blackhole-p150 via niche-tt-metalbenchmarks/ttmetal-p150-whisper-distil-large-v3-b1-tpsu
verifiedmedium · 0.75
Blackhole p150 tt-metal: Whisper distil-large-v3 batch-1 TTFT
63.0 ms time-to-first-token, batch 1; vendor marks Blackhole numbers as still-optimizingWhisper distil-large-v3 on tenstorrent-blackhole-p150 via niche-tt-metalbenchmarks/ttmetal-p150-whisper-distil-large-v3-b1-ttft
verifiedmedium · 0.75
cadamcat: 1x A100-80GB vs 2x RX 7900 XT TP=2 batch-1 decode speedup ratio (vLLM)
1.4 x (A100 tok/s vs dual-7900XT tok/s, batch-1 decode, 500-32K ctx ladder)gemma-4-12B-it (dense) + gemma-4-26B-A4B (MoE) on nvidia-a100-80gb-sxm4, amd-radeon-rx-7900-xtx via vllmbenchmarks/cadamcat-a100-vs-dual-7900xt-decode-speedup
verifiedmedium · 0.80
Cisco UCS C880A M8 8xH200-SXM at a DECLARED H200 TGP of 700W, MLPerf Inference v5.1 Offline: Llama 2 70B
34548.2 output tokens/s (aggregate, harness 'Tokens per second')Llama 2 70B (MLPerf v5.1 submission, -99.9 accuracy track, FP8 weights) on nvidia-h200-sxm via tensorrt-llmbenchmarks/bench-mlperf-v5-1-cisco-h200-tgp700w-llama2-70b-offline
verifiedhigh · 0.95
Coral Dev Board Edge TPU: MobileNet v2 224x224, 2.6ms -> 384.6 inf/s
384.6 inferences/s (from 2.6 ms per inference, vendor C++ benchmark)MobileNet v2 224x224 TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-devboard-mobilenetv2-inf-s
verifiedmedium · 0.80
Coral Edge TPU (USB3): EfficientNet-EdgeTPU-S, 5.1ms -> 196.1 inf/s
196.1 inferences/s (from 5.1 ms per inference, vendor C++ benchmark)EfficientNet-EdgeTPU-S TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-efficientnet-s-inf-s
verifiedmedium · 0.80
Coral Edge TPU (USB3): Inception v4 299x299, 85ms -> 11.8 inf/s
11.8 inferences/s (from 85.0 ms per inference, vendor C++ benchmark)Inception v4 299x299 TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-inceptionv4-inf-s
verifiedmedium · 0.80
Coral Edge TPU (USB3): MobileNet v1 224x224, 2.4ms -> 416.7 inf/s
416.7 inferences/s (from 2.4 ms per inference, vendor C++ benchmark)MobileNet v1 224x224 TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-mobilenetv1-inf-s
verifiedmedium · 0.80
Coral Edge TPU (USB3): MobileNet v2 224x224, 2.6ms -> 384.6 inf/s
384.6 inferences/s (from 2.6 ms per inference, vendor C++ benchmark)MobileNet v2 224x224 TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-mobilenetv2-inf-s
verifiedmedium · 0.80
Coral Edge TPU (USB3): MobileNet v2 SSD 224x224, 7.2ms -> 138.9 inf/s
138.9 inferences/s (from 7.2 ms per inference, vendor C++ benchmark)MobileNet v2 SSD 224x224 TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-mobilenetv2ssd-inf-s
verifiedmedium · 0.80
Coral Edge TPU (USB3): ResNet-50 v1 299x299, 49ms -> 20.4 inf/s
20.4 inferences/s (from 49.0 ms per inference, vendor C++ benchmark)ResNet-50 v1 299x299 TFLite INT8 on google-coral-edge-tpu via local-litertbenchmarks/coral-edgetpu-resnet50v1-inf-s
verifiedmedium · 0.80
CoreWeave GB200 NVL72, MLPerf Inference v6.1 Offline: Llama 2 70B across a full 72-GPU rack (18 nodes)
894002 output tokens/s (aggregate across all 72 GPUs, harness 'Tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-gb200-nvl72 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-coreweave-gb200-nvl72-llama2-70b-offline
verifiedhigh · 0.95
CoreWeave GB200 NVL72, MLPerf Inference v6.1 Server: Llama 2 70B across a full 72-GPU rack (18 nodes) under the Server latency constraint
910621.36 output tokens/s (aggregate across all 72 GPUs, harness 'Completed tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-gb200-nvl72 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-coreweave-gb200-nvl72-llama2-70b-server
verifiedhigh · 0.95
CoreWeave GB300 NVL72, MLPerf Inference v6.1 Offline: Llama 2 70B across a full 72-GPU rack (18 nodes), the largest single v6.1 LLM result
1136100 output tokens/s (aggregate across all 72 GPUs; harness prints 1.1361e+06 in scientific notation)Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-b300 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-coreweave-gb300-nvl72-llama2-70b-offline
verifiedhigh · 0.95
CoreWeave GB300 NVL72, MLPerf Inference v6.1 Server: Llama 2 70B across a full 72-GPU rack (18 nodes) under the Server latency constraint
944902.14 output tokens/s (aggregate across all 72 GPUs, harness 'Completed tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-b300 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-coreweave-gb300-nvl72-llama2-70b-server
verifiedhigh · 0.95
DeepSeek-V3 MTP speculative decoding in SGLang on 8x AMD MI300X: batch 1 latency speedup
2.11 speedup_ratio: 2.11x speedup over baseline without MTP (end-to-end latency reduced from 17348.25 ms to 8229.37 ms per request) at concurrency 1. BASELINE: DeepSeek-V3 served without MTP on identical 8x MI300X node in SGLang.deepseek-ai/DeepSeek-V3 on amd-instinct-mi300x via sglangbenchmarks/sd-deepseek-v3-mtp-sglang-mi300x-b1
verifiedhigh · 0.90
DeepSeek-V3 MTP speculative decoding in SGLang on 8x AMD MI300X: batch 64 latency speedup
1.25 speedup_ratio: 1.25x speedup over baseline without MTP (end-to-end latency reduced from 43848.20 ms to 35074.26 ms per request) at concurrency 64. BASELINE: DeepSeek-V3 served without MTP on identical 8x MI300X node in SGLang.deepseek-ai/DeepSeek-V3 on amd-instinct-mi300x via sglangbenchmarks/sd-deepseek-v3-mtp-sglang-mi300x-b64
verifiedhigh · 0.90
Dell PowerEdge XE9785 8xMI355X at a 1000W PER-GPU POWER CAP, MLPerf Inference v6.0 Offline: Llama 2 70B, and its uncapped twin
81870.5 output tokens/s (aggregate under a 1000W per-GPU cap; v6.0 result ID 6.0-0033)Llama 2 70B (MLPerf v6.0 submission, -99.9 accuracy track, FP8 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-0-dell-mangoboost-mi355x-powercap-1000w-offline
verifiedhigh · 0.95
Dell XE9785L 8xMI355X, MLPerf Inference v6.1 INTERACTIVE: Llama 3.1 8B while holding an inter-token latency cap
130539.57 output tokens/s (aggregate, harness 'Completed tokens per second')Llama 3.1 8B (MLPerf v6.1 submission, FP8 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-dell-mi355x-llama3-1-8b-interactive
verifiedhigh · 0.95
Dell XE9785L 8xMI355X, MLPerf Inference v6.1 Offline: Llama 3.1 8B, third-party OEM reproduction of AMD silicon
158458 output tokens/s (aggregate, harness 'Tokens per second')Llama 3.1 8B (MLPerf v6.1 submission, FP8 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-dell-mi355x-llama3-1-8b-offline
verifiedhigh · 0.95
Dell XE9785L 8xMI355X, MLPerf Inference v6.1 Server: Llama 3.1 8B aggregate output throughput under the Server latency constraint
148563.48 output tokens/s (aggregate, harness 'Completed tokens per second')Llama 3.1 8B (MLPerf v6.1 submission, FP8 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-dell-mi355x-llama3-1-8b-server
verifiedhigh · 0.95
Dual EPYC 9334 (2x32c) llama.cpp DeepSeek-8B Q4 decode
27.8 tok/sDeepSeek-8B Q4 on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/comm-dual-epyc-9334-deepseek-8b-decode
verifiedmedium · 0.75
Dual EPYC 9334 (2x32c) llama.cpp gpt-oss-20b Q4 decode
26.0 tok/sgpt-oss-20b Q4 on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/comm-dual-epyc-9334-gpt-oss-20b-q4-decode
verifiedmedium · 0.70
Dual EPYC 9334 (32c x2, Genoa) gpt-oss-20b Q4 decode
26.0 tok/s decode (dual-socket, NUMA-distributed)gpt-oss-20b Q4-class on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/comm-dual-epyc-9334-gptoss-20b-decode
verifiedmedium · 0.75
EAGLE-1 acceptance length and speedup on Vicuna-13B (temperature=0)
3.82 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-eagle1-acceptance-vicuna13b-t0
verifiedhigh · 0.90
EAGLE-2 acceptance length and speedup on LLaMA2-Chat 70B (temperature=0)
4.48 tokens_per_cycleLLaMA2-Chat 70B on ? via vllmbenchmarks/sd-eagle2-acceptance-llama2-70b-t0
verifiedhigh · 0.90
EAGLE-2 acceptance length and speedup on LLaMA3-Instruct 8B (temperature=0)
4.53 tokens_per_cycleLLaMA3-Instruct 8B on ? via vllmbenchmarks/sd-eagle2-acceptance-llama3-8b-t0
verifiedhigh · 0.90
EAGLE-2 acceptance length and speedup on Vicuna-13B (temperature=0)
4.65 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-eagle2-acceptance-vicuna13b-t0
verifiedhigh · 0.90
EAGLE-2 acceptance length and speedup on Vicuna-13B (temperature=1)
4.26 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-eagle2-acceptance-vicuna13b-t1
verifiedhigh · 0.90
EAGLE-3 throughput improvement in SGLang at batch size 64
1.38 value 1.38 with metric speedup_ratio: 1.38x the BASELINE throughput at batch size 64 in SGLang, NOT 1.38 output tokens/s. BASELINE: the same SGLang build without EAGLE-3, batch 64, draft tree depth 8.LLaMA-Instruct 3.1 8B on ? via sglangbenchmarks/sd-eagle3-throughput-sglang-batch64
verifiedhigh · 0.85
Enabling batch-invariant (deterministic) kernels in vLLM costs 1.6x-2.1x wall clock: Qwen-3-8B, 1 GPU, 1000 sequences at 90-110 output tokens
55.0 seconds for 1000 sequences of 90-110 output tokens on ONE GPU - unoptimized deterministic vLLM; baseline vLLM default is 26 s and the improved-attention-kernel variant is 42 s, i.e. 2.11x and 1.62xQwen-3-8B on nvidia-h100-sxm via vllmbenchmarks/vllm-batch-invariant-determinism-throughput-cost-1-6x
verifiedmedium · 0.82
EPYC 9454P (12ch DDR5) llama.cpp Qwen3-30B-A3B MoE decode
63.1 tok/s (single-stream)Qwen3-30B-A3B (MoE ~3B active) Q4_K_M-class on amd-epyc-9454p-genoa via llama-cppbenchmarks/comm-epyc-9454p-qwen3-moe-30b-decode
verifiedmedium · 0.75
EPYC 9454P 48c Qwen3-Next-80B token-gen
11.76 tok/s decode 12ch DDR5Qwen3-Next-80B (~3B active) on amd-epyc-9454p-genoa via llama-cppbenchmarks/comm-epyc-9454p-qwen3-next-80b-tg
verifiedmedium · 0.75
EPYC 9554 (Genoa, 12ch DDR5) QwQ-32B Q4_K_M decode
13.9 tok/s decode, single-streamQwQ-32B Q4_K_M on amd-epyc-9554-genoa via llama-cppbenchmarks/comm-epyc-9554-qwq-32b-q4-decode
verifiedmedium · 0.75
EPYC 9554 (Genoa) DeepSeek-R1-70B Q4_K_M decode
7.11 tok/s decode, single-streamDeepSeek-R1-70B Q4_K_M on amd-epyc-9554-genoa via llama-cppbenchmarks/comm-epyc-9554-deepseek-r1-70b-q4-decode
verifiedmedium · 0.75
EPYC 9554 64c Llama-3.3-70B Q4_K_M token-gen
7.12 tok/s decode (--numa distribute)Llama-3.3-70B on amd-epyc-9554-genoa via llama-cppbenchmarks/comm-epyc-9554-llama33-70b-q4-tg
verifiedmedium · 0.75
EPYC 9554 DeepSeek R1 Distill Llama 70B (Q4_K_M) decode
7.11 tok/s (tg128)DeepSeek R1 Distill Llama 70B (Q4_K_M) on amd-epyc-9554-genoa via llama-cppbenchmarks/comm-epyc-9554-r1-70b-q4-decode
verifiedmedium · 0.70
EPYC 9554 DeepSeek R1 Distill Llama 8B (Q4_K_M) decode
49.96 tok/s (tg128)DeepSeek R1 Distill Llama 8B (Q4_K_M) on amd-epyc-9554-genoa via llama-cppbenchmarks/comm-epyc-9554-r1-8b-q4-decode
verifiedmedium · 0.70
EPYC 9554 DeepSeek-R1-Distill-Llama-70B Q4_K_M tg128 (ahelpme)
7.14 tok/s (tg decode, batch 1, per-request)DeepSeek-R1-Distill-Llama-70B on amd-epyc-9554-genoa via llama-cppbenchmarks/ah-epyc9554-deepseek-r1-llama-70b-q4km-tg128
verifiedmedium · 0.70
EPYC 9554 DeepSeek-R1-Distill-Llama-70B Q4_K_M tg2048 (ahelpme)
7.03 tok/s (tg decode, batch 1, per-request)DeepSeek-R1-Distill-Llama-70B on amd-epyc-9554-genoa via llama-cppbenchmarks/ah-epyc9554-deepseek-r1-llama-70b-q4km-tg2048
verifiedmedium · 0.70
EPYC 9554 DeepSeek-R1-Distill-Llama-8B Q4_K_M tg128 (ahelpme)
49.96 tok/s (tg decode, batch 1, per-request)DeepSeek-R1-Distill-Llama-8B on amd-epyc-9554-genoa via llama-cppbenchmarks/ah-epyc9554-deepseek-r1-llama-8b-q4km-tg128
verifiedmedium · 0.70
EPYC 9554 DeepSeek-R1-Distill-Llama-8B Q4_K_M tg2048 (ahelpme)
48.82 tok/s (tg decode, batch 1, per-request)DeepSeek-R1-Distill-Llama-8B on amd-epyc-9554-genoa via llama-cppbenchmarks/ah-epyc9554-deepseek-r1-llama-8b-q4km-tg2048
verifiedmedium · 0.70
EPYC 9554 Llama 3.1 8B Instruct (Q4_K_M) decode
49.64 tok/s (tg128)Llama 3.1 8B Instruct (Q4_K_M) on amd-epyc-9554-genoa via llama-cppbenchmarks/comm-epyc-9554-llama31-8b-q4-decode
verifiedmedium · 0.70
EPYC 9554 Llama-3.1-8B-Instruct Q4_K_M tg128 (ahelpme)
49.64 tok/s (tg decode, batch 1, per-request)Llama-3.1-8B-Instruct on amd-epyc-9554-genoa via llama-cppbenchmarks/ah-epyc9554-llama31-8b-q4km-tg128
verifiedmedium · 0.70
EPYC 9554 QwQ-32B Q4_K_M tg128 (ahelpme)
13.94 tok/s (tg decode, batch 1, per-request)QwQ-32B on amd-epyc-9554-genoa via llama-cppbenchmarks/ah-epyc9554-qwq-32b-q4km-tg128
verifiedmedium · 0.70
EPYC Genoa, one 64GB DIMM, 34B Q4_0 decode (llama.cpp #3167)
1.5 tok/s, single-request llama.cpp CPU decode, one of twelve channels populated34B Q4_0 (family not named in the thread, ~34B dense) on amd-epyc-9554-genoa via llama-cppbenchmarks/llamacpp-epyc-genoa-one-dimm-34b-q4-decode
verifiedlow · 0.60
EPYC Genoa, two DIMMs, 34B Q4_0 decode (llama.cpp #3167)
4.0 tok/s, single-request llama.cpp CPU decode, two of twelve channels populated34B Q4_0 (family not named in the thread, ~34B dense) on amd-epyc-9554-genoa via llama-cppbenchmarks/llamacpp-epyc-genoa-two-dimm-34b-q4-decode
verifiedlow · 0.60
Geerling ai-benchmarks: AMD Radeon AI Pro R9700 (in Intel 265K PC), llama.cpp Vulkan, DeepSeek-R1-14b (Q4 distill), 53.72 tok/s eval rate, 330.8W peak
53.72 tok/s eval ratedeepseek-ai/DeepSeek-R1-Distill-Qwen-14B (Q4 GGUF) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/geerling-r9700-deepseekr1-14b-vulkan-llamacpp-decode
verifiedmedium · 0.75
Geerling ai-benchmarks: M1 Ultra (64-core GPU, 128GB), Ollama, Llama-3.1-70b (Q4), 9.84 tok/s eval rate
9.84 tok/s eval rate (Ollama-reported decode)llama3.1:70b (Ollama Q4_K_M tag) on apple-m1-ultra via ollamabenchmarks/geerling-m1ultra-64gpu-llama31-70b-q4-ollama-decode
verifiedmedium · 0.70
Geerling ai-benchmarks: Mac Studio M3 Ultra 512GB, Ollama, DeepSeek-R1-14b (Q4 distill), 51.85 tok/s eval rate, 227W peak
51.85 tok/s eval rate (Ollama-reported decode)deepseek-ai/DeepSeek-R1-Distill-Qwen-14B (Ollama default Q4) on apple-m3-ultra via ollamabenchmarks/geerling-m3ultra-deepseekr1-14b-q4-ollama-decode
verifiedmedium · 0.75
Geerling ai-benchmarks: Mac Studio M3 Ultra 512GB, Ollama, Llama-3.1-70b (Q4), 14.08 tok/s eval rate, 243W peak
14.08 tok/s eval rate (Ollama-reported decode)llama3.1:70b (Ollama Q4_K_M tag) on apple-m3-ultra via ollamabenchmarks/geerling-m3ultra-llama31-70b-q4-ollama-decode
verifiedmedium · 0.75
Geerling ai-benchmarks: RTX 4090 (in Intel 265K PC), Ollama, DeepSeek-R1-14b (Q4 distill), 92.49 tok/s eval rate, 464W peak
92.49 tok/s eval rate (Ollama-reported decode)deepseek-ai/DeepSeek-R1-Distill-Qwen-14B (Ollama default Q4) on nvidia-rtx-4090 via ollamabenchmarks/geerling-rtx4090-deepseekr1-14b-q4-ollama-decode
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pc host): llama-2-13b Q4_K_M pp2048 1516.04 tok/s
1516.04 tok/s (pp2048)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pc-llama-2-13b-q4km-pp2048
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pc host): llama-2-13b Q4_K_M pp2048+tg128 564.96 tok/s
564.96 tok/s (pp2048+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pc-llama-2-13b-q4km-pp2048-tg128
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pc host): llama-2-13b Q4_K_M pp512 1565.08 tok/s
1565.08 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pc-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pc host): llama-2-13b Q4_K_M tg128 60.51 tok/s
60.51 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pc-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pi host): llama-2-13b Q4_K_M pp2048 1452.11 tok/s
1452.11 tok/s (pp2048)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pi-llama-2-13b-q4km-pp2048
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pi host): llama-2-13b Q4_K_M pp2048+tg128 551.31 tok/s
551.31 tok/s (pp2048+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pi-llama-2-13b-q4km-pp2048-tg128
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pi host): llama-2-13b Q4_K_M pp512 1493.64 tok/s
1493.64 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pi-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp CUDA on rtx2080ti (pi host): llama-2-13b Q4_K_M tg128 59.45 tok/s
59.45 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-cuda-rtx2080ti-pi-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 719.90 tok/s
719.9 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 370.67 tok/s
370.67 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 933.19 tok/s
933.19 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 27.35 tok/s
27.35 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): gpt-oss-20b Q4_K_M pp512 1459.87 tok/s
1459.87 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): gpt-oss-20b Q4_K_M tg128 77.14 tok/s
77.14 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-13b Q4_K_M pp2048 1381.92 tok/s
1381.92 tok/s (pp2048)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-13b-q4km-pp2048
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-13b Q4_K_M pp512 1421.43 tok/s
1421.43 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-13b Q4_K_M tg128 54.85 tok/s
54.85 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-7b Q4_K_M pp4096 1328.54 tok/s
1328.54 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-7b Q4_K_M pp4096+tg128 611.88 tok/s
611.88 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-7b Q4_K_M pp512 1787.60 tok/s
1787.6 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-2-7b Q4_K_M tg128 56.53 tok/s
56.53 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-32-3b Q4_K_M pp4096 2143.23 tok/s
2143.23 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-32-3b Q4_K_M pp4096+tg128 1348.80 tok/s
1348.8 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-32-3b Q4_K_M pp512 3185.00 tok/s
3185.0 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): llama-32-3b Q4_K_M tg128 99.24 tok/s
99.24 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): tinyllama-11b Q4_K_M pp4096 4128.25 tok/s
4128.25 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): tinyllama-11b Q4_K_M pp4096+tg128 2938.08 tok/s
2938.08 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): tinyllama-11b Q4_K_M pp512 7389.30 tok/s
7389.3 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pc host): tinyllama-11b Q4_K_M tg128 234.18 tok/s
234.18 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pc-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 185.78 tok/s
185.78 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 143.29 tok/s
143.29 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 222.49 tok/s
222.49 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 24.63 tok/s
24.63 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): gpt-oss-20b Q4_K_M pp512 381.67 tok/s
381.67 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): gpt-oss-20b Q4_K_M tg128 55.32 tok/s
55.32 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-13b Q4_K_M pp2048 1312.45 tok/s
1312.45 tok/s (pp2048)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-13b-q4km-pp2048
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-13b Q4_K_M pp512 1350.00 tok/s
1350.0 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-13b Q4_K_M tg128 53.78 tok/s
53.78 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-7b Q4_K_M pp4096 339.70 tok/s
339.7 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-7b Q4_K_M pp4096+tg128 254.95 tok/s
254.95 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-7b Q4_K_M pp512 422.37 tok/s
422.37 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-2-7b Q4_K_M tg128 50.63 tok/s
50.63 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-32-3b Q4_K_M pp4096 563.29 tok/s
563.29 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-32-3b Q4_K_M pp4096+tg128 447.85 tok/s
447.85 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-32-3b Q4_K_M pp512 743.39 tok/s
743.39 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): llama-32-3b Q4_K_M tg128 92.60 tok/s
92.6 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): tinyllama-11b Q4_K_M pp4096 1101.43 tok/s
1101.43 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): tinyllama-11b Q4_K_M pp4096+tg128 974.24 tok/s
974.24 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): tinyllama-11b Q4_K_M pp512 1641.37 tok/s
1641.37 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx2080ti (pi host): tinyllama-11b Q4_K_M tg128 213.96 tok/s
213.96 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/geerling-vulkan-rtx2080ti-pi-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 783.57 tok/s
783.57 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 403.45 tok/s
403.45 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 1015.72 tok/s
1015.72 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 29.77 tok/s
29.77 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): gpt-oss-20b Q4_K_M pp512 1588.97 tok/s
1588.97 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): gpt-oss-20b Q4_K_M tg128 83.96 tok/s
83.96 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-13b Q4_K_M pp4096 842.69 tok/s
842.69 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-13b Q4_K_M pp4096+tg128 378.36 tok/s
378.36 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-13b Q4_K_M pp512 1060.76 tok/s
1060.76 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-13b Q4_K_M tg128 32.32 tok/s
32.32 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-7b Q4_K_M pp4096 1446.03 tok/s
1446.03 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-7b Q4_K_M pp4096+tg128 665.99 tok/s
665.99 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-7b Q4_K_M pp512 1945.69 tok/s
1945.69 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-2-7b Q4_K_M tg128 61.53 tok/s
61.53 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-32-3b Q4_K_M pp4096 2332.77 tok/s
2332.77 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-32-3b Q4_K_M pp4096+tg128 1468.09 tok/s
1468.09 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-32-3b Q4_K_M pp512 3466.51 tok/s
3466.51 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): llama-32-3b Q4_K_M tg128 108.02 tok/s
108.02 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): tinyllama-11b Q4_K_M pp4096 4493.45 tok/s
4493.45 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): tinyllama-11b Q4_K_M pp4096+tg128 3197.91 tok/s
3197.91 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): tinyllama-11b Q4_K_M pp512 8042.78 tok/s
8042.78 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pc host): tinyllama-11b Q4_K_M tg128 254.89 tok/s
254.89 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pc-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 202.21 tok/s
202.21 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 155.98 tok/s
155.98 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 242.17 tok/s
242.17 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 26.81 tok/s
26.81 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): gpt-oss-20b Q4_K_M pp512 415.43 tok/s
415.43 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): gpt-oss-20b Q4_K_M tg128 60.21 tok/s
60.21 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-13b Q4_K_M pp4096 217.36 tok/s
217.36 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-13b Q4_K_M pp4096+tg128 157.57 tok/s
157.57 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-13b Q4_K_M pp512 259.71 tok/s
259.71 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-13b Q4_K_M tg128 29.01 tok/s
29.01 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-7b Q4_K_M pp4096 369.73 tok/s
369.73 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-7b Q4_K_M pp4096+tg128 277.51 tok/s
277.51 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-7b Q4_K_M pp512 459.73 tok/s
459.73 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-2-7b Q4_K_M tg128 55.11 tok/s
55.11 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-32-3b Q4_K_M pp4096 613.12 tok/s
613.12 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-32-3b Q4_K_M pp4096+tg128 487.46 tok/s
487.46 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-32-3b Q4_K_M pp512 809.14 tok/s
809.14 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): llama-32-3b Q4_K_M tg128 100.79 tok/s
100.79 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): tinyllama-11b Q4_K_M pp4096 1201.56 tok/s
1201.56 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): tinyllama-11b Q4_K_M pp4096+tg128 1060.62 tok/s
1060.62 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): tinyllama-11b Q4_K_M pp512 1786.72 tok/s
1786.72 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3060 (pi host): tinyllama-11b Q4_K_M tg128 232.86 tok/s
232.86 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3060 via llama-cppbenchmarks/geerling-vulkan-rtx3060-pi-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 1568.14 tok/s
1568.14 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 806.91 tok/s
806.91 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 2032.71 tok/s
2032.71 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 59.54 tok/s
59.54 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): gpt-oss-20b Q4_K_M pp512 3179.22 tok/s
3179.22 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): gpt-oss-20b Q4_K_M pp512+tg128 1142.29 tok/s
1142.29 tok/s (pp512+tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-gpt-oss-20b-q4km-pp512-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): gpt-oss-20b Q4_K_M tg128 167.86 tok/s
167.86 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-13b Q4_K_M pp4096 1686.11 tok/s
1686.11 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-13b Q4_K_M pp4096+tg128 756.69 tok/s
756.69 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-13b Q4_K_M pp512 2122.79 tok/s
2122.79 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-13b Q4_K_M tg128 64.63 tok/s
64.63 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-7b Q4_K_M pp4096 2893.84 tok/s
2893.84 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-7b Q4_K_M pp4096+tg128 1332.01 tok/s
1332.01 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-7b Q4_K_M pp512 3893.83 tok/s
3893.83 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-2-7b Q4_K_M tg128 122.99 tok/s
122.99 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-32-3b Q4_K_M pp4096 4662.92 tok/s
4662.92 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-32-3b Q4_K_M pp4096+tg128 2934.69 tok/s
2934.69 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-32-3b Q4_K_M pp512 6944.67 tok/s
6944.67 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): llama-32-3b Q4_K_M tg128 216.01 tok/s
216.01 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): tinyllama-11b Q4_K_M pp4096 8984.35 tok/s
8984.35 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): tinyllama-11b Q4_K_M pp4096+tg128 6399.16 tok/s
6399.16 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): tinyllama-11b Q4_K_M pp512 16101.00 tok/s
16101.0 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pc host): tinyllama-11b Q4_K_M tg128 509.73 tok/s
509.73 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pc-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 247.74 tok/s
247.74 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 188.18 tok/s
188.18 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 296.91 tok/s
296.91 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 32.69 tok/s
32.69 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): gpt-oss-20b Q4_K_M pp512 509.64 tok/s
509.64 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): gpt-oss-20b Q4_K_M tg128 73.61 tok/s
73.61 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-13b Q4_K_M pp4096 266.24 tok/s
266.24 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-13b Q4_K_M pp4096+tg128 190.13 tok/s
190.13 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-13b Q4_K_M pp512 318.36 tok/s
318.36 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-13b Q4_K_M tg128 35.37 tok/s
35.37 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-7b Q4_K_M pp4096 453.13 tok/s
453.13 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-7b Q4_K_M pp4096+tg128 334.63 tok/s
334.63 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-7b Q4_K_M pp512 563.27 tok/s
563.27 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-2-7b Q4_K_M tg128 67.16 tok/s
67.16 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-32-3b Q4_K_M pp4096 750.97 tok/s
750.97 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-32-3b Q4_K_M pp4096+tg128 589.03 tok/s
589.03 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-32-3b Q4_K_M pp512 990.16 tok/s
990.16 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): llama-32-3b Q4_K_M tg128 122.95 tok/s
122.95 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): tinyllama-11b Q4_K_M pp4096 1472.81 tok/s
1472.81 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): tinyllama-11b Q4_K_M pp4096+tg128 1285.54 tok/s
1285.54 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): tinyllama-11b Q4_K_M pp512 2193.87 tok/s
2193.87 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx3080ti (pi host): tinyllama-11b Q4_K_M tg128 286.13 tok/s
286.13 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-3080-ti via llama-cppbenchmarks/geerling-vulkan-rtx3080ti-pi-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 1731.79 tok/s
1731.79 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 806.77 tok/s
806.77 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 2606.48 tok/s
2606.48 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 49.74 tok/s
49.74 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): gpt-oss-20b Q4_K_M pp512 4420.61 tok/s
4420.61 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): gpt-oss-20b Q4_K_M tg128 163.14 tok/s
163.14 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-13b Q4_K_M pp4096 1878.24 tok/s
1878.24 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-13b Q4_K_M pp4096+tg128 762.19 tok/s
762.19 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-13b Q4_K_M pp512 2814.73 tok/s
2814.73 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-13b Q4_K_M tg128 53.38 tok/s
53.38 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-7b Q4_K_M pp4096 3184.71 tok/s
3184.71 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-7b Q4_K_M pp4096+tg128 1343.28 tok/s
1343.28 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-7b Q4_K_M pp512 5062.43 tok/s
5062.43 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-2-7b Q4_K_M tg128 103.69 tok/s
103.69 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-32-3b Q4_K_M pp4096 5830.35 tok/s
5830.35 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-32-3b Q4_K_M pp4096+tg128 3668.37 tok/s
3668.37 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-32-3b Q4_K_M pp512 8680.84 tok/s
8680.84 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): llama-32-3b Q4_K_M tg128 270.02 tok/s
270.02 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): tinyllama-11b Q4_K_M pp4096 11011.13 tok/s
11011.13 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): tinyllama-11b Q4_K_M pp4096+tg128 7843.82 tok/s
7843.82 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): tinyllama-11b Q4_K_M pp512 19735.84 tok/s
19735.84 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pc host): tinyllama-11b Q4_K_M tg128 624.62 tok/s
624.62 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pc-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 310.32 tok/s
310.32 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 235.78 tok/s
235.78 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 371.97 tok/s
371.97 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 40.94 tok/s
40.94 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): gpt-oss-20b Q4_K_M pp512 638.57 tok/s
638.57 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): gpt-oss-20b Q4_K_M tg128 92.19 tok/s
92.19 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-13b Q4_K_M pp4096 333.47 tok/s
333.47 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-13b Q4_K_M pp4096+tg128 238.22 tok/s
238.22 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-13b Q4_K_M pp512 398.84 tok/s
398.84 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-13b Q4_K_M tg128 44.30 tok/s
44.3 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-7b Q4_K_M pp4096 567.55 tok/s
567.55 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-7b Q4_K_M pp4096+tg128 419.21 tok/s
419.21 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-7b Q4_K_M pp512 705.51 tok/s
705.51 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-2-7b Q4_K_M tg128 84.15 tok/s
84.15 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-32-3b Q4_K_M pp4096 941.26 tok/s
941.26 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-32-3b Q4_K_M pp4096+tg128 738.07 tok/s
738.07 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-32-3b Q4_K_M pp512 1241.36 tok/s
1241.36 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): llama-32-3b Q4_K_M tg128 154.12 tok/s
154.12 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): tinyllama-11b Q4_K_M pp4096 1843.92 tok/s
1843.92 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): tinyllama-11b Q4_K_M pp4096+tg128 1609.44 tok/s
1609.44 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): tinyllama-11b Q4_K_M pp512 2746.80 tok/s
2746.8 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtx4070ti (pi host): tinyllama-11b Q4_K_M tg128 358.16 tok/s
358.16 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/geerling-vulkan-rtx4070ti-pi-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 1569.71 tok/s
1569.71 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 807.72 tok/s
807.72 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 2034.74 tok/s
2034.74 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 59.60 tok/s
59.6 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): gpt-oss-20b Q4_K_M pp4096 2646.57 tok/s
2646.57 tok/s (pp4096)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-gpt-oss-20b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): gpt-oss-20b Q4_K_M pp4096+tg128 1362.07 tok/s
1362.07 tok/s (pp4096+tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-gpt-oss-20b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): gpt-oss-20b Q4_K_M pp512 3182.40 tok/s
3182.4 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): gpt-oss-20b Q4_K_M tg128 168.03 tok/s
168.03 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-13b Q4_K_M pp4096 1687.80 tok/s
1687.8 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-13b Q4_K_M pp4096+tg128 757.45 tok/s
757.45 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-13b Q4_K_M pp512 2124.92 tok/s
2124.92 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-13b Q4_K_M tg128 64.69 tok/s
64.69 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-7b Q4_K_M pp4096 2896.74 tok/s
2896.74 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-7b Q4_K_M pp4096+tg128 1333.34 tok/s
1333.34 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-7b Q4_K_M pp512 3897.73 tok/s
3897.73 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-2-7b Q4_K_M tg128 123.11 tok/s
123.11 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-32-3b Q4_K_M pp4096 4667.59 tok/s
4667.59 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-32-3b Q4_K_M pp4096+tg128 2937.62 tok/s
2937.62 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-32-3b Q4_K_M pp512 6951.62 tok/s
6951.62 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): llama-32-3b Q4_K_M tg128 216.23 tok/s
216.23 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): tinyllama-11b Q4_K_M pp4096 8993.35 tok/s
8993.35 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): tinyllama-11b Q4_K_M pp4096+tg128 6405.55 tok/s
6405.55 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): tinyllama-11b Q4_K_M pp512 16117.14 tok/s
16117.14 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pc host): tinyllama-11b Q4_K_M tg128 510.24 tok/s
510.24 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pc-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096 247.99 tok/s
247.99 tok/s (pp4096)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp4096+tg128 188.37 tok/s
188.37 tok/s (pp4096+tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-deepseek-r1-distill-qwen-14b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M pp512 297.20 tok/s
297.2 tok/s (pp512)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-deepseek-r1-distill-qwen-14b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): deepseek-r1-distill-qwen-14b Q4_K_M tg128 32.72 tok/s
32.72 tok/s (tg128)DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf (8.37 GiB, 14.77 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-deepseek-r1-distill-qwen-14b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): gpt-oss-20b Q4_K_M pp4096 424.23 tok/s
424.23 tok/s (pp4096)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-gpt-oss-20b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): gpt-oss-20b Q4_K_M pp4096+tg128 337.59 tok/s
337.59 tok/s (pp4096+tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-gpt-oss-20b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): gpt-oss-20b Q4_K_M pp512 510.14 tok/s
510.14 tok/s (pp512)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-gpt-oss-20b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): gpt-oss-20b Q4_K_M tg128 73.68 tok/s
73.68 tok/s (tg128)gpt-oss-20b-Q4_K_M.gguf (10.81 GiB, 20.91 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-gpt-oss-20b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-13b Q4_K_M pp4096 266.51 tok/s
266.51 tok/s (pp4096)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-13b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-13b Q4_K_M pp4096+tg128 190.32 tok/s
190.32 tok/s (pp4096+tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-13b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-13b Q4_K_M pp512 318.68 tok/s
318.68 tok/s (pp512)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-13b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-13b Q4_K_M tg128 35.41 tok/s
35.41 tok/s (tg128)llama-2-13b.Q4_K_M.gguf (7.33 GiB, 13.02 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-13b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-7b Q4_K_M pp4096 453.58 tok/s
453.58 tok/s (pp4096)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-7b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-7b Q4_K_M pp4096+tg128 334.96 tok/s
334.96 tok/s (pp4096+tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-7b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-7b Q4_K_M pp512 563.84 tok/s
563.84 tok/s (pp512)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-7b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-2-7b Q4_K_M tg128 67.23 tok/s
67.23 tok/s (tg128)llama-2-7b.Q4_K_M.gguf (3.80 GiB, 6.74 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-2-7b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-32-3b Q4_K_M pp4096 751.72 tok/s
751.72 tok/s (pp4096)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-32-3b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-32-3b Q4_K_M pp4096+tg128 589.62 tok/s
589.62 tok/s (pp4096+tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-32-3b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-32-3b Q4_K_M pp512 991.15 tok/s
991.15 tok/s (pp512)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-32-3b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): llama-32-3b Q4_K_M tg128 123.07 tok/s
123.07 tok/s (tg128)Llama-3.2-3B Q4_K_M GGUF (1.87 GiB, 3.21 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-llama-32-3b-q4km-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): tinyllama-11b Q4_K_M pp4096 1474.29 tok/s
1474.29 tok/s (pp4096)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-tinyllama-11b-q4km-pp4096
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): tinyllama-11b Q4_K_M pp4096+tg128 1286.83 tok/s
1286.83 tok/s (pp4096+tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-tinyllama-11b-q4km-pp4096-tg128
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): tinyllama-11b Q4_K_M pp512 2196.07 tok/s
2196.07 tok/s (pp512)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-tinyllama-11b-q4km-pp512
verifiedmedium · 0.75
Geerling llama.cpp Vulkan on rtxa4000 (pi host): tinyllama-11b Q4_K_M tg128 286.42 tok/s
286.42 tok/s (tg128)TinyLlama-1.1B-Chat Q4_K_M GGUF (528.14 MiB, 1.10 B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/geerling-vulkan-rtxa4000-pi-tinyllama-11b-q4km-tg128
verifiedmedium · 0.75
GeneralCompute: TTFT at 128K input for 6 open models (batch=1, cold cache)
2100 ms (TTFT, 128K input, batch=1, cold cache, 512 output tokens)Qwen2.5 7B / Llama 4 Scout (~17B active MoE) / Gemma 3 27B / DeepSeek V3 (~37B active MoE) / Qwen2.5 72B / Mistral Large 123B on ? via vllmbenchmarks/generalcompute-128k-ttft-6-models
verifiedmedium · 0.70
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 pp2048 d0 (b7941)
5948.74 tok/s (pp2048, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-pp2048-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 pp2048 d16384 (b7941)
5340.86 tok/s (pp2048, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-pp2048-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 pp2048 d32768 (b7941)
5023.3 tok/s (pp2048, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-pp2048-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 pp2048 d4096 (b7941)
5652.69 tok/s (pp2048, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-pp2048-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 pp2048 d8192 (b7941)
5509.57 tok/s (pp2048, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-pp2048-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 tg32 d0 (b7941)
81.05 tok/s (tg32, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-tg32-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 tg32 d16384 (b7941)
70.89 tok/s (tg32, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-tg32-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 tg32 d32768 (b7941)
62.28 tok/s (tg32, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-tg32-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 tg32 d4096 (b7941)
76.37 tok/s (tg32, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-tg32-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Gemma 3 4B QAT Q4_0 tg32 d8192 (b7941)
71.61 tok/s (tg32, batch 1, per-request)gemma-3-4b-it-qat Q4_0 (ggml-org/gemma-3-4b-it-qat-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gemma3-4b-qat-tg32-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 pp2048 d0 (b7941)
2364.18 tok/s (pp2048, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-pp2048-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 pp2048 d16384 (b7941)
914.05 tok/s (pp2048, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-pp2048-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 pp2048 d32768 (b7941)
567.2 tok/s (pp2048, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-pp2048-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 pp2048 d4096 (b7941)
1684.13 tok/s (pp2048, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-pp2048-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 pp2048 d8192 (b7941)
1314.68 tok/s (pp2048, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-pp2048-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 tg32 d0 (b7941)
48.68 tok/s (tg32, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-tg32-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 tg32 d16384 (b7941)
38.72 tok/s (tg32, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-tg32-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 tg32 d32768 (b7941)
32.65 tok/s (tg32, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-tg32-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 tg32 d4096 (b7941)
44.62 tok/s (tg32, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-tg32-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: GLM-4.7-Flash Q8_0 tg32 d8192 (b7941)
42.59 tok/s (tg32, batch 1, per-request)GLM-4.7-Flash Q8_0 (ggml-org/GLM-4.7-Flash-GGUF; llama-bench labels the arch deepseek2 30B.A3B) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-glm47-flash-tg32-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 pp2048 d0 (b7941)
2443.91 tok/s (pp2048, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-pp2048-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 pp2048 d16384 (b7941)
1956.31 tok/s (pp2048, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-pp2048-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 pp2048 d32768 (b7941)
1567.08 tok/s (pp2048, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-pp2048-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 pp2048 d4096 (b7941)
2309.84 tok/s (pp2048, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-pp2048-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 pp2048 d8192 (b7941)
2216.68 tok/s (pp2048, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-pp2048-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 tg32 d0 (b7941)
58.72 tok/s (tg32, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-tg32-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 tg32 d16384 (b7941)
49.45 tok/s (tg32, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-tg32-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 tg32 d32768 (b7941)
42.76 tok/s (tg32, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-tg32-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 tg32 d4096 (b7941)
55.67 tok/s (tg32, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-tg32-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-120b MXFP4 tg32 d8192 (b7941)
52.87 tok/s (tg32, batch 1, per-request)gpt-oss-120b MXFP4 MoE (ggml-org/gpt-oss-120b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss120b-tg32-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 pp2048 d0 (b7941)
4505.82 tok/s (pp2048, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-pp2048-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 pp2048 d16384 (b7941)
3449.98 tok/s (pp2048, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-pp2048-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 pp2048 d32768 (b7941)
2689.42 tok/s (pp2048, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-pp2048-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 pp2048 d4096 (b7941)
4158.34 tok/s (pp2048, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-pp2048-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 pp2048 d8192 (b7941)
3993.81 tok/s (pp2048, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-pp2048-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 tg32 d0 (b7941)
83.43 tok/s (tg32, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-tg32-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 tg32 d16384 (b7941)
70.36 tok/s (tg32, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-tg32-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 tg32 d32768 (b7941)
61.65 tok/s (tg32, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-tg32-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 tg32 d4096 (b7941)
79.22 tok/s (tg32, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-tg32-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: gpt-oss-20b MXFP4 tg32 d8192 (b7941)
75.22 tok/s (tg32, batch 1, per-request)gpt-oss-20b MXFP4 MoE (ggml-org/gpt-oss-20b-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-gptoss20b-tg32-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 pp2048 d0 (b7941)
2250.28 tok/s (pp2048, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-pp2048-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 pp2048 d16384 (b7941)
1779.11 tok/s (pp2048, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-pp2048-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 pp2048 d32768 (b7941)
1471.23 tok/s (pp2048, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-pp2048-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 pp2048 d4096 (b7941)
2100.19 tok/s (pp2048, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-pp2048-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 pp2048 d8192 (b7941)
2007.56 tok/s (pp2048, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-pp2048-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 tg32 d0 (b7941)
29.43 tok/s (tg32, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-tg32-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 tg32 d16384 (b7941)
25.72 tok/s (tg32, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-tg32-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 tg32 d32768 (b7941)
22.51 tok/s (tg32, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-tg32-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 tg32 d4096 (b7941)
28.61 tok/s (tg32, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-tg32-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen2.5-Coder-7B Q8_0 tg32 d8192 (b7941)
27.38 tok/s (tg32, batch 1, per-request)Qwen2.5-Coder-7B Q8_0 (ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen25-coder-7b-tg32-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 pp2048 d0 (b7941)
2986.97 tok/s (pp2048, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-pp2048-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 pp2048 d16384 (b7941)
1908.86 tok/s (pp2048, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-pp2048-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 pp2048 d32768 (b7941)
1348.17 tok/s (pp2048, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-pp2048-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 pp2048 d4096 (b7941)
2633.45 tok/s (pp2048, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-pp2048-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 pp2048 d8192 (b7941)
2354.14 tok/s (pp2048, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-pp2048-d8192-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 tg32 d0 (b7941)
61.06 tok/s (tg32, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-tg32-d0-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 tg32 d16384 (b7941)
40.23 tok/s (tg32, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-tg32-d16384-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 tg32 d32768 (b7941)
30.21 tok/s (tg32, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-tg32-d32768-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 tg32 d4096 (b7941)
54.77 tok/s (tg32, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-tg32-d4096-b7941
verifiedhigh · 0.85
ggml CUDA on DGX Spark GB10: Qwen3-Coder-30B-A3B Q8_0 tg32 d8192 (b7941)
48.02 tok/s (tg32, batch 1, per-request)Qwen3-Coder-30B-A3B-Instruct Q8_0 (ggml-org/Qwen3-Coder-30B-A3B-Instruct-Q8_0-GGUF) on nvidia-gb10 via llama-cppbenchmarks/ggml-cuda-gb10-spark-qwen3-coder-30b-a3b-tg32-d8192-b7941
verifiedhigh · 0.85
GGUF Q3_K_S GSM8K accuracy delta vs f16, Llama-3.1-8B-Instruct
-9.32 accuracy_delta_pp (GSM8K percentage points, quantized minus f16)Llama-3.1-8B-Instruct (Q3_K_S GGUF) on ? via lm-eval-harnessbenchmarks/gguf-q3-k-s-gsm8k-accuracy-delta-llama3-1-8b
verifiedhigh · 0.90
GGUF Q4_K_M WikiText-2 perplexity delta vs f16, Llama-3.1-8B-Instruct
0.24 perplexity_delta (WikiText-2 ppl points, quantized minus f16)Llama-3.1-8B-Instruct (F16 GGUF baseline) on ? via llama-cppbenchmarks/gguf-q4-k-m-wikitext2-ppl-delta-llama3-1-8b
verifiedhigh · 0.90
Google global fleet average annual PUE 1.09 in 2024, first sub-1.10 reading in six years, against a stated industry average of 1.56
1.09 dimensionless PUE ratio (total facility energy / IT equipment energy). Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-google-fleet-average-2024-1-09
verifiedhigh · 0.90
Google TPU v5e: 197 BF16 TFLOPS per chip
197 TFLOPS BF16N/A (peak compute specification) on google-tpu-v5e via alt-tpu-perf-guidebenchmarks/alt-tpu-v5e-bf16-tps
verifiedhigh · 0.90
Google TPU v6e: 918 BF16 TFLOPS per chip
918 TFLOPS BF16N/A (peak compute specification) on google-tpu-v6e via alt-tpu-perf-guidebenchmarks/alt-tpu-v6e-bf16-tps
verifiedhigh · 0.90
Groq LPU: no published inference throughput figure (honest negative)
0 tokens per second (not published)N/A (no throughput figure published) on groq-lpu via alt-groq-lpu-archbenchmarks/alt-groq-no-throughput-figure
verifiedhigh · 0.95
GuideLLM (Red Hat): A100-PCIE-40GB, Ollama 0.9.2 (NUM_PARALLEL=32), Llama-3.1-8B fp16, peak aggregate output tok/s
41.0 output tok/s (aggregate at saturation, concurrency up to 256)meta-llama/Llama-3.1-8B-Instruct on nvidia-a100-40gb-pcie via ollamabenchmarks/guidellm-a100-pcie-40gb-llama31-8b-fp16-ollama-peak-tps
verifiedmedium · 0.80
GuideLLM (Red Hat): A100-PCIE-40GB, vLLM 0.9.1, Llama-3.1-8B fp16, peak aggregate output tok/s
793.0 output tok/s (aggregate at saturation, concurrency up to 256)meta-llama/Llama-3.1-8B-Instruct on nvidia-a100-40gb-pcie via vllmbenchmarks/guidellm-a100-pcie-40gb-llama31-8b-fp16-vllm-peak-tps
verifiedmedium · 0.80
H100 reference column for MI300X vs H100, vLLM/TRT-LLM, Llama 3.1 405B FP8 TP8: output token throughput at 128-in/4096-out
1957 output tok/s (aggregate over all in-flight requests)Llama 3.1 405B on nvidia-h100-sxm via vllmbenchmarks/mi300x-vs-h100-vllm-llama31-405b-fp8-tp8-h100
verifiedhigh · 0.85
H100 reference column for MI300X vs H100, vLLM/TRT-LLM, Llama 3.1 70B FP8 TP8 batch 1: end-to-end time-to-last-token
20923 ms (time to last token, end-to-end, batch 1, per request)Llama 3.1 70B on nvidia-h100-sxm via vllmbenchmarks/mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-ttlt-batch1-h100
verifiedhigh · 0.85
H100 reference column for MI300X vs H100, vLLM/TRT-LLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out
15810 output tok/s (aggregate over all in-flight requests)Llama 3.1 70B on nvidia-h100-sxm via vllmbenchmarks/mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-h100
verifiedhigh · 0.85
Hailo-10H (Pi AI HAT+ 2) DeepSeek-R1-Distill-1.5B compiled-model TPS
7.96 tokens/sec compiled-model TPS (Hailo-Ollama on HailoRT)DeepSeek-R1-Distill-1.5B on hailo-10h via llama-cppbenchmarks/comm-hailo10h-pi-ai-hat2-deepseek-r1-1p5b-tps
verifiedmedium · 0.75
Hailo-10H (Pi AI HAT+ 2) Llama-3.2-1B TPS (vendor)
9.89 tokens/sec compiled-model TPSLlama-3.2-1B-Instruct on hailo-10h via llama-cppbenchmarks/comm-hailo10h-llama32-1b-tps
verifiedmedium · 0.70
Hailo-10H (Pi AI HAT+ 2) Qwen2-1.5B compiled-model TPS
8.06 tokens/sec compiled-model TPS (Hailo-Ollama on HailoRT)Qwen2-1.5B on hailo-10h via llama-cppbenchmarks/comm-hailo10h-pi-ai-hat2-qwen2-1p5b-tps
verifiedmedium · 0.75
Hailo-10H (Pi AI HAT+ 2) Qwen2-VL-2B compiled-model TPS
7.04 tokens/sec compiled-model TPS (Hailo-Ollama on HailoRT)Qwen2-VL-2B on hailo-10h via llama-cppbenchmarks/comm-hailo10h-pi-ai-hat2-qwen2-vl-2b-tps
verifiedmedium · 0.75
Hailo-10H (Pi AI HAT+ 2) Qwen2.5-Coder-1.5B compiled-model TPS
8.13 tokens/sec compiled-model TPS (Hailo-Ollama on HailoRT)Qwen2.5-Coder-1.5B on hailo-10h via llama-cppbenchmarks/comm-hailo10h-pi-ai-hat2-qwen25-coder-1p5b-tps
verifiedmedium · 0.75
Hailo-10H (Pi AI HAT+ 2) Qwen3-1.7B compiled-model TPS
4.78 tokens/sec compiled-model TPS (Hailo-Ollama on HailoRT)Qwen3-1.7B on hailo-10h via llama-cppbenchmarks/comm-hailo10h-pi-ai-hat2-qwen3-1p7b-tps
verifiedmedium · 0.75
Hailo-10H (Pi AI HAT+ 2), HailoRT/Ollama: Qwen3-1.7B compiled-model TPS (vendor table)
4.78 tokens/sec (Hailo compiled-model TPS table, single-stream)Qwen3-1.7B (Hailo-compiled) on hailo-10h via llama-cppbenchmarks/comm-hailo10h-qwen3-1p7b-tps
verifiedmedium · 0.80
Hailo-10H Pi AI HAT+ 2 Llama-3.2-1B compiled-model TPS
9.9 tok/s (Hailo compiled model, vendor TPS table)Llama-3.2-1B on hailo-10h via hailo-rtbenchmarks/comm-hailo-10h-llama32-1b-compiled-tps
verifiedmedium · 0.70
Hailo-8: ArcFace R50 INT8, 720 fps (Model Zoo)
720.0 inferences/s (fps, INT8, batch 1)ArcFace R50 112x112 INT8 on hailo-8 via hailo-rtbenchmarks/hailo8-modelzoo-arcface-fps
verifiedlow · 0.55
Hailo-8: EfficientNet-B0 INT8, 1850 fps (Model Zoo)
1850.0 inferences/s (fps, INT8, batch 1)EfficientNet-B0 224x224 INT8 on hailo-8 via hailo-rtbenchmarks/hailo8-modelzoo-efficientnetb0-fps
verifiedlow · 0.55
Hailo-8: MobileNetV3 INT8, 2800 fps (Model Zoo)
2800.0 inferences/s (fps, INT8, batch 1)MobileNetV3 224x224 INT8 on hailo-8 via hailo-rtbenchmarks/hailo8-modelzoo-mobilenetv3-fps
verifiedlow · 0.55
Hailo-8: ResNet-50 INT8, 1390 fps (Model Zoo)
1390.0 inferences/s (fps, INT8, batch 1)ResNet-50 224x224 INT8 on hailo-8 via hailo-rtbenchmarks/hailo8-modelzoo-resnet50-fps
verifiedlow · 0.55
Hailo-8: YOLOv8n INT8, 235 fps (Model Zoo)
235.0 inferences/s (fps, INT8, batch 1)YOLOv8n 640x640 INT8 on hailo-8 via hailo-rtbenchmarks/hailo8-modelzoo-yolov8n-fps
verifiedlow · 0.55
Hardware Corner Dec 2025 ranking: RTX 3080 Ti, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1773.87 t/s @16K ctx
1773.87 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-rankpp-rtx3080ti-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3080 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 68.07 t/s @32K ctx
68.07 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-rank-rtx3080ti-qwen3-8b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3080 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1761.17 t/s @32K ctx
1761.17 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-rankpp-rtx3080ti-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3080 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2658.54 t/s @16K ctx
2658.54 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-rankpp-rtx3080ti-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090 Ti, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 137.55 t/s @16K ctx
137.55 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc-rank-rtx3090ti-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090 Ti, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 121.88 t/s @16K ctx
121.88 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc-rank-rtx3090ti-qwen3-30b-a3b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090 Ti, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 32.61 t/s @16K ctx
32.61 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc-rank-rtx3090ti-qwen3-32b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2834.15 t/s @16K ctx
2834.15 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hc-rankpp-rtx3090ti-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 112.55 t/s @32K ctx
112.55 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 128.51 t/s @16K ctx
128.51 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 62.18 t/s @131K ctx
62.18 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-gptoss-20b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 89.64 t/s @65K ctx
89.64 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 1720.56 t/s @65K ctx
1720.56 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 2547.19 t/s @32K ctx
2547.19 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 3243.60 t/s @16K ctx
3243.6 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 923.79 t/s @131K ctx
923.79 tok/s per-request prefill (batch 1, ~131072-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-gpt-oss-20b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 38.64 t/s @32K ctx
38.64 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 52.14 t/s @16K ctx
52.14 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1175.71 t/s @32K ctx
1175.71 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1678.68 t/s @16K ctx
1678.68 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 734.10 t/s @65K ctx
734.1 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-14b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 113.84 t/s @16K ctx
113.84 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-30b-a3b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 87.21 t/s @32K ctx
87.21 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-30b-a3b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 1336.79 t/s @32K ctx
1336.79 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 1958.99 t/s @16K ctx
1958.99 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 30.28 t/s @16K ctx
30.28 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-32b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 767.82 t/s @16K ctx
767.82 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-32b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 28.09 t/s @131K ctx
28.09 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-8b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 46.59 t/s @65K ctx
46.59 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rank-rtx3090-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1014.28 t/s @65K ctx
1014.28 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1714.63 t/s @32K ctx
1714.63 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2572.49 t/s @16K ctx
2572.49 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 3090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 569.97 t/s @131K ctx
569.97 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc-rankpp-rtx3090-qwen3-8b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4070 Ti, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1951.36 t/s @16K ctx
1951.36 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc-rankpp-rtx4070ti-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1099.67 t/s @32K ctx
1099.67 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc-rankpp-rtx4070ti-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2274.84 t/s @16K ctx
2274.84 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc-rankpp-rtx4070ti-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 120.16 t/s @16K ctx
120.16 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 59.95 t/s @131K ctx
59.95 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-gptoss-20b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 81.34 t/s @65K ctx
81.34 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 1558.98 t/s @65K ctx
1558.98 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 2831.05 t/s @32K ctx
2831.05 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 4329.15 t/s @16K ctx
4329.15 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 898.45 t/s @131K ctx
898.45 tok/s per-request prefill (batch 1, ~131072-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-gpt-oss-20b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 40.56 t/s @32K ctx
40.56 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 51.22 t/s @16K ctx
51.22 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1395.65 t/s @32K ctx
1395.65 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 39.01 t/s @65K ctx
39.01 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 58.98 t/s @32K ctx
58.98 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-qwen3-8b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 77.86 t/s @16K ctx
77.86 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rank-rtx4080-qwen3-8b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1968.35 t/s @32K ctx
1968.35 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 3809.77 t/s @16K ctx
3809.77 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hc-rankpp-rtx4080-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 103.01 t/s @65K ctx
103.01 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 140.34 t/s @32K ctx
140.34 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 163.92 t/s @16K ctx
163.92 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 70.27 t/s @131K ctx
70.27 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-gptoss-20b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 2092.21 t/s @65K ctx
2092.21 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 4641.50 t/s @32K ctx
4641.5 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 6302.76 t/s @16K ctx
6302.76 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 38.73 t/s @65K ctx
38.73 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-14b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 55.49 t/s @32K ctx
55.49 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 69.14 t/s @16K ctx
69.14 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1545.47 t/s @65K ctx
1545.47 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-14b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 2511.27 t/s @32K ctx
2511.27 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 3927.70 t/s @16K ctx
3927.7 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 105.57 t/s @32K ctx
105.57 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-30b-a3b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 139.71 t/s @16K ctx
139.71 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-30b-a3b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 1502.45 t/s @65K ctx
1502.45 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-30b-a3b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 2726.03 t/s @32K ctx
2726.03 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 4548.53 t/s @16K ctx
4548.53 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 1684.85 t/s @16K ctx
1684.85 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-32b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 104.31 t/s @16K ctx
104.31 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-8b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 32.27 t/s @131K ctx
32.27 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-8b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 78.42 t/s @32K ctx
78.42 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rank-rtx4090-qwen3-8b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1450.99 t/s @131K ctx
1450.99 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-8b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1874.47 t/s @65K ctx
1874.47 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 4223.63 t/s @32K ctx
4223.63 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 6720.91 t/s @16K ctx
6720.91 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc-rankpp-rtx4090-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 73.21 t/s @32K ctx
73.21 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rank-rtx5060ti-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 1102.29 t/s @65K ctx
1102.29 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 1737.71 t/s @32K ctx
1737.71 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 685.26 t/s @131K ctx
685.26 tok/s per-request prefill (batch 1, ~131072-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-gpt-oss-20b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 25.85 t/s @32K ctx
25.85 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rank-rtx5060ti-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 620.98 t/s @32K ctx
620.98 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 25.82 t/s @65K ctx
25.82 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rank-rtx5060ti-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 38.93 t/s @32K ctx
38.93 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rank-rtx5060ti-qwen3-8b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1447.92 t/s @16K ctx
1447.92 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 529.78 t/s @65K ctx
529.78 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5060 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 915.13 t/s @32K ctx
915.13 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc-rankpp-rtx5060ti-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 115.88 t/s @32K ctx
115.88 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rank-rtx5070ti-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 133.05 t/s @16K ctx
133.05 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rank-rtx5070ti-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 91.25 t/s @65K ctx
91.25 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rank-rtx5070ti-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 2644.93 t/s @65K ctx
2644.93 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 3903.47 t/s @32K ctx
3903.47 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 4940.71 t/s @16K ctx
4940.71 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 57.98 t/s @16K ctx
57.98 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rank-rtx5070ti-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1658.23 t/s @32K ctx
1658.23 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 40.41 t/s @65K ctx
40.41 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rank-rtx5070ti-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 87.54 t/s @16K ctx
87.54 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rank-rtx5070ti-qwen3-8b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1078.71 t/s @65K ctx
1078.71 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2268.97 t/s @32K ctx
2268.97 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 3653.77 t/s @16K ctx
3653.77 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hc-rankpp-rtx5070ti-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5070, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 40.59 t/s @16K ctx
40.59 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-rank-rtx5070-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 128.12 t/s @32K ctx
128.12 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rank-rtx5080-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 140.48 t/s @16K ctx
140.48 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rank-rtx5080-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 69.62 t/s @131K ctx
69.62 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rank-rtx5080-gptoss-20b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 98.53 t/s @65K ctx
98.53 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rank-rtx5080-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 1834.71 t/s @65K ctx
1834.71 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 3653.30 t/s @32K ctx
3653.3 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 4932.33 t/s @16K ctx
4932.33 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 937.72 t/s @131K ctx
937.72 tok/s per-request prefill (batch 1, ~131072-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-gpt-oss-20b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 51.87 t/s @32K ctx
51.87 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rank-rtx5080-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 64.04 t/s @16K ctx
64.04 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rank-rtx5080-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1326.05 t/s @32K ctx
1326.05 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 2542.01 t/s @16K ctx
2542.01 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1124.63 t/s @65K ctx
1124.63 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1943.27 t/s @32K ctx
1943.27 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5080, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 4024.38 t/s @16K ctx
4024.38 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5080 via llama-cppbenchmarks/hc-rankpp-rtx5080-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 112.01 t/s @131K ctx
112.01 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-gptoss-20b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 168.95 t/s @65K ctx
168.95 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 215.08 t/s @32K ctx
215.08 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 249.19 t/s @16K ctx
249.19 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 3019.74 t/s @65K ctx
3019.74 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-gpt-oss-20b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 5183.08 t/s @32K ctx
5183.08 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 57.58 t/s @65K ctx
57.58 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-14b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 82.35 t/s @32K ctx
82.35 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1707.24 t/s @65K ctx
1707.24 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-14b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 2908.42 t/s @32K ctx
2908.42 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 4473.08 t/s @16K ctx
4473.08 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 908.39 t/s @131K ctx
908.39 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-14b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 110.65 t/s @32K ctx
110.65 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-30b-a3b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 141.63 t/s @16K ctx
141.63 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-30b-a3b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 52.84 t/s @131K ctx
52.84 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-30b-a3b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 76.56 t/s @65K ctx
76.56 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-30b-a3b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 1512.46 t/s @65K ctx
1512.46 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-30b-a3b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 2877.53 t/s @32K ctx
2877.53 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 4669.17 t/s @16K ctx
4669.17 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 716.76 t/s @131K ctx
716.76 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-30b-a3b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 43.82 t/s @32K ctx
43.82 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-32b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 50.92 t/s @16K ctx
50.92 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-32b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 1451.08 t/s @32K ctx
1451.08 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-32b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 2077.16 t/s @16K ctx
2077.16 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-32b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 145.34 t/s @16K ctx
145.34 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-8b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 49.44 t/s @131K ctx
49.44 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-8b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 80.14 t/s @65K ctx
80.14 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rank-rtx5090-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2211.55 t/s @65K ctx
2211.55 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 6956.11 t/s @16K ctx
6956.11 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 5090, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 948.40 t/s @131K ctx
948.4 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rankpp-rtx5090-qwen3-8b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 116.25 t/s @32K ctx
116.25 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 3818.65 t/s @32K ctx
3818.65 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 5350.48 t/s @16K ctx
5350.48 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Llama 3.3 70B Q4_K_XL GGUF, prefill 526.03 t/s @16K ctx
526.03 tok/s per-request prefill (batch 1, ~16384-token context)Llama 3.3 70B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-llama33-70b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 15.18 t/s @131K ctx
15.18 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-14b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 27.01 t/s @65K ctx
27.01 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-14b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 42.27 t/s @32K ctx
42.27 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 2367.17 t/s @16K ctx
2367.17 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 358.58 t/s @131K ctx
358.58 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-14b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 727.43 t/s @65K ctx
727.43 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-14b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 120.12 t/s @16K ctx
120.12 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-30b-a3b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 20.71 t/s @131K ctx
20.71 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-30b-a3b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 40.83 t/s @65K ctx
40.83 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-30b-a3b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 74.31 t/s @32K ctx
74.31 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-30b-a3b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 2010.62 t/s @32K ctx
2010.62 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 448.63 t/s @131K ctx
448.63 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-30b-a3b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 18.17 t/s @32K ctx
18.17 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-32b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 25.08 t/s @16K ctx
25.08 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-32b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 348.64 t/s @65K ctx
348.64 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-32b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 595.46 t/s @32K ctx
595.46 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-32b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 977.52 t/s @16K ctx
977.52 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-32b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 20.76 t/s @131K ctx
20.76 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-8b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 39.89 t/s @65K ctx
39.89 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 68.87 t/s @32K ctx
68.87 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-8b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 98.68 t/s @16K ctx
98.68 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rank-rtx6000ada-qwen3-8b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1152.63 t/s @65K ctx
1152.63 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 2218.45 t/s @32K ctx
2218.45 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 4096.35 t/s @16K ctx
4096.35 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 541.27 t/s @131K ctx
541.27 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc-rankpp-rtx6000ada-qwen3-8b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, GLM-4.5-Air 106B Q4_K_XL GGUF, decode 17.72 t/s @131K ctx
17.72 tok/s per-request decode (batch 1, ~131072-token context)GLM-4.5-Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-glm45-air-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, GLM-4.5-Air 106B Q4_K_XL GGUF, decode 30.54 t/s @65K ctx
30.54 tok/s per-request decode (batch 1, ~65536-token context)GLM-4.5-Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-glm45-air-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, GLM-4.5-Air 106B Q4_K_XL GGUF, decode 64.18 t/s @32K ctx
64.18 tok/s per-request decode (batch 1, ~32768-token context)GLM-4.5-Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-glm45-air-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, GLM-4.5-Air 106B Q4_K_XL GGUF, decode 82.87 t/s @16K ctx
82.87 tok/s per-request decode (batch 1, ~16384-token context)GLM-4.5-Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-glm45-air-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 120B MXFP4 GGUF, decode 133.07 t/s @65K ctx
133.07 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 120B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-120b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 120B MXFP4 GGUF, decode 161.49 t/s @32K ctx
161.49 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 120B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-120b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 120B MXFP4 GGUF, decode 182.23 t/s @16K ctx
182.23 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 120B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-120b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 120B MXFP4 GGUF, decode 99.79 t/s @131K ctx
99.79 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 120B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-120b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 118.44 t/s @131K ctx
118.44 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-20b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 171.33 t/s @65K ctx
171.33 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-20b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 207.45 t/s @32K ctx
207.45 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-20b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, gpt-oss 20B MXFP4 GGUF, decode 237.92 t/s @16K ctx
237.92 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B MXFP4 GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-gptoss-20b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Llama 70B Q4_K_XL GGUF, decode 16.62 t/s @131K ctx
16.62 tok/s per-request decode (batch 1, ~131072-token context)Llama 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-llama-70b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Llama 70B Q4_K_XL GGUF, decode 21.67 t/s @65K ctx
21.67 tok/s per-request decode (batch 1, ~65536-token context)Llama 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-llama-70b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Llama 70B Q4_K_XL GGUF, decode 25.67 t/s @32K ctx
25.67 tok/s per-request decode (batch 1, ~32768-token context)Llama 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-llama-70b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Llama 70B Q4_K_XL GGUF, decode 28.24 t/s @16K ctx
28.24 tok/s per-request decode (batch 1, ~16384-token context)Llama 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-llama-70b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Mistral Large 123B Q4_K_XL GGUF, decode 10.52 t/s @65K ctx
10.52 tok/s per-request decode (batch 1, ~65536-token context)Mistral Large 123B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-mistral-large-123b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Mistral Large 123B Q4_K_XL GGUF, decode 15.9 t/s @32K ctx
15.9 tok/s per-request decode (batch 1, ~32768-token context)Mistral Large 123B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-mistral-large-123b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 60.36 t/s @65K ctx
60.36 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-14b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 80.32 t/s @32K ctx
80.32 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-14b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 96.86 t/s @16K ctx
96.86 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-14b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 103.33 t/s @32K ctx
103.33 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-30b-a3b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 139.76 t/s @16K ctx
139.76 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-30b-a3b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 56.14 t/s @131K ctx
56.14 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-30b-a3b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 80.15 t/s @65K ctx
80.15 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-30b-a3b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 23.05 t/s @131K ctx
23.05 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-32b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 39.91 t/s @32K ctx
39.91 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-32b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 45.72 t/s @16K ctx
45.72 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-32b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 111.1 t/s @32K ctx
111.1 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-8b-q4kxl-tg-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 140.62 t/s @16K ctx
140.62 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-8b-q4kxl-tg-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 48.29 t/s @131K ctx
48.29 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-8b-q4kxl-tg-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000 WS, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 77.94 t/s @65K ctx
77.94 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rank-rtxpro6000ws-qwen3-8b-q4kxl-tg-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, GLM Air 106B Q4_K_XL GGUF, prefill 1450.34 t/s @32K ctx
1450.34 tok/s per-request prefill (batch 1, ~32768-token context)GLM Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-glm45-air-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, GLM Air 106B Q4_K_XL GGUF, prefill 2107.16 t/s @16K ctx
2107.16 tok/s per-request prefill (batch 1, ~16384-token context)GLM Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-glm45-air-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, GLM Air 106B Q4_K_XL GGUF, prefill 320.19 t/s @131K ctx
320.19 tok/s per-request prefill (batch 1, ~131072-token context)GLM Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-glm45-air-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, GLM Air 106B Q4_K_XL GGUF, prefill 698.67 t/s @65K ctx
698.67 tok/s per-request prefill (batch 1, ~65536-token context)GLM Air 106B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-glm45-air-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, gpt-oss 120B Q4_K_XL GGUF (MXFP4), prefill 1289.76 t/s @131K ctx
1289.76 tok/s per-request prefill (batch 1, ~131072-token context)gpt-oss 120B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-gpt-oss-120b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, gpt-oss 120B Q4_K_XL GGUF (MXFP4), prefill 2360.34 t/s @65K ctx
2360.34 tok/s per-request prefill (batch 1, ~65536-token context)gpt-oss 120B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-gpt-oss-120b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, gpt-oss 120B Q4_K_XL GGUF (MXFP4), prefill 3368.32 t/s @32K ctx
3368.32 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 120B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-gpt-oss-120b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, gpt-oss 120B Q4_K_XL GGUF (MXFP4), prefill 4060.65 t/s @16K ctx
4060.65 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 120B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-gpt-oss-120b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 5039.46 t/s @32K ctx
5039.46 tok/s per-request prefill (batch 1, ~32768-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-gpt-oss-20b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF (MXFP4), prefill 7478.71 t/s @16K ctx
7478.71 tok/s per-request prefill (batch 1, ~16384-token context)gpt-oss 20B Q4_K_XL GGUF (MXFP4) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-gpt-oss-20b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Llama 3.3 70B Q4_K_XL GGUF, prefill 1008.35 t/s @32K ctx
1008.35 tok/s per-request prefill (batch 1, ~32768-token context)Llama 3.3 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-llama33-70b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Llama 3.3 70B Q4_K_XL GGUF, prefill 1355.35 t/s @16K ctx
1355.35 tok/s per-request prefill (batch 1, ~16384-token context)Llama 3.3 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-llama33-70b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Llama 3.3 70B Q4_K_XL GGUF, prefill 528.23 t/s @65K ctx
528.23 tok/s per-request prefill (batch 1, ~65536-token context)Llama 3.3 70B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-llama33-70b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Mistral Large 123B Q4_K_XL GGUF, prefill 343.07 t/s @65K ctx
343.07 tok/s per-request prefill (batch 1, ~65536-token context)Mistral Large 123B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-mistral-large-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Mistral Large 123B Q4_K_XL GGUF, prefill 600.76 t/s @32K ctx
600.76 tok/s per-request prefill (batch 1, ~32768-token context)Mistral Large 123B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-mistral-large-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Mistral Large 123B Q4_K_XL GGUF, prefill 796.64 t/s @16K ctx
796.64 tok/s per-request prefill (batch 1, ~16384-token context)Mistral Large 123B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-mistral-large-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 1378.08 t/s @65K ctx
1378.08 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-14b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 3536.82 t/s @32K ctx
3536.82 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-14b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 5027.24 t/s @16K ctx
5027.24 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-14b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, prefill 717.01 t/s @131K ctx
717.01 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 14B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-14b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 1270.31 t/s @131K ctx
1270.31 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-30b-a3b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 2864.83 t/s @65K ctx
2864.83 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-30b-a3b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 3863.04 t/s @32K ctx
3863.04 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, prefill 5084.45 t/s @16K ctx
5084.45 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 1687.05 t/s @32K ctx
1687.05 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-32b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 2373.96 t/s @16K ctx
2373.96 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-32b-pp-16k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 330.38 t/s @131K ctx
330.38 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-32b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, prefill 707.21 t/s @65K ctx
707.21 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 32B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-32b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1009.72 t/s @131K ctx
1009.72 tok/s per-request prefill (batch 1, ~131072-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-8b-pp-131k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 1921.61 t/s @65K ctx
1921.61 tok/s per-request prefill (batch 1, ~65536-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-8b-pp-65k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 5300.66 t/s @32K ctx
5300.66 tok/s per-request prefill (batch 1, ~32768-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-8b-pp-32k
verifiedmedium · 0.70
Hardware Corner Dec 2025 ranking: RTX Pro 6000, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, prefill 7587.74 t/s @16K ctx
7587.74 tok/s per-request prefill (batch 1, ~16384-token context)Qwen3 8B Q4_K_XL GGUF on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc-rankpp-rtxpro6000-qwen3-8b-pp-16k
verifiedmedium · 0.70
Hardware Corner Jun 2025: Quadro RTX 8000, ExLlamaV2 0.2.9, Llama 3.3 70B 5.0 bpw, decode 10.43 t/s at 8k
10.43 tok/s per-request decode (8192-token context, 8-bit cache)Llama 3.3 70B EXL2 5.0 bpw on nvidia-quadro-rtx-8000 via exllamav2benchmarks/exllamav2-quadro-rtx-8000-llama33-70b-5bpw-8k-tg
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, ExLlamaV2 0.2.9, Llama 3.3 70B 5.0 bpw, prefill 279 t/s at 8k
279.0 tok/s per-request prefill (8192-token context, 8-bit cache)Llama 3.3 70B EXL2 5.0 bpw on nvidia-quadro-rtx-8000 via exllamav2benchmarks/exllamav2-quadro-rtx-8000-llama33-70b-5bpw-8k-pp
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, ExLlamaV2 0.2.9, Qwen1.5 32B 5.0 bpw, decode 14.2 t/s at 8k
14.2 tok/s per-request decode (8192-token context, 8-bit cache)Qwen1.5 32B EXL2 5.0 bpw on nvidia-quadro-rtx-8000 via exllamav2benchmarks/exllamav2-quadro-rtx-8000-qwen15-32b-5bpw-8k-tg
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, ExLlamaV2 0.2.9, Qwen1.5 32B 5.0 bpw, prefill 458 t/s at 8k
458.0 tok/s per-request prefill (8192-token context, 8-bit cache)Qwen1.5 32B EXL2 5.0 bpw on nvidia-quadro-rtx-8000 via exllamav2benchmarks/exllamav2-quadro-rtx-8000-qwen15-32b-5bpw-8k-pp
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, llama.cpp b5508, Llama 3.3 70B Q5_K_M, decode 7.59 t/s at 8k
7.59 tok/s per-request decode (batch 1, 8192-token context)Llama 3.3 70B Q5_K_M GGUF on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-b5508-quadro-rtx-8000-llama33-70b-q5km-8k-tg
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, llama.cpp b5508, Llama 3.3 70B Q5_K_M, prefill 190 t/s at 8k
190.0 tok/s per-request prefill (batch 1, 8192-token context)Llama 3.3 70B Q5_K_M GGUF on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-b5508-quadro-rtx-8000-llama33-70b-q5km-8k-pp
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, llama.cpp b5508, Qwen1.5 32B Q5_K_M, decode 11 t/s at 8k
11.0 tok/s per-request decode (batch 1, 8192-token context)Qwen1.5 32B Q5_K_M GGUF on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-b5508-quadro-rtx-8000-qwen15-32b-q5km-8k-tg
verifiedmedium · 0.65
Hardware Corner Jun 2025: Quadro RTX 8000, llama.cpp b5508, Qwen1.5 32B Q5_K_M, prefill 344 t/s at 8k
344.0 tok/s per-request prefill (batch 1, 8192-token context)Qwen1.5 32B Q5_K_M GGUF on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-b5508-quadro-rtx-8000-qwen15-32b-q5km-8k-pp
verifiedmedium · 0.65
Hardware Corner lab: RTX 5070 Qwen3-8B Q4_K_XL decode @ 16k ctx
59.13 tok/s (decode, per-request, batch 1)Qwen3 8B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.80
Hardware Corner lab: RTX 5070 Qwen3-8B Q4_K_XL decode @ 32k ctx
43.6 tok/s (decode, per-request, batch 1)Qwen3 8B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.80
Hardware Corner lab: RTX 5080 Qwen3-8B Q4_K_XL decode @ 16k ctx
94.14 tok/s (decode, per-request, batch 1)Qwen3 8B on nvidia-rtx-5080 via llama-cppbenchmarks/hwcorner-rtx-5080-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.80
Hardware Corner lab: RTX 5080 Qwen3-8B Q4_K_XL decode @ 32k ctx
72.49 tok/s (decode, per-request, batch 1)Qwen3 8B on nvidia-rtx-5080 via llama-cppbenchmarks/hwcorner-rtx-5080-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.80
Hardware Corner lab: RTX 5080 Qwen3-8B Q4_K_XL decode @ 65k ctx
49.0 tok/s (decode, per-request, batch 1)Qwen3 8B on nvidia-rtx-5080 via llama-cppbenchmarks/hwcorner-rtx-5080-qwen3-8b-q4kxl-decode-65k
verifiedmedium · 0.80
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 14B Q4_K decode @ 16k ctx
22.7 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 14B Q4_K decode @ 4k ctx
31.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 14B Q4_K prefill @ 16k ctx
678.2 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 14B Q4_K prefill @ 4k ctx
972.6 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 8B Q4_K decode @ 16k ctx
42.0 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-8b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 8B Q4_K decode @ 32k ctx
31.9 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 8B Q4_K decode @ 4k ctx
55.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 8B Q4_K prefill @ 16k ctx
1119.2 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 8B Q4_K prefill @ 32k ctx
764.7 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3060 12GB Qwen3 8B Q4_K prefill @ 4k ctx
1696.8 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-rtx3060-12gb-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 14B Q4_K decode @ 16k ctx
52.3 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 14B Q4_K decode @ 4k ctx
69.9 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 14B Q4_K prefill @ 16k ctx
1773.9 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 14B Q4_K prefill @ 4k ctx
2600.1 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 8B Q4_K decode @ 16k ctx
87.9 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-8b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 8B Q4_K decode @ 32k ctx
68.1 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 8B Q4_K decode @ 4k ctx
115.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 8B Q4_K prefill @ 16k ctx
2658.5 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 8B Q4_K prefill @ 32k ctx
1761.2 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3080 Ti Qwen3 8B Q4_K prefill @ 4k ctx
4211.7 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-rtx3080ti-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) decode @ 128k ctx
41.4 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) decode @ 16k ctx
121.8 tok/s (decode, per-request, batch 1, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) decode @ 32k ctx
94.8 tok/s (decode, per-request, batch 1, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) decode @ 4k ctx
155.4 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) decode @ 64k ctx
67.7 tok/s (decode, per-request, batch 1, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) prefill @ 128k ctx
723.2 tok/s (prefill, per-request, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) prefill @ 16k ctx
4114.6 tok/s (prefill, per-request, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) prefill @ 32k ctx
2605.7 tok/s (prefill, per-request, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) prefill @ 4k ctx
6294.8 tok/s (prefill, per-request, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 gpt-oss 20B (MXFP4) prefill @ 64k ctx
1435.4 tok/s (prefill, per-request, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gptoss-20b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) decode @ 16k ctx
52.1 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) decode @ 32k ctx
40.7 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) decode @ 4k ctx
75.4 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) decode @ 64k ctx
27.3 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) prefill @ 16k ctx
1679.0 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) prefill @ 32k ctx
1090.8 tok/s (prefill, per-request, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) prefill @ 4k ctx
2961.3 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 14B (Q4_K) prefill @ 64k ctx
643.2 tok/s (prefill, per-request, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-14b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) decode @ 16k ctx
98.3 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) decode @ 32k ctx
58.8 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) decode @ 4k ctx
179.0 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) decode @ 64k ctx
32.8 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) prefill @ 16k ctx
2375.7 tok/s (prefill, per-request, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) prefill @ 32k ctx
1432.6 tok/s (prefill, per-request, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) prefill @ 4k ctx
4137.0 tok/s (prefill, per-request, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 30B A3B (Q4_K) prefill @ 64k ctx
808.4 tok/s (prefill, per-request, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 32B (Q4_K) decode @ 16k ctx
28.1 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-32b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 32B (Q4_K) decode @ 4k ctx
36.0 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-32b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 32B (Q4_K) prefill @ 16k ctx
812.1 tok/s (prefill, per-request, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-32b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 32B (Q4_K) prefill @ 4k ctx
1298.7 tok/s (prefill, per-request, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-32b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) decode @ 128k ctx
25.0 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) decode @ 16k ctx
91.7 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) decode @ 32k ctx
67.7 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) decode @ 4k ctx
126.5 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) decode @ 64k ctx
44.1 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) prefill @ 128k ctx
522.0 tok/s (prefill, per-request, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) prefill @ 16k ctx
3122.5 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) prefill @ 32k ctx
1932.5 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) prefill @ 4k ctx
5259.5 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 3090 Qwen3 8B (Q4_K) prefill @ 64k ctx
1019.4 tok/s (prefill, per-request, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-8b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 decode @ 128k ctx
31.1 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-tg-128k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 decode @ 16k ctx
57.8 tok/s (decode, per-request, batch 1, KV depth 16k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 decode @ 32k ctx
51.5 tok/s (decode, per-request, batch 1, KV depth 32k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 decode @ 4k ctx
63.2 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 decode @ 64k ctx
41.1 tok/s (decode, per-request, batch 1, KV depth 64k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-tg-64k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 prefill @ 128k ctx
780.2 tok/s (prefill, per-request, KV depth 128k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-pp-128k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 prefill @ 16k ctx
2552.9 tok/s (prefill, per-request, KV depth 16k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 prefill @ 32k ctx
1964.9 tok/s (prefill, per-request, KV depth 32k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 prefill @ 4k ctx
3274.2 tok/s (prefill, per-request, KV depth 4k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB gpt-oss 20B MXFP4 prefill @ 64k ctx
1332.2 tok/s (prefill, per-request, KV depth 64k)gpt-oss 20B MXFP4 on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-gptoss20b-mxfp4-pp-64k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 14B Q4_K decode @ 16k ctx
22.4 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 14B Q4_K decode @ 32k ctx
17.9 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 14B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-14b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 14B Q4_K decode @ 4k ctx
27.4 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 14B Q4_K prefill @ 16k ctx
917.6 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 14B Q4_K prefill @ 32k ctx
541.4 tok/s (prefill, per-request, KV depth 32k)Qwen3 14B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-14b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 14B Q4_K prefill @ 4k ctx
1645.7 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K decode @ 16k ctx
34.3 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K decode @ 32k ctx
25.5 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K decode @ 4k ctx
45.8 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K decode @ 64k ctx
13.0 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-tg-64k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K prefill @ 16k ctx
1480.8 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K prefill @ 32k ctx
760.3 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K prefill @ 4k ctx
2675.2 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4060 Ti 16GB Qwen3 8B Q4_K prefill @ 64k ctx
392.1 tok/s (prefill, per-request, KV depth 64k)Qwen3 8B Q4_K on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc26-rtx4060ti16-qwen3-8b-q4k-pp-64k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 14B Q4_K decode @ 16k ctx
32.7 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 14B Q4_K decode @ 4k ctx
42.5 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 14B Q4_K prefill @ 16k ctx
1355.8 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 14B Q4_K prefill @ 4k ctx
2099.9 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 8B Q4_K decode @ 16k ctx
52.1 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-8b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 8B Q4_K decode @ 32k ctx
38.1 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 8B Q4_K decode @ 4k ctx
71.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 8B Q4_K prefill @ 16k ctx
2064.3 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 8B Q4_K prefill @ 32k ctx
1116.7 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 Qwen3 8B Q4_K prefill @ 4k ctx
3564.1 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-rtx4070-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 14B Q4_K decode @ 16k ctx
37.2 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 14B Q4_K decode @ 4k ctx
45.5 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 14B Q4_K prefill @ 16k ctx
1578.1 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 14B Q4_K prefill @ 4k ctx
2522.2 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 8B Q4_K decode @ 16k ctx
56.2 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-8b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 8B Q4_K decode @ 32k ctx
42.2 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 8B Q4_K decode @ 4k ctx
75.4 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 8B Q4_K prefill @ 16k ctx
2525.8 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 8B Q4_K prefill @ 32k ctx
1595.7 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4070 SUPER Qwen3 8B Q4_K prefill @ 4k ctx
4321.7 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B Q4_K on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-rtx4070s-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) decode @ 128k ctx
70.3 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) decode @ 16k ctx
163.3 tok/s (decode, per-request, batch 1, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) decode @ 32k ctx
140.7 tok/s (decode, per-request, batch 1, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) decode @ 4k ctx
190.6 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) decode @ 64k ctx
104.0 tok/s (decode, per-request, batch 1, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) prefill @ 128k ctx
1114.7 tok/s (prefill, per-request, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) prefill @ 16k ctx
5856.2 tok/s (prefill, per-request, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) prefill @ 32k ctx
3705.6 tok/s (prefill, per-request, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) prefill @ 4k ctx
8369.3 tok/s (prefill, per-request, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 gpt-oss 20B (MXFP4) prefill @ 64k ctx
2092.2 tok/s (prefill, per-request, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-gptoss-20b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) decode @ 16k ctx
69.8 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) decode @ 32k ctx
55.4 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) decode @ 4k ctx
84.4 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) decode @ 64k ctx
38.8 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) prefill @ 16k ctx
3452.3 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) prefill @ 32k ctx
2324.2 tok/s (prefill, per-request, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) prefill @ 4k ctx
5265.4 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 14B (Q4_K) prefill @ 64k ctx
1398.3 tok/s (prefill, per-request, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-14b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) decode @ 16k ctx
139.2 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) decode @ 32k ctx
105.1 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) decode @ 4k ctx
207.3 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) decode @ 64k ctx
68.2 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) prefill @ 16k ctx
4027.3 tok/s (prefill, per-request, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) prefill @ 32k ctx
2627.7 tok/s (prefill, per-request, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) prefill @ 4k ctx
6159.4 tok/s (prefill, per-request, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 30B A3B (Q4_K) prefill @ 64k ctx
1502.5 tok/s (prefill, per-request, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-30b-a3b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 32B (Q4_K) decode @ 16k ctx
34.4 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-32b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 32B (Q4_K) decode @ 4k ctx
39.6 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-32b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 32B (Q4_K) prefill @ 16k ctx
1707.4 tok/s (prefill, per-request, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-32b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 32B (Q4_K) prefill @ 4k ctx
2448.0 tok/s (prefill, per-request, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-32b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) decode @ 128k ctx
33.8 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) decode @ 16k ctx
108.0 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) decode @ 32k ctx
82.3 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) decode @ 4k ctx
141.3 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) decode @ 64k ctx
56.1 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) prefill @ 128k ctx
1059.9 tok/s (prefill, per-request, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) prefill @ 16k ctx
5530.5 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) prefill @ 32k ctx
3560.1 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) prefill @ 4k ctx
9250.5 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 4090 Qwen3 8B (Q4_K) prefill @ 64k ctx
2028.6 tok/s (prefill, per-request, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-4090 via llama-cppbenchmarks/hc26-4090-qwen3-8b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 5070 Qwen3 14B Q4_K decode @ 16k ctx
40.6 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 14B Q4_K decode @ 4k ctx
54.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 14B Q4_K prefill @ 16k ctx
1315.2 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 14B Q4_K prefill @ 4k ctx
2144.2 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 8B Q4_K decode @ 16k ctx
59.1 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-8b-q4k-tg-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 8B Q4_K decode @ 32k ctx
43.6 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 8B Q4_K decode @ 4k ctx
85.8 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 8B Q4_K prefill @ 16k ctx
1600.8 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 8B Q4_K prefill @ 32k ctx
898.8 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 5070 Qwen3 8B Q4_K prefill @ 4k ctx
3487.7 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B Q4_K on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-rtx5070-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.70
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) decode @ 128k ctx
57.5 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) decode @ 16k ctx
137.1 tok/s (decode, per-request, batch 1, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) decode @ 32k ctx
116.3 tok/s (decode, per-request, batch 1, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) decode @ 4k ctx
160.8 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) decode @ 64k ctx
88.4 tok/s (decode, per-request, batch 1, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) prefill @ 128k ctx
1177.7 tok/s (prefill, per-request, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) prefill @ 16k ctx
5350.5 tok/s (prefill, per-request, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) prefill @ 32k ctx
3818.7 tok/s (prefill, per-request, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) prefill @ 4k ctx
6495.8 tok/s (prefill, per-request, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada gpt-oss 20B (MXFP4) prefill @ 64k ctx
2270.3 tok/s (prefill, per-request, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B (Q4_K) decode @ 16k ctx
13.7 tok/s (decode, per-request, batch 1, KV depth 16k)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B (Q4_K) decode @ 4k ctx
18.1 tok/s (decode, per-request, batch 1, KV depth 4k)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B (Q4_K) prefill @ 16k ctx
526.0 tok/s (prefill, per-request, KV depth 16k)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B (Q4_K) prefill @ 4k ctx
678.1 tok/s (prefill, per-request, KV depth 4k)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B Q4_K decode @ 16k ctx
13.7 tok/s (decode, per-request, batch 1, KV depth 16k)Llama 3.3 70B on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-rtx6000ada-llama33-70b-q4k-decode-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B Q4_K decode @ 4k ctx
18.1 tok/s (decode, per-request, batch 1, KV depth 4k)Llama 3.3 70B on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-rtx6000ada-llama33-70b-q4k-decode-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B Q4_K prefill @ 16k ctx
526.0 tok/s (prefill, per-request, KV depth 16k)Llama 3.3 70B on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-rtx6000ada-llama33-70b-q4k-prefill-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Llama 3.3 70B Q4_K prefill @ 4k ctx
678.1 tok/s (prefill, per-request, KV depth 4k)Llama 3.3 70B on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-rtx6000ada-llama33-70b-q4k-prefill-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) decode @ 128k ctx
15.2 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) decode @ 16k ctx
58.5 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) decode @ 32k ctx
42.3 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) decode @ 4k ctx
81.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) decode @ 64k ctx
27.0 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) prefill @ 128k ctx
358.6 tok/s (prefill, per-request, KV depth 128k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) prefill @ 16k ctx
2367.2 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) prefill @ 32k ctx
1310.2 tok/s (prefill, per-request, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) prefill @ 4k ctx
2912.1 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 14B (Q4_K) prefill @ 64k ctx
727.4 tok/s (prefill, per-request, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) decode @ 128k ctx
20.7 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) decode @ 16k ctx
120.1 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) decode @ 32k ctx
74.3 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) decode @ 4k ctx
199.6 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) decode @ 64k ctx
40.8 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) prefill @ 128k ctx
448.6 tok/s (prefill, per-request, KV depth 128k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) prefill @ 16k ctx
3156.5 tok/s (prefill, per-request, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) prefill @ 32k ctx
2010.6 tok/s (prefill, per-request, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) prefill @ 4k ctx
3888.0 tok/s (prefill, per-request, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B A3B (Q4_K) prefill @ 64k ctx
913.2 tok/s (prefill, per-request, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 30B-A3B Q4_K decode @ 128k ctx
20.7 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 30B-A3B on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-rtx6000ada-qwen3-30b-a3b-q4k-decode-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) decode @ 16k ctx
25.1 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) decode @ 32k ctx
18.2 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) decode @ 4k ctx
37.4 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) decode @ 64k ctx
11.8 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) prefill @ 16k ctx
977.5 tok/s (prefill, per-request, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) prefill @ 32k ctx
595.5 tok/s (prefill, per-request, KV depth 32k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) prefill @ 4k ctx
1203.9 tok/s (prefill, per-request, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B (Q4_K) prefill @ 64k ctx
348.6 tok/s (prefill, per-request, KV depth 64k)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 32B Q4_K decode @ 64k ctx
11.8 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 32B on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-rtx6000ada-qwen3-32b-q4k-decode-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) decode @ 128k ctx
20.8 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) decode @ 16k ctx
98.7 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) decode @ 32k ctx
68.9 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) decode @ 4k ctx
133.9 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) decode @ 64k ctx
39.9 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) prefill @ 128k ctx
541.3 tok/s (prefill, per-request, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) prefill @ 16k ctx
4096.4 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) prefill @ 32k ctx
2218.5 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) prefill @ 4k ctx
4932.5 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX 6000 Ada Qwen3 8B (Q4_K) prefill @ 64k ctx
1152.6 tok/s (prefill, per-request, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) decode @ 128k ctx
33.4 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) decode @ 16k ctx
91.2 tok/s (decode, per-request, batch 1, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) decode @ 32k ctx
74.4 tok/s (decode, per-request, batch 1, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) decode @ 4k ctx
108.4 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) decode @ 64k ctx
53.3 tok/s (decode, per-request, batch 1, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) prefill @ 128k ctx
702.8 tok/s (prefill, per-request, KV depth 128k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) prefill @ 16k ctx
2649.2 tok/s (prefill, per-request, KV depth 16k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) prefill @ 32k ctx
1891.2 tok/s (prefill, per-request, KV depth 32k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) prefill @ 4k ctx
3490.6 tok/s (prefill, per-request, KV depth 4k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 gpt-oss 20B (MXFP4) prefill @ 64k ctx
1207.2 tok/s (prefill, per-request, KV depth 64k)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B (Q4_K) decode @ 16k ctx
9.1 tok/s (decode, per-request, batch 1, KV depth 16k)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B (Q4_K) decode @ 4k ctx
12.6 tok/s (decode, per-request, batch 1, KV depth 4k)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B (Q4_K) prefill @ 16k ctx
280.3 tok/s (prefill, per-request, KV depth 16k)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B (Q4_K) prefill @ 4k ctx
423.0 tok/s (prefill, per-request, KV depth 4k)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B Q4_K decode @ 16k ctx
9.1 tok/s (decode, per-request, batch 1, KV depth 16k)Llama 3.3 70B on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-decode-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B Q4_K decode @ 4k ctx
12.6 tok/s (decode, per-request, batch 1, KV depth 4k)Llama 3.3 70B on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-decode-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B Q4_K prefill @ 16k ctx
280.3 tok/s (prefill, per-request, KV depth 16k)Llama 3.3 70B on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-prefill-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Llama 3.3 70B Q4_K prefill @ 4k ctx
423.0 tok/s (prefill, per-request, KV depth 4k)Llama 3.3 70B on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-prefill-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) decode @ 128k ctx
12.1 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) decode @ 16k ctx
40.7 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) decode @ 32k ctx
30.8 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) decode @ 4k ctx
53.8 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) decode @ 64k ctx
20.5 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) prefill @ 128k ctx
382.4 tok/s (prefill, per-request, KV depth 128k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) prefill @ 16k ctx
1529.0 tok/s (prefill, per-request, KV depth 16k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) prefill @ 32k ctx
1063.4 tok/s (prefill, per-request, KV depth 32k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) prefill @ 4k ctx
1713.3 tok/s (prefill, per-request, KV depth 4k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 14B (Q4_K) prefill @ 64k ctx
668.9 tok/s (prefill, per-request, KV depth 64k)Qwen3 14B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-14b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) decode @ 128k ctx
14.8 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) decode @ 16k ctx
75.4 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) decode @ 32k ctx
48.1 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) decode @ 4k ctx
131.2 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) decode @ 64k ctx
27.6 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) prefill @ 128k ctx
429.2 tok/s (prefill, per-request, KV depth 128k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) prefill @ 16k ctx
1901.1 tok/s (prefill, per-request, KV depth 16k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) prefill @ 32k ctx
1275.6 tok/s (prefill, per-request, KV depth 32k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) prefill @ 4k ctx
2142.4 tok/s (prefill, per-request, KV depth 4k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B A3B (Q4_K) prefill @ 64k ctx
764.2 tok/s (prefill, per-request, KV depth 64k)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 30B-A3B Q4_K decode @ 128k ctx
14.8 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 30B-A3B on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-q4k-decode-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) decode @ 16k ctx
18.3 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) decode @ 32k ctx
13.5 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) decode @ 4k ctx
24.6 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) decode @ 64k ctx
8.7 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) prefill @ 16k ctx
704.8 tok/s (prefill, per-request, KV depth 16k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) prefill @ 32k ctx
502.7 tok/s (prefill, per-request, KV depth 32k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) prefill @ 4k ctx
736.2 tok/s (prefill, per-request, KV depth 4k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B (Q4_K) prefill @ 64k ctx
319.0 tok/s (prefill, per-request, KV depth 64k)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 32B Q4_K decode @ 64k ctx
8.7 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 32B on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-q4k-decode-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) decode @ 128k ctx
16.0 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) decode @ 16k ctx
64.3 tok/s (decode, per-request, batch 1, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) decode @ 32k ctx
44.6 tok/s (decode, per-request, batch 1, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-tg-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) decode @ 4k ctx
90.5 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) decode @ 64k ctx
27.9 tok/s (decode, per-request, batch 1, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) prefill @ 128k ctx
527.9 tok/s (prefill, per-request, KV depth 128k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) prefill @ 16k ctx
2428.0 tok/s (prefill, per-request, KV depth 16k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) prefill @ 32k ctx
1593.3 tok/s (prefill, per-request, KV depth 32k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-pp-32k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) prefill @ 4k ctx
2756.0 tok/s (prefill, per-request, KV depth 4k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX A6000 Qwen3 8B (Q4_K) prefill @ 64k ctx
951.0 tok/s (prefill, per-request, KV depth 64k)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 26B Q4_K decode @ 256k ctx
112.5 tok/s (decode, per-request, batch 1, KV depth 256k)Gemma4 26B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-26b-q4k-tg-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 26B Q4_K decode @ 4k ctx
196.9 tok/s (decode, per-request, batch 1, KV depth 4k)Gemma4 26B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-26b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 26B Q4_K prefill @ 256k ctx
2245.7 tok/s (prefill, per-request, KV depth 256k)Gemma4 26B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-26b-q4k-pp-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 26B Q4_K prefill @ 4k ctx
9437.2 tok/s (prefill, per-request, KV depth 4k)Gemma4 26B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-26b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 31B Q4_K decode @ 256k ctx
34.4 tok/s (decode, per-request, batch 1, KV depth 256k)Gemma4 31B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-31b-q4k-tg-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 31B Q4_K decode @ 4k ctx
61.6 tok/s (decode, per-request, batch 1, KV depth 4k)Gemma4 31B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-31b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 31B Q4_K prefill @ 256k ctx
506.8 tok/s (prefill, per-request, KV depth 256k)Gemma4 31B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-31b-q4k-pp-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Gemma4 31B Q4_K prefill @ 4k ctx
3749.8 tok/s (prefill, per-request, KV depth 4k)Gemma4 31B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gemma4-31b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell GLM 4.5 Air 106B Q4_K decode @ 128k ctx
17.7 tok/s (decode, per-request, batch 1, KV depth 128k)GLM 4.5 Air 106B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-glm45-air-106b-q4k-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell GLM 4.5 Air 106B Q4_K decode @ 4k ctx
101.1 tok/s (decode, per-request, batch 1, KV depth 4k)GLM 4.5 Air 106B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-glm45-air-106b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell GLM 4.5 Air 106B Q4_K prefill @ 128k ctx
320.2 tok/s (prefill, per-request, KV depth 128k)GLM 4.5 Air 106B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-glm45-air-106b-q4k-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell GLM 4.5 Air 106B Q4_K prefill @ 4k ctx
2795.2 tok/s (prefill, per-request, KV depth 4k)GLM 4.5 Air 106B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-glm45-air-106b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B MXFP4 decode @ 128k ctx
99.8 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 120B MXFP4 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-mxfp4-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B MXFP4 decode @ 4k ctx
210.1 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 120B MXFP4 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-mxfp4-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B MXFP4 prefill @ 128k ctx
1289.8 tok/s (prefill, per-request, KV depth 128k)gpt-oss 120B MXFP4 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-mxfp4-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B MXFP4 prefill @ 4k ctx
4578.6 tok/s (prefill, per-request, KV depth 4k)gpt-oss 120B MXFP4 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-mxfp4-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B Q8_0 decode @ 128k ctx
123.3 tok/s (decode, per-request, batch 1, KV depth 128k)gpt-oss 120B Q8_0 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-q8-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B Q8_0 decode @ 4k ctx
221.3 tok/s (decode, per-request, batch 1, KV depth 4k)gpt-oss 120B Q8_0 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-q8-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B Q8_0 prefill @ 128k ctx
2036.6 tok/s (prefill, per-request, KV depth 128k)gpt-oss 120B Q8_0 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-q8-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell gpt-oss 120B Q8_0 prefill @ 4k ctx
6512.8 tok/s (prefill, per-request, KV depth 4k)gpt-oss 120B Q8_0 on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-gptoss-120b-q8-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Llama 3.3 70B Q4_K decode @ 128k ctx
16.6 tok/s (decode, per-request, batch 1, KV depth 128k)Llama 3.3 70B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-llama33-70b-q4k-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Llama 3.3 70B Q4_K decode @ 16k ctx
28.2 tok/s (decode, per-request, batch 1, KV depth 16k)Llama 3.3 70B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-llama33-70b-q4k-tg-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Llama 3.3 70B Q4_K decode @ 4k ctx
32.0 tok/s (decode, per-request, batch 1, KV depth 4k)Llama 3.3 70B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-llama33-70b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Llama 3.3 70B Q4_K prefill @ 128k ctx
266.1 tok/s (prefill, per-request, KV depth 128k)Llama 3.3 70B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-llama33-70b-q4k-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Llama 3.3 70B Q4_K prefill @ 16k ctx
1355.4 tok/s (prefill, per-request, KV depth 16k)Llama 3.3 70B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-llama33-70b-q4k-pp-16k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Llama 3.3 70B Q4_K prefill @ 4k ctx
1689.1 tok/s (prefill, per-request, KV depth 4k)Llama 3.3 70B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-llama33-70b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Mistral 123B Q4_K decode @ 4k ctx
19.1 tok/s (decode, per-request, batch 1, KV depth 4k)Mistral 123B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-mistral-123b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Mistral 123B Q4_K decode @ 64k ctx
10.5 tok/s (decode, per-request, batch 1, KV depth 64k)Mistral 123B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-mistral-123b-q4k-tg-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Mistral 123B Q4_K prefill @ 4k ctx
994.2 tok/s (prefill, per-request, KV depth 4k)Mistral 123B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-mistral-123b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Mistral 123B Q4_K prefill @ 64k ctx
343.1 tok/s (prefill, per-request, KV depth 64k)Mistral 123B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-mistral-123b-q4k-pp-64k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 30B A3B Q4_K decode @ 256k ctx
38.4 tok/s (decode, per-request, batch 1, KV depth 256k)Qwen3 30B A3B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-30b-a3b-q4k-tg-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 30B A3B Q4_K decode @ 4k ctx
198.7 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 30B A3B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-30b-a3b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 30B A3B Q4_K prefill @ 256k ctx
637.4 tok/s (prefill, per-request, KV depth 256k)Qwen3 30B A3B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-30b-a3b-q4k-pp-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 30B A3B Q4_K prefill @ 4k ctx
6546.0 tok/s (prefill, per-request, KV depth 4k)Qwen3 30B A3B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-30b-a3b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 32B Q4_K decode @ 128k ctx
23.1 tok/s (decode, per-request, batch 1, KV depth 128k)Qwen3 32B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-32b-q4k-tg-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 32B Q4_K decode @ 4k ctx
54.9 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3 32B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-32b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 32B Q4_K prefill @ 128k ctx
330.4 tok/s (prefill, per-request, KV depth 128k)Qwen3 32B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-32b-q4k-pp-128k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3 32B Q4_K prefill @ 4k ctx
3137.2 tok/s (prefill, per-request, KV depth 4k)Qwen3 32B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen3-32b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 122B Q4_K decode @ 256k ctx
65.6 tok/s (decode, per-request, batch 1, KV depth 256k)Qwen3.5 122B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-122b-q4k-tg-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 122B Q4_K decode @ 4k ctx
98.4 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3.5 122B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-122b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 122B Q4_K prefill @ 256k ctx
1013.4 tok/s (prefill, per-request, KV depth 256k)Qwen3.5 122B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-122b-q4k-pp-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 122B Q4_K prefill @ 4k ctx
3055.0 tok/s (prefill, per-request, KV depth 4k)Qwen3.5 122B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-122b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 27B Q4_K decode @ 256k ctx
36.3 tok/s (decode, per-request, batch 1, KV depth 256k)Qwen3.5 27B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-27b-q4k-tg-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 27B Q4_K decode @ 4k ctx
60.9 tok/s (decode, per-request, batch 1, KV depth 4k)Qwen3.5 27B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-27b-q4k-tg-4k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 27B Q4_K prefill @ 256k ctx
903.1 tok/s (prefill, per-request, KV depth 256k)Qwen3.5 27B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-27b-q4k-pp-256k
verifiedmedium · 0.75
Hardware Corner Mar 2026: RTX PRO 6000 Blackwell Qwen3.5 27B Q4_K prefill @ 4k ctx
3338.5 tok/s (prefill, per-request, KV depth 4k)Qwen3.5 27B Q4_K on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/hc26-rtxpro6000-qwen35-27b-q4k-pp-4k
verifiedmedium · 0.75
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1315.2 tok/s at 16k context
1315.2 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2144.2 tok/s at 4k context
2144.2 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 14B Q4_K GGUF token generation 40.6 tok/s at 16k context
40.6 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 14B Q4_K GGUF token generation 54.2 tok/s at 4k context
54.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1600.8 tok/s at 16k context
1600.8 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 3487.7 tok/s at 4k context
3487.7 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 898.8 tok/s at 32k context
898.8 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 8B Q4_K GGUF token generation 43.6 tok/s at 32k context
43.6 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 8B Q4_K GGUF token generation 59.1 tok/s at 16k context
59.1 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: local-nvidia-rtx-5070 llama.cpp Qwen3 8B Q4_K GGUF token generation 85.8 tok/s at 4k context
85.8 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwc26-rtx-5070-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 678.2 tok/s at 16k context
678.2 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 972.6 tok/s at 4k context
972.6 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 14B Q4_K GGUF token generation 22.7 tok/s at 16k context
22.7 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 14B Q4_K GGUF token generation 31.2 tok/s at 4k context
31.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1119.2 tok/s at 16k context
1119.2 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1696.8 tok/s at 4k context
1696.8 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 764.7 tok/s at 32k context
764.7 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 8B Q4_K GGUF token generation 31.9 tok/s at 32k context
31.9 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 8B Q4_K GGUF token generation 42.0 tok/s at 16k context
42.0 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3060 llama.cpp Qwen3 8B Q4_K GGUF token generation 55.2 tok/s at 4k context
55.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwc26-rtx-3060-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1773.9 tok/s at 16k context
1773.9 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2600.1 tok/s at 4k context
2600.1 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 52.3 tok/s at 16k context
52.3 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 69.9 tok/s at 4k context
69.9 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1761.2 tok/s at 32k context
1761.2 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 2658.5 tok/s at 16k context
2658.5 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 4211.7 tok/s at 4k context
4211.7 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 115.2 tok/s at 4k context
115.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 68.1 tok/s at 32k context
68.1 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3080-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 87.9 tok/s at 16k context
87.9 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwc26-rtx-3080-ti-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Gemma4 31B Q4_K GGUF prompt processing 1155.8 tok/s at 4k context
1155.8 tok/s per-request prefill (batch 1, ~4096-token context tier)Gemma4 31B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gemma4-31b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Gemma4 31B Q4_K GGUF prompt processing 723.7 tok/s at 32k context
723.7 tok/s per-request prefill (batch 1, ~32768-token context tier)Gemma4 31B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gemma4-31b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Gemma4 31B Q4_K GGUF token generation 31.4 tok/s at 32k context
31.4 tok/s per-request decode (batch 1, ~32768-token context tier)Gemma4 31B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gemma4-31b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Gemma4 31B Q4_K GGUF token generation 34.7 tok/s at 4k context
34.7 tok/s per-request decode (batch 1, ~4096-token context tier)Gemma4 31B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gemma4-31b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 4400.3 tok/s at 4k context
4400.3 tok/s per-request prefill (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gptoss-20b-mxfp4-pp-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 923.8 tok/s at 128k context
923.8 tok/s per-request prefill (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gptoss-20b-mxfp4-pp-128k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp gpt-oss-20B MXFP4 GGUF token generation 147.5 tok/s at 4k context
147.5 tok/s per-request decode (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gptoss-20b-mxfp4-tg-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp gpt-oss-20B MXFP4 GGUF token generation 62.2 tok/s at 128k context
62.2 tok/s per-request decode (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-gptoss-20b-mxfp4-tg-128k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2459.0 tok/s at 4k context
2459.0 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-14b-q4k-pp-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 14B Q4_K GGUF token generation 25.4 tok/s at 64k context
25.4 tok/s per-request decode (batch 1, ~65536-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-14b-q4k-tg-64k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 14B Q4_K GGUF token generation 70.0 tok/s at 4k context
70.0 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-14b-q4k-tg-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 30B A3B Q4_K GGUF prompt processing 1959.0 tok/s at 16k context
1959.0 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 30B A3B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-30b-a3b-q4k-pp-16k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 30B A3B Q4_K GGUF prompt processing 2988.6 tok/s at 4k context
2988.6 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 30B A3B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-30b-a3b-q4k-pp-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 30B A3B Q4_K GGUF prompt processing 800.9 tok/s at 64k context
800.9 tok/s per-request prefill (batch 1, ~65536-token context tier)Qwen3 30B A3B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-30b-a3b-q4k-pp-64k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 30B A3B Q4_K GGUF token generation 113.8 tok/s at 16k context
113.8 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 30B A3B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-30b-a3b-q4k-tg-16k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 30B A3B Q4_K GGUF token generation 153.6 tok/s at 4k context
153.6 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 30B A3B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-30b-a3b-q4k-tg-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 30B A3B Q4_K GGUF token generation 58.3 tok/s at 64k context
58.3 tok/s per-request decode (batch 1, ~65536-token context tier)Qwen3 30B A3B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-30b-a3b-q4k-tg-64k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 32B Q4_K GGUF prompt processing 1087.9 tok/s at 4k context
1087.9 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 32B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-32b-q4k-pp-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 32B Q4_K GGUF prompt processing 767.8 tok/s at 16k context
767.8 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 32B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-32b-q4k-pp-16k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 32B Q4_K GGUF token generation 30.3 tok/s at 16k context
30.3 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 32B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-32b-q4k-tg-16k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 32B Q4_K GGUF token generation 35.1 tok/s at 4k context
35.1 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 32B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-32b-q4k-tg-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 4049.6 tok/s at 4k context
4049.6 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-8b-q4k-pp-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 8B Q4_K GGUF token generation 115.3 tok/s at 4k context
115.3 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-8b-q4k-tg-4k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3 8B Q4_K GGUF token generation 28.1 tok/s at 128k context
28.1 tok/s per-request decode (batch 1, ~131072-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen3-8b-q4k-tg-128k
verifiedlow · 0.62
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3.5 35B MXFP4 GGUF prompt processing 1288.9 tok/s at 128k context
1288.9 tok/s per-request prefill (batch 1, ~131072-token context tier)Qwen3.5 35B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen35-35b-mxfp4-pp-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3.5 35B MXFP4 GGUF prompt processing 2622.1 tok/s at 4k context
2622.1 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3.5 35B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen35-35b-mxfp4-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3.5 35B MXFP4 GGUF token generation 111.2 tok/s at 4k context
111.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3.5 35B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen35-35b-mxfp4-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-3090 llama.cpp Qwen3.5 35B MXFP4 GGUF token generation 79.4 tok/s at 128k context
79.4 tok/s per-request decode (batch 1, ~131072-token context tier)Qwen3.5 35B MXFP4 GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hwc26-rtx-3090-qwen35-35b-mxfp4-tg-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 1332.2 tok/s at 64k context
1332.2 tok/s per-request prefill (batch 1, ~65536-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 1964.9 tok/s at 32k context
1964.9 tok/s per-request prefill (batch 1, ~32768-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 2552.9 tok/s at 16k context
2552.9 tok/s per-request prefill (batch 1, ~16384-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 3274.2 tok/s at 4k context
3274.2 tok/s per-request prefill (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 780.2 tok/s at 128k context
780.2 tok/s per-request prefill (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-pp-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF token generation 31.1 tok/s at 128k context
31.1 tok/s per-request decode (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-tg-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF token generation 41.1 tok/s at 64k context
41.1 tok/s per-request decode (batch 1, ~65536-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF token generation 51.5 tok/s at 32k context
51.5 tok/s per-request decode (batch 1, ~32768-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF token generation 57.8 tok/s at 16k context
57.8 tok/s per-request decode (batch 1, ~16384-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp gpt-oss-20B MXFP4 GGUF token generation 63.2 tok/s at 4k context
63.2 tok/s per-request decode (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-gptoss-20b-mxfp4-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1645.7 tok/s at 4k context
1645.7 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 541.4 tok/s at 32k context
541.4 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-14b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 917.6 tok/s at 16k context
917.6 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 17.9 tok/s at 32k context
17.9 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-14b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 22.4 tok/s at 16k context
22.4 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 27.4 tok/s at 4k context
27.4 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1480.8 tok/s at 16k context
1480.8 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 2675.2 tok/s at 4k context
2675.2 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 392.1 tok/s at 64k context
392.1 tok/s per-request prefill (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 760.3 tok/s at 32k context
760.3 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 13.0 tok/s at 64k context
13.0 tok/s per-request decode (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 25.5 tok/s at 32k context
25.5 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 34.3 tok/s at 16k context
34.3 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4060-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 45.8 tok/s at 4k context
45.8 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwc26-rtx-4060-ti-16gb-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1355.8 tok/s at 16k context
1355.8 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2099.9 tok/s at 4k context
2099.9 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 14B Q4_K GGUF token generation 32.7 tok/s at 16k context
32.7 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 14B Q4_K GGUF token generation 42.5 tok/s at 4k context
42.5 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1116.7 tok/s at 32k context
1116.7 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 2064.3 tok/s at 16k context
2064.3 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 3564.1 tok/s at 4k context
3564.1 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 8B Q4_K GGUF token generation 38.1 tok/s at 32k context
38.1 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 8B Q4_K GGUF token generation 52.1 tok/s at 16k context
52.1 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070 llama.cpp Qwen3 8B Q4_K GGUF token generation 71.2 tok/s at 4k context
71.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwc26-rtx-4070-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1578.1 tok/s at 16k context
1578.1 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2522.2 tok/s at 4k context
2522.2 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 14B Q4_K GGUF token generation 37.2 tok/s at 16k context
37.2 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 14B Q4_K GGUF token generation 45.5 tok/s at 4k context
45.5 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1595.7 tok/s at 32k context
1595.7 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 2525.8 tok/s at 16k context
2525.8 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 4321.7 tok/s at 4k context
4321.7 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 8B Q4_K GGUF token generation 42.2 tok/s at 32k context
42.2 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 8B Q4_K GGUF token generation 56.2 tok/s at 16k context
56.2 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-super llama.cpp Qwen3 8B Q4_K GGUF token generation 75.4 tok/s at 4k context
75.4 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwc26-rtx-4070-super-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1951.4 tok/s at 16k context
1951.4 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2866.6 tok/s at 4k context
2866.6 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 37.8 tok/s at 16k context
37.8 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 14B Q4_K GGUF token generation 45.8 tok/s at 4k context
45.8 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1099.7 tok/s at 32k context
1099.7 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 2274.8 tok/s at 16k context
2274.8 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 8B Q4_K GGUF prompt processing 4877.7 tok/s at 4k context
4877.7 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 42.1 tok/s at 32k context
42.1 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 57.6 tok/s at 16k context
57.6 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti llama.cpp Qwen3 8B Q4_K GGUF token generation 75.8 tok/s at 4k context
75.8 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwc26-rtx-4070-ti-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 1716.2 tok/s at 64k context
1716.2 tok/s per-request prefill (batch 1, ~65536-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 3020.0 tok/s at 32k context
3020.0 tok/s per-request prefill (batch 1, ~32768-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 4182.7 tok/s at 16k context
4182.7 tok/s per-request prefill (batch 1, ~16384-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 5628.1 tok/s at 4k context
5628.1 tok/s per-request prefill (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 993.5 tok/s at 128k context
993.5 tok/s per-request prefill (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 113.6 tok/s at 16k context
113.6 tok/s per-request decode (batch 1, ~16384-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 128.9 tok/s at 4k context
128.9 tok/s per-request decode (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 57.5 tok/s at 128k context
57.5 tok/s per-request decode (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 78.7 tok/s at 64k context
78.7 tok/s per-request decode (batch 1, ~65536-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 98.7 tok/s at 32k context
98.7 tok/s per-request decode (batch 1, ~32768-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1236.9 tok/s at 32k context
1236.9 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-14b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2003.4 tok/s at 16k context
2003.4 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 3048.0 tok/s at 4k context
3048.0 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 14B Q4_K GGUF token generation 37.7 tok/s at 32k context
37.7 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-14b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 14B Q4_K GGUF token generation 47.2 tok/s at 16k context
47.2 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 14B Q4_K GGUF token generation 58.1 tok/s at 4k context
58.1 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1616.6 tok/s at 32k context
1616.6 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 3050.7 tok/s at 16k context
3050.7 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 5220.1 tok/s at 4k context
5220.1 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 829.4 tok/s at 64k context
829.4 tok/s per-request prefill (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF token generation 36.6 tok/s at 64k context
36.6 tok/s per-request decode (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF token generation 54.5 tok/s at 32k context
54.5 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF token generation 72.2 tok/s at 16k context
72.2 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4070-ti-super llama.cpp Qwen3 8B Q4_K GGUF token generation 96.3 tok/s at 4k context
96.3 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwc26-rtx-4070-ti-super-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 6218.5 tok/s at 4k context
6218.5 tok/s per-request prefill (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-gptoss-20b-mxfp4-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 898.5 tok/s at 128k context
898.5 tok/s per-request prefill (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-gptoss-20b-mxfp4-pp-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp gpt-oss-20B MXFP4 GGUF token generation 136.5 tok/s at 4k context
136.5 tok/s per-request decode (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-gptoss-20b-mxfp4-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp gpt-oss-20B MXFP4 GGUF token generation 60.0 tok/s at 128k context
60.0 tok/s per-request decode (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-gptoss-20b-mxfp4-tg-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1395.7 tok/s at 32k context
1395.7 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-14b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 14B Q4_K GGUF prompt processing 3574.7 tok/s at 4k context
3574.7 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 14B Q4_K GGUF token generation 40.6 tok/s at 32k context
40.6 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-14b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 14B Q4_K GGUF token generation 62.0 tok/s at 4k context
62.0 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 6177.9 tok/s at 4k context
6177.9 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 8B Q4_K GGUF prompt processing 937.9 tok/s at 64k context
937.9 tok/s per-request prefill (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-8b-q4k-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 8B Q4_K GGUF token generation 102.7 tok/s at 4k context
102.7 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080 llama.cpp Qwen3 8B Q4_K GGUF token generation 39.0 tok/s at 64k context
39.0 tok/s per-request decode (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwc26-rtx-4080-qwen3-8b-q4k-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 1145.0 tok/s at 128k context
1145.0 tok/s per-request prefill (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-pp-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 1961.5 tok/s at 64k context
1961.5 tok/s per-request prefill (batch 1, ~65536-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 3328.1 tok/s at 32k context
3328.1 tok/s per-request prefill (batch 1, ~32768-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 4708.7 tok/s at 16k context
4708.7 tok/s per-request prefill (batch 1, ~16384-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF prompt processing 6364.0 tok/s at 4k context
6364.0 tok/s per-request prefill (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 107.2 tok/s at 32k context
107.2 tok/s per-request decode (batch 1, ~32768-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 123.0 tok/s at 16k context
123.0 tok/s per-request decode (batch 1, ~16384-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 139.1 tok/s at 4k context
139.1 tok/s per-request decode (batch 1, ~4096-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 60.0 tok/s at 128k context
60.0 tok/s per-request decode (batch 1, ~131072-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-tg-128k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp gpt-oss-20B MXFP4 GGUF token generation 81.5 tok/s at 64k context
81.5 tok/s per-request decode (batch 1, ~65536-token context tier)gpt-oss-20B MXFP4 GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-gptoss-20b-mxfp4-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 1769.3 tok/s at 32k context
1769.3 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-14b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 2526.7 tok/s at 16k context
2526.7 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-14b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 14B Q4_K GGUF prompt processing 3745.0 tok/s at 4k context
3745.0 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-14b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 14B Q4_K GGUF token generation 42.6 tok/s at 32k context
42.6 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-14b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 14B Q4_K GGUF token generation 52.8 tok/s at 16k context
52.8 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-14b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 14B Q4_K GGUF token generation 64.2 tok/s at 4k context
64.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-14b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 1501.5 tok/s at 64k context
1501.5 tok/s per-request prefill (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-pp-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 2537.1 tok/s at 32k context
2537.1 tok/s per-request prefill (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-pp-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 3858.1 tok/s at 16k context
3858.1 tok/s per-request prefill (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-pp-16k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF prompt processing 6137.0 tok/s at 4k context
6137.0 tok/s per-request prefill (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-pp-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF token generation 104.2 tok/s at 4k context
104.2 tok/s per-request decode (batch 1, ~4096-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-tg-4k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF token generation 39.1 tok/s at 64k context
39.1 tok/s per-request decode (batch 1, ~65536-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-tg-64k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF token generation 59.5 tok/s at 32k context
59.5 tok/s per-request decode (batch 1, ~32768-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-tg-32k
verifiedlow · 0.60
Hardware Corner Mar-2026: nvidia-rtx-4080-super llama.cpp Qwen3 8B Q4_K GGUF token generation 79.4 tok/s at 16k context
79.4 tok/s per-request decode (batch 1, ~16384-token context tier)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwc26-rtx-4080-super-qwen3-8b-q4k-tg-16k
verifiedlow · 0.60
Hardware Corner Qwen3.8-27B Q4_K_S: Apple M5 Max 128GB unified pp 834.1 t/s @4k ctx
834.1 tok/s per-request prefill (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on local-apple-m5-max via llama-cppbenchmarks/hc38-apple-m5-max-qwen38-27b-pp-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: Apple M5 Max 128GB unified tg 18.2 t/s @64k ctx
18.2 tok/s per-request decode (batch 1, 64k context)Qwen3.8 27B Q4_K Small GGUF on local-apple-m5-max via llama-cppbenchmarks/hc38-apple-m5-max-qwen38-27b-tg-64k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: Apple M5 Max 128GB unified tg 31.4 t/s @4k ctx
31.4 tok/s per-request decode (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on local-apple-m5-max via llama-cppbenchmarks/hc38-apple-m5-max-qwen38-27b-tg-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: dual RTX 5060 Ti 16GB (2x15849 MiB) pp 1167.7 t/s @4k ctx
1167.7 tok/s per-request prefill (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc38-dual-rtx-5060-ti-qwen38-27b-pp-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: dual RTX 5060 Ti 16GB (2x15849 MiB) tg 15.55 t/s @128k ctx
15.55 tok/s per-request decode (batch 1, 128k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc38-dual-rtx-5060-ti-qwen38-27b-tg-128k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: dual RTX 5060 Ti 16GB (2x15849 MiB) tg 22.47 t/s @4k ctx
22.47 tok/s per-request decode (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hc38-dual-rtx-5060-ti-qwen38-27b-tg-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: NVIDIA GB10 (Dell Pro Max / DGX Spark) 128GB unified pp 812.6 t/s @4k ctx
812.6 tok/s per-request prefill (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-gb10 via llama-cppbenchmarks/hc38-nvidia-gb10-qwen38-27b-pp-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: NVIDIA GB10 (Dell Pro Max / DGX Spark) 128GB unified tg 12.2 t/s @4k ctx
12.2 tok/s per-request decode (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-gb10 via llama-cppbenchmarks/hc38-nvidia-gb10-qwen38-27b-tg-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: NVIDIA GB10 (Dell Pro Max / DGX Spark) 128GB unified tg 5.94 t/s @256k ctx
5.94 tok/s per-request decode (batch 1, 256k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-gb10 via llama-cppbenchmarks/hc38-nvidia-gb10-qwen38-27b-tg-256k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 3090 24GB pp 1308.05 t/s @4k ctx
1308.05 tok/s per-request prefill (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc38-rtx-3090-qwen38-27b-pp-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 3090 24GB tg 33.95 t/s @64k ctx
33.95 tok/s per-request decode (batch 1, 64k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc38-rtx-3090-qwen38-27b-tg-64k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 3090 24GB tg 40.31 t/s @4k ctx
40.31 tok/s per-request decode (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-3090 via llama-cppbenchmarks/hc38-rtx-3090-qwen38-27b-tg-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 4090 24GB pp 2962.59 t/s @4k ctx
2962.59 tok/s per-request prefill (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc38-rtx-4090-qwen38-27b-pp-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 4090 24GB tg 46.16 t/s @4k ctx
46.16 tok/s per-request decode (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hc38-rtx-4090-qwen38-27b-tg-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 5090 32GB pp 3749.51 t/s @4k ctx
3749.51 tok/s per-request prefill (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc38-rtx-5090-qwen38-27b-pp-4k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 5090 32GB tg 22.79 t/s @128k ctx
22.79 tok/s per-request decode (batch 1, 128k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc38-rtx-5090-qwen38-27b-tg-128k
verifiedmedium · 0.70
Hardware Corner Qwen3.8-27B Q4_K_S: RTX 5090 32GB tg 74.83 t/s @4k ctx
74.83 tok/s per-request decode (batch 1, 4k context)Qwen3.8 27B Q4_K Small GGUF on nvidia-rtx-5090 via llama-cppbenchmarks/hc38-rtx-5090-qwen38-27b-tg-4k
verifiedmedium · 0.70
Hardware Corner: RTX 3060 12GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 678.15 t/s @16k ctx
678.15 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwcorner-rtx-3060-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3060 12GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 22.66 t/s @16k ctx
22.66 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwcorner-rtx-3060-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3060 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 31.86 t/s @32k ctx
31.86 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwcorner-rtx-3060-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3060 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1119.23 t/s @16k ctx
1119.23 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwcorner-rtx-3060-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3060 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 764.71 t/s @32k ctx
764.71 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwcorner-rtx-3060-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3060 12GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 41.97 t/s @16k ctx
41.97 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3060 via llama-cppbenchmarks/hwcorner-rtx-3060-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3080 10GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 55.7 t/s @32k ctx
55.7 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3080-10gb via llama-cppbenchmarks/hwcorner-rtx-3080-10gb-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3080 10GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1525.33 t/s @32k ctx
1525.33 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3080-10gb via llama-cppbenchmarks/hwcorner-rtx-3080-10gb-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3080 10GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 2287.66 t/s @16k ctx
2287.66 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3080-10gb via llama-cppbenchmarks/hwcorner-rtx-3080-10gb-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3080 10GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 74.24 t/s @16k ctx
74.24 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3080-10gb via llama-cppbenchmarks/hwcorner-rtx-3080-10gb-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1773.9 t/s @16k ctx
1773.9 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2600.1 t/s @4k ctx
2600.1 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 69.9 t/s @4k ctx
69.9 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 52.29 t/s @16k ctx
52.29 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1761.2 t/s @32k ctx
1761.2 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 2658.5 t/s @16k ctx
2658.5 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 4211.7 t/s @4k ctx
4211.7 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 115.2 t/s @4k ctx
115.2 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 68.1 t/s @32k ctx
68.1 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 3080 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 87.94 t/s @16k ctx
87.94 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3080-ti via llama-cppbenchmarks/hwcorner-rtx-3080-ti-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 119.69 t/s @32k ctx
119.69 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 66.45 t/s @131k ctx
66.45 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 94.91 t/s @65k ctx
94.91 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1666.51 t/s @65k ctx
1666.51 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 2521.32 t/s @32k ctx
2521.32 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 3440.06 t/s @16k ctx
3440.06 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 936.39 t/s @131k ctx
936.39 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-gptoss-20b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 28.2 t/s @65k ctx
28.2 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-14b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1914.75 t/s @16k ctx
1914.75 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 828.45 t/s @65k ctx
828.45 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-14b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 91.97 t/s @32k ctx
91.97 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-30b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, pp512 1483.93 t/s @32k ctx
1483.93 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-30b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, pp512 2205.58 t/s @16k ctx
2205.58 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-30b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 32B dense Q4_K XL GGUF, pp512 862.69 t/s @16k ctx
862.69 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 32B dense Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-32b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 29.01 t/s @131k ctx
29.01 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 48.98 t/s @65k ctx
48.98 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 71.75 t/s @32k ctx
71.75 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1111.11 t/s @65k ctx
1111.11 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1867.96 t/s @32k ctx
1867.96 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti 24GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 612.37 t/s @131k ctx
612.37 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 98.25 t/s @16k ctx
98.25 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-30b-a3b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti, llama.cpp CUDA, Qwen3 32B Q4_K XL GGUF, decode 35.42 t/s @16k ctx
35.42 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-32b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 126.94 t/s @16k ctx
126.94 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 3090 Ti, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, prompt-processing 3648.57 t/s @16k ctx
3648.57 tok/s prompt-processing (pp512, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-3090-ti via llama-cppbenchmarks/hwcorner-rtx-3090-ti-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 31.08 t/s @131k ctx
31.08 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 41.11 t/s @65k ctx
41.11 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 51.48 t/s @32k ctx
51.48 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1332.19 t/s @65k ctx
1332.19 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1964.9 t/s @32k ctx
1964.9 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 2552.88 t/s @16k ctx
2552.88 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 731.23 t/s @131k ctx
731.23 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF, decode 57.76 t/s @16k ctx
57.76 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-gptoss-20b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 17.86 t/s @32k ctx
17.86 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-14b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 541.36 t/s @32k ctx
541.36 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-14b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 917.56 t/s @16k ctx
917.56 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 22.36 t/s @16k ctx
22.36 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 13.01 t/s @65k ctx
13.01 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-8b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 25.53 t/s @32k ctx
25.53 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1480.81 t/s @16k ctx
1480.81 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 392.12 t/s @65k ctx
392.12 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-8b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 760.33 t/s @32k ctx
760.33 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 34.31 t/s @16k ctx
34.31 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4060-ti via llama-cppbenchmarks/hwcorner-rtx-4060-ti-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1356.4 t/s @16k ctx
1356.4 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2364.7 t/s @4k ctx
2364.7 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 45.1 t/s @4k ctx
45.1 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1355.83 t/s @16k ctx
1355.83 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 32.66 t/s @16k ctx
32.66 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1355.8 t/s @32k ctx
1355.8 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 2064.3 t/s @16k ctx
2064.3 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 4122.2 t/s @4k ctx
4122.2 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 38.1 t/s @32k ctx
38.1 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 75.4 t/s @4k ctx
75.4 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 38.07 t/s @32k ctx
38.07 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1116.68 t/s @32k ctx
1116.68 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 2064.25 t/s @16k ctx
2064.25 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 52.07 t/s @16k ctx
52.07 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070 via llama-cppbenchmarks/hwcorner-rtx-4070-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1578.1 t/s @16k ctx
1578.1 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2522.2 t/s @4k ctx
2522.2 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 45.5 t/s @4k ctx
45.5 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1578.12 t/s @16k ctx
1578.12 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 37.15 t/s @16k ctx
37.15 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1595.7 t/s @32k ctx
1595.7 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 2525.8 t/s @16k ctx
2525.8 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 4321.7 t/s @4k ctx
4321.7 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 42.2 t/s @32k ctx
42.2 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 75.4 t/s @4k ctx
75.4 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 42.17 t/s @32k ctx
42.17 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1595.71 t/s @32k ctx
1595.71 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 2525.76 t/s @16k ctx
2525.76 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 SUPER 12GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 56.21 t/s @16k ctx
56.21 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-super via llama-cppbenchmarks/hwcorner-rtx-4070-super-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1951.4 t/s @16k ctx
1951.4 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2866.6 t/s @4k ctx
2866.6 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 45.8 t/s @4k ctx
45.8 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 37.75 t/s @16k ctx
37.75 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1099.7 t/s @32k ctx
1099.7 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 2274.8 t/s @16k ctx
2274.8 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 4877.7 t/s @4k ctx
4877.7 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 42.1 t/s @32k ctx
42.1 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 75.8 t/s @4k ctx
75.8 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti 12GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 57.55 t/s @16k ctx
57.55 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti via llama-cppbenchmarks/hwcorner-rtx-4070-ti-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 1716.2 t/s @64k ctx
1716.2 tok/s per-request prompt processing (pp, ~65536-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 3020.0 t/s @32k ctx
3020.0 tok/s per-request prompt processing (pp, ~32768-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 4182.7 t/s @16k ctx
4182.7 tok/s per-request prompt processing (pp, ~16384-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 5628.1 t/s @4k ctx
5628.1 tok/s per-request prompt processing (pp, ~4096-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 993.5 t/s @128k ctx
993.5 tok/s per-request prompt processing (pp, ~131072-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-pp-128k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 128.9 t/s @4k ctx
128.9 tok/s per-request decode (batch 1, ~4096-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 57.5 t/s @128k ctx
57.5 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-128k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 78.7 t/s @64k ctx
78.7 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 98.7 t/s @32k ctx
98.7 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-mxfp4-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 57.47 t/s @131k ctx
57.47 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 78.67 t/s @65k ctx
78.67 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 98.66 t/s @32k ctx
98.66 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1716.2 t/s @65k ctx
1716.2 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 3020.03 t/s @32k ctx
3020.03 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 4182.67 t/s @16k ctx
4182.67 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 930.45 t/s @131k ctx
930.45 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF, decode 113.6 t/s @16k ctx
113.6 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-gptoss-20b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1236.9 t/s @32k ctx
1236.9 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2003.4 t/s @16k ctx
2003.4 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 3048.0 t/s @4k ctx
3048.0 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 37.7 t/s @32k ctx
37.7 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 58.1 t/s @4k ctx
58.1 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 37.72 t/s @32k ctx
37.72 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1236.9 t/s @32k ctx
1236.9 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 2003.44 t/s @16k ctx
2003.44 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 47.18 t/s @16k ctx
47.18 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1616.6 t/s @32k ctx
1616.6 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 3050.7 t/s @16k ctx
3050.7 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 5220.1 t/s @4k ctx
5220.1 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 829.4 t/s @64k ctx
829.4 tok/s per-request prompt processing (pp, ~65536-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-pp-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 36.6 t/s @64k ctx
36.6 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-tg-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 54.5 t/s @32k ctx
54.5 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 96.3 t/s @4k ctx
96.3 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 36.6 t/s @65k ctx
36.6 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 54.5 t/s @32k ctx
54.5 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1616.55 t/s @32k ctx
1616.55 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 3050.73 t/s @16k ctx
3050.73 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 829.44 t/s @65k ctx
829.44 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4070 Ti SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 72.2 t/s @16k ctx
72.2 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hwcorner-rtx-4070-ti-super-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 1145.0 t/s @128k ctx
1145.0 tok/s per-request prompt processing (pp, ~131072-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-pp-128k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 1961.5 t/s @64k ctx
1961.5 tok/s per-request prompt processing (pp, ~65536-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-pp-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 3328.1 t/s @32k ctx
3328.1 tok/s per-request prompt processing (pp, ~32768-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 4708.7 t/s @16k ctx
4708.7 tok/s per-request prompt processing (pp, ~16384-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 6364.0 t/s @4k ctx
6364.0 tok/s per-request prompt processing (pp, ~4096-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 107.2 t/s @32k ctx
107.2 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 139.1 t/s @4k ctx
139.1 tok/s per-request decode (batch 1, ~4096-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 60.0 t/s @128k ctx
60.0 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-tg-128k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 81.5 t/s @64k ctx
81.5 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-mxfp4-tg-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 107.19 t/s @32k ctx
107.19 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 60.01 t/s @131k ctx
60.01 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 81.45 t/s @65k ctx
81.45 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1145.01 t/s @131k ctx
1145.01 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1961.53 t/s @65k ctx
1961.53 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 3328.05 t/s @32k ctx
3328.05 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 4708.66 t/s @16k ctx
4708.66 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF, decode 122.96 t/s @16k ctx
122.96 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-gptoss-20b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1769.3 t/s @32k ctx
1769.3 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2526.7 t/s @16k ctx
2526.7 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 3745.0 t/s @4k ctx
3745.0 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 42.6 t/s @32k ctx
42.6 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 64.2 t/s @4k ctx
64.2 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 42.64 t/s @32k ctx
42.64 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1769.29 t/s @32k ctx
1769.29 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 2526.69 t/s @16k ctx
2526.69 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 52.76 t/s @16k ctx
52.76 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1501.5 t/s @64k ctx
1501.5 tok/s per-request prompt processing (pp, ~65536-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-pp-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 2537.1 t/s @32k ctx
2537.1 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 3858.1 t/s @16k ctx
3858.1 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 6137.0 t/s @4k ctx
6137.0 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 104.2 t/s @4k ctx
104.2 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 39.1 t/s @64k ctx
39.1 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-tg-64k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 59.5 t/s @32k ctx
59.5 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 39.12 t/s @65k ctx
39.12 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 59.47 t/s @32k ctx
59.47 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1501.49 t/s @65k ctx
1501.49 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 2537.08 t/s @32k ctx
2537.08 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 3858.11 t/s @16k ctx
3858.11 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080 SUPER 16GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 79.36 t/s @16k ctx
79.36 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080-super via llama-cppbenchmarks/hwcorner-rtx-4080-super-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 85.04 t/s @16k ctx
85.04 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwcorner-rtx-4080-qwen3-30b-a3b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 132.37 t/s @16k ctx
132.37 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwcorner-rtx-4080-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4080, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, prompt-processing 3868.43 t/s @16k ctx
3868.43 tok/s prompt-processing (pp512, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4080 via llama-cppbenchmarks/hwcorner-rtx-4080-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4090, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 109.57 t/s @16k ctx
109.57 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hwcorner-rtx-4090-qwen3-30b-a3b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4090, llama.cpp CUDA, Qwen3 32B Q4_K XL GGUF, decode 45.28 t/s @16k ctx
45.28 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hwcorner-rtx-4090-qwen3-32b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 160.82 t/s @16k ctx
160.82 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hwcorner-rtx-4090-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 4090, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, prompt-processing 5224.3 t/s @16k ctx
5224.3 tok/s prompt-processing (pp512, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-4090 via llama-cppbenchmarks/hwcorner-rtx-4090-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 1102.3 t/s @64k ctx
1102.3 tok/s per-request prompt processing (pp, ~65536-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-pp-64k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 1737.7 t/s @32k ctx
1737.7 tok/s per-request prompt processing (pp, ~32768-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 2753.3 t/s @16k ctx
2753.3 tok/s per-request prompt processing (pp, ~16384-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 3585.2 t/s @4k ctx
3585.2 tok/s per-request prompt processing (pp, ~4096-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, prompt processing (pp) 685.3 t/s @128k ctx
685.3 tok/s per-request prompt processing (pp, ~131072-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-pp-128k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 43.8 t/s @128k ctx
43.8 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-tg-128k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 58.1 t/s @64k ctx
58.1 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-tg-64k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 73.2 t/s @32k ctx
73.2 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B MXFP4, token generation (tg) 92.1 t/s @4k ctx
92.1 tok/s per-request decode (batch 1, ~4096-token context)gpt-oss 20B MXFP4 on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-mxfp4-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF, decode 82.42 t/s @16k ctx
82.42 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-gptoss-20b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1743.0 t/s @4k ctx
1743.0 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 621.0 t/s @32k ctx
621.0 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-14b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 942.6 t/s @16k ctx
942.6 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 25.9 t/s @32k ctx
25.9 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-14b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 41.1 t/s @4k ctx
41.1 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 32.91 t/s @16k ctx
32.91 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1447.9 t/s @16k ctx
1447.9 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 2965.1 t/s @4k ctx
2965.1 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 529.8 t/s @64k ctx
529.8 tok/s per-request prompt processing (pp, ~65536-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-pp-64k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 915.1 t/s @32k ctx
915.1 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 25.8 t/s @64k ctx
25.8 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-tg-64k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 38.9 t/s @32k ctx
38.9 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-tg-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 69.2 t/s @4k ctx
69.2 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5060 Ti 16GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 51.41 t/s @16k ctx
51.41 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-5060-ti via llama-cppbenchmarks/hwcorner-rtx-5060-ti-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 1315.2 t/s @16k ctx
1315.2 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-14b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, prompt processing (pp) 2144.2 t/s @4k ctx
2144.2 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-14b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 40.6 t/s @16k ctx
40.6 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-14b-q4k-tg-16k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K GGUF, token generation (tg) 54.2 t/s @4k ctx
54.2 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 14B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-14b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1315.22 t/s @16k ctx
1315.22 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 1600.8 t/s @16k ctx
1600.8 tok/s per-request prompt processing (pp, ~16384-token context)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4k-pp-16k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 3487.7 t/s @4k ctx
3487.7 tok/s per-request prompt processing (pp, ~4096-token context)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4k-pp-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, prompt processing (pp) 898.8 t/s @32k ctx
898.8 tok/s per-request prompt processing (pp, ~32768-token context)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4k-pp-32k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K GGUF, token generation (tg) 85.8 t/s @4k ctx
85.8 tok/s per-request decode (batch 1, ~4096-token context)Qwen3 8B Q4_K GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4k-tg-4k
verifiedmedium · 0.65
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1600.78 t/s @16k ctx
1600.78 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 12GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 898.83 t/s @32k ctx
898.83 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on local-nvidia-rtx-5070 via llama-cppbenchmarks/hwcorner-rtx-5070-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 Ti, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 111.11 t/s @16k ctx
111.11 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hwcorner-rtx-5070-ti-qwen3-30b-a3b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 Ti, llama.cpp CUDA, Qwen3 32B Q4_K XL GGUF, decode 37.29 t/s @16k ctx
37.29 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hwcorner-rtx-5070-ti-qwen3-32b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 127.72 t/s @16k ctx
127.72 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hwcorner-rtx-5070-ti-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 5070 Ti, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, prompt-processing 3360.59 t/s @16k ctx
3360.59 tok/s prompt-processing (pp512, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-5070-ti via llama-cppbenchmarks/hwcorner-rtx-5070-ti-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX 6000 Ada, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 104.88 t/s @16k ctx
104.88 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hwcorner-rtx-6000-ada-qwen3-30b-a3b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 6000 Ada, llama.cpp CUDA, Qwen3 32B Q4_K XL GGUF, decode 43.75 t/s @16k ctx
43.75 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hwcorner-rtx-6000-ada-qwen3-32b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 157.59 t/s @16k ctx
157.59 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hwcorner-rtx-6000-ada-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX 6000 Ada, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, prompt-processing 4270.99 t/s @16k ctx
4270.99 tok/s prompt-processing (pp512, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-6000-ada via llama-cppbenchmarks/hwcorner-rtx-6000-ada-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 33.42 t/s @131k ctx
33.42 tok/s per-request decode (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 53.25 t/s @65k ctx
53.25 tok/s per-request decode (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, decode 74.41 t/s @32k ctx
74.41 tok/s per-request decode (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1207.23 t/s @65k ctx
1207.23 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 1891.23 t/s @32k ctx
1891.23 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 2649.16 t/s @16k ctx
2649.16 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K XL GGUF, pp512 702.79 t/s @131k ctx
702.79 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, gpt-oss 20B Q4_K_XL GGUF, decode 91.19 t/s @16k ctx
91.19 tok/s per-request decode (batch 1, ~16384-token context)gpt-oss 20B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-gptoss-20b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Llama 3.x 70B Q4_K XL GGUF, pp512 238.73 t/s @16k ctx
238.73 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Llama 3.x 70B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-llama-70b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 12.1 t/s @131k ctx
12.1 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 20.51 t/s @65k ctx
20.51 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, decode 30.75 t/s @32k ctx
30.75 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1063.35 t/s @32k ctx
1063.35 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 1529.0 t/s @16k ctx
1529.0 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 382.35 t/s @131k ctx
382.35 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K XL GGUF, pp512 668.87 t/s @65k ctx
668.87 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 14B Q4_K_XL GGUF, decode 40.69 t/s @16k ctx
40.69 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 14B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-14b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 14.77 t/s @131k ctx
14.77 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 27.57 t/s @65k ctx
27.57 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, decode 48.13 t/s @32k ctx
48.13 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, pp512 1275.59 t/s @32k ctx
1275.59 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, pp512 1901.13 t/s @16k ctx
1901.13 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, pp512 429.18 t/s @131k ctx
429.18 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B (MoE) Q4_K XL GGUF, pp512 764.16 t/s @65k ctx
764.16 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 30B-A3B (MoE) Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 30B-A3B Q4_K_XL GGUF, decode 75.35 t/s @16k ctx
75.35 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 30B-A3B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-30b-a3b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 32B dense Q4_K XL GGUF, decode 13.52 t/s @32k ctx
13.52 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 32B dense Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-32b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 32B dense Q4_K XL GGUF, decode 8.68 t/s @65k ctx
8.68 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 32B dense Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-32b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 32B dense Q4_K XL GGUF, pp512 318.95 t/s @65k ctx
318.95 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 32B dense Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-32b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 32B dense Q4_K XL GGUF, pp512 502.73 t/s @32k ctx
502.73 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 32B dense Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-32b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 32B dense Q4_K XL GGUF, pp512 704.82 t/s @16k ctx
704.82 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 32B dense Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-32b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 32B Q4_K_XL GGUF, decode 18.32 t/s @16k ctx
18.32 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 32B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-32b-q4kxl-decode-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 15.95 t/s @131k ctx
15.95 tok/s per-request decode (batch 1, ~131072-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-decode-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 27.93 t/s @65k ctx
27.93 tok/s per-request decode (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-decode-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, decode 44.55 t/s @32k ctx
44.55 tok/s per-request decode (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-decode-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 1593.26 t/s @32k ctx
1593.26 tok/s per-request prompt processing pp512 (batch 1, ~32768-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-pp512-32k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 2427.98 t/s @16k ctx
2427.98 tok/s per-request prompt processing pp512 (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-pp512-16k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 527.88 t/s @131k ctx
527.88 tok/s per-request prompt processing pp512 (batch 1, ~131072-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-pp512-131k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K XL GGUF, pp512 950.96 t/s @65k ctx
950.96 tok/s per-request prompt processing pp512 (batch 1, ~65536-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-pp512-65k
verifiedmedium · 0.70
Hardware Corner: RTX A6000 48GB, llama.cpp CUDA, Qwen3 8B Q4_K_XL GGUF, decode 64.26 t/s @16k ctx
64.26 tok/s per-request decode (batch 1, ~16384-token context)Qwen3 8B Q4_K XL GGUF on nvidia-rtx-a6000 via llama-cppbenchmarks/hwcorner-rtx-a6000-qwen3-8b-q4kxl-decode-16k
verifiedmedium · 0.70
HF optimum-neuron on AWS inf2.48xlarge: Llama2 13B (latency config, bs1), 145 tok/s at 256 new tokens
145.0 tok/s (single request; Neuron-compiled)meta-llama/Llama-2-13b-hf on aws-inferentia2 via hf-optimumbenchmarks/hf-optimum-neuron-inf2-llama2-13b-l-1x-256new-tps
verifiedmedium · 0.80
HF optimum-neuron on AWS inf2.48xlarge: Llama2 13B (throughput config, bs4), 504 tok/s at 256 new tokens
504.0 tok/s (aggregate across batch 4; Neuron-compiled)meta-llama/Llama-2-13b-hf on aws-inferentia2 via hf-optimumbenchmarks/hf-optimum-neuron-inf2-llama2-13b-t-4x-256new-tps
verifiedmedium · 0.80
HF optimum-neuron on AWS inf2.48xlarge: Llama2 7B (latency config, bs1), 227 tok/s at 256 new tokens
227.0 tok/s (single request; Neuron-compiled)meta-llama/Llama-2-7b-hf on aws-inferentia2 via hf-optimumbenchmarks/hf-optimum-neuron-inf2-llama2-7b-l-1x-256new-tps
verifiedmedium · 0.80
HF optimum-neuron on AWS inf2.48xlarge: Llama2 7B (throughput config, bs4), 750 tok/s at 256 new tokens
750.0 tok/s (aggregate across batch 4; Neuron-compiled)meta-llama/Llama-2-7b-hf on aws-inferentia2 via hf-optimumbenchmarks/hf-optimum-neuron-inf2-llama2-7b-t-4x-256new-tps
verifiedmedium · 0.80
HF optimum-neuron on AWS inf2.xlarge: Llama2 7B (budget config, bs1), 32 tok/s at 256 new tokens
32.0 tok/s (single request; Neuron-compiled)meta-llama/Llama-2-7b-hf on aws-inferentia2 via hf-optimumbenchmarks/hf-optimum-neuron-inf2-llama2-7b-b-1x-256new-tps
verifiedmedium · 0.80
Huawei Atlas 300I Duo (96GB) Llama-3 decode - forum report
150.0 ~tok/s, multi-chip 96GB Duo - NOT a single-card figureLlama-3 (70B-class) on huawei-ascend-310p, huawei-atlas-300i-duo via niche-vllm-ascendbenchmarks/comm-ascend-300i-duo-llama3-70b-decode
verifiedmedium · 0.75
Hydra acceptance length and speedup on Vicuna-13B (temperature=0)
3.4 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-hydra-acceptance-vicuna13b-t0
verifiedhigh · 0.90
INT4 per-channel affine WikiText-2 perplexity delta vs fp16, Llama2-7B (RTN)
0.79 perplexity_delta (WikiText-2-test ppl points, quantized minus FP16)Llama2-7B on ? via mxptq-research-harnessbenchmarks/int4-perchannel-wikitext2-ppl-delta-llama2-7b-rtn
verifiedhigh · 0.90
Intel Arc Pro B50 x4, MLPerf v6.0 Offline: Llama-3.1-8B INT8 aggregate tok/s
1721.84 output tok/s AGGREGATE across 4x Arc Pro B50 (Offline; ~430 tok/s per card)Llama-3.1-8B (INT8) on intel-arc-pro-b50 via vllmbenchmarks/b50-4x-mlperf-v6-0-llama31-8b-offline-tokens
verifiedhigh · 0.85
Intel Arc Pro B50 x4, MLPerf v6.0 Server: Llama-3.1-8B INT8 aggregate tok/s
1187.13 output tok/s AGGREGATE across 4x Arc Pro B50 (Server scenario; latency-constrained)Llama-3.1-8B (INT8) on intel-arc-pro-b50 via vllmbenchmarks/b50-4x-mlperf-v6-0-llama31-8b-server-tokens
verifiedhigh · 0.85
Intel Arc Pro B60 24GB x4, MLPerf v5.1 Offline: Llama-2-70B aggregate tok/s
3009.38 output tok/s AGGREGATE across 4x Arc Pro B60 (Offline; ~752 tok/s per card)Llama-2-70B (FP8) on intel-arc-pro-b60 via vllmbenchmarks/prob60dual-4x-mlperf-v5-1-llama2-70b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB x4, MLPerf v5.1 Offline: Llama-3.1-8B aggregate tok/s
6472.37 output tok/s AGGREGATE across 4x Arc Pro B60 (Offline; ~1618 tok/s per card)Llama-3.1-8B (FP8) on intel-arc-pro-b60 via vllmbenchmarks/prob60dual-4x-mlperf-v5-1-llama31-8b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB x4, MLPerf v5.1 Server: Llama-2-70B aggregate tok/s
2110.05 output tok/s AGGREGATE across 4x Arc Pro B60 (Server; ~528 tok/s per card)Llama-2-70B (FP8) on intel-arc-pro-b60 via vllmbenchmarks/prob60dual-4x-mlperf-v5-1-llama2-70b-server-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB x4, MLPerf v6.0 Offline: gpt-oss-120b aggregate tok/s
841.038 output tok/s AGGREGATE across 4x Arc Pro B60 (Offline; ~210 tok/s per card)gpt-oss-120b (FP8) on intel-arc-pro-b60 via vllmbenchmarks/b60-4x-mlperf-v6-0-gptoss-120b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB x4, MLPerf v6.0 Offline: Llama-2-70B aggregate tok/s
1697.66 output tok/s AGGREGATE across 4x Arc Pro B60 (Offline; ~424 tok/s per card)Llama-2-70B (FP8) on intel-arc-pro-b60 via vllmbenchmarks/b60-4x-mlperf-v6-0-llama2-70b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB x4, MLPerf v6.0 Offline: Llama-3.1-8B aggregate tok/s
3347.43 output tok/s AGGREGATE across 4x Arc Pro B60 (Offline; ~837 tok/s per card)Llama-3.1-8B (FP8) on intel-arc-pro-b60 via vllmbenchmarks/b60-4x-mlperf-v6-0-llama31-8b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB x4, MLPerf v6.0 Server: Llama-2-70B aggregate tok/s
1106.26 output tok/s AGGREGATE across 4x Arc Pro B60 (Server; ~277 tok/s per card)Llama-2-70B (FP8) on intel-arc-pro-b60 via vllmbenchmarks/b60-4x-mlperf-v6-0-llama2-70b-server-tokens
verifiedmedium · 0.80
Intel Arc Pro B60 24GB, llama.cpp b8175 SYCL+Vulkan: Qwen3.5-9B Q4_K_XL decode tok/s (L1 measured)
23.0 decode tok/s (tg128, SYCL backend; Vulkan 17.8)Qwen3.5-9B Q4_K_XL (GGUF) on intel-arc-pro-b60 via llama-cppbenchmarks/comm-arc-pro-b60-sycl-qwen35-9b-decode
verifiedmedium · 0.80
Intel Arc Pro B70 32GB x4, MLPerf v6.0 Offline: gpt-oss-120b aggregate tok/s
1536.9 output tok/s AGGREGATE across 4x Arc Pro B70 (Offline; ~384 tok/s per card)gpt-oss-120b (FP8) on intel-arc-pro-b70 via vllmbenchmarks/b70-4x-mlperf-v6-0-gptoss-120b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B70 32GB x4, MLPerf v6.0 Offline: Llama-2-70B aggregate tok/s
2459.18 output tok/s AGGREGATE across 4x Arc Pro B70 (Offline; ~615 tok/s per card)Llama-2-70B (FP8) on intel-arc-pro-b70 via vllmbenchmarks/b70-4x-mlperf-v6-0-llama2-70b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B70 32GB x4, MLPerf v6.0 Offline: Llama-3.1-8B aggregate tok/s
4617.4 output tok/s AGGREGATE across 4x Arc Pro B70 (Offline; ~1154 tok/s per card)Llama-3.1-8B (FP8) on intel-arc-pro-b70 via vllmbenchmarks/b70-4x-mlperf-v6-0-llama31-8b-offline-tokens
verifiedmedium · 0.80
Intel Arc Pro B70 32GB x4, MLPerf v6.0 Server: Llama-2-70B aggregate tok/s
1698.57 output tok/s AGGREGATE across 4x Arc Pro B70 (Server; ~425 tok/s per card)Llama-2-70B (FP8) on intel-arc-pro-b70 via vllmbenchmarks/b70-4x-mlperf-v6-0-llama2-70b-server-tokens
verifiedmedium · 0.80
Intel Arc Pro B70 32GB, vLLM: Qwen3.5-27B single-request decode tok/s + 4-card aggregate (wendell measured)
13.4 tok/s single-request floor (dynamic FP8); 4x B70 TP=4 50-req ~370 t/s avg / 550 peakQwen3.5-27B (dynamic FP8) on intel-arc-pro-b70 via vllmbenchmarks/comm-arc-pro-b70-vllm-qwen35-27b-decode
verifiedmedium · 0.80
Intel Arc Pro B70, llama.cpp Vulkan (b863507): Llama-2-7B Q4_0 tg128 t/s
112.02 tok/s per-request tg128 (no FA); ±1.08Llama-2-7B Q4_0 GGUF on intel-arc-pro-b70 via llama-cppbenchmarks/b70-vulkan-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
Intel Arc Pro B70, MLPerf Inference v6.1 Offline: Llama 2 70B on 4x Arc Pro B70 (GDDR6), aggregate output throughput
2464.98 output tokens/s (aggregate; harness labels it '(inferred)')Llama 2 70B (MLPerf v6.1 submission) on intel-arc-pro-b70 via vllmbenchmarks/arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens
verifiedhigh · 0.90
Intel Gaudi 2, MLPerf Inference v4.0 Offline: Llama 2 70B fp8 on 8xGaudi 2, aggregate output throughput
8034.85 output tokens/s (aggregate, MLPerf 'Tokens per second')Llama 2 70B (fp8 weights, vLLM-fork) on intel-gaudi2 via vllmbenchmarks/gaudi2-mlperf-v4-0-llama2-70b-offline-tokens
verifiedhigh · 0.95
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 2HPU bs175 2048in/2048out aggregate tok/s
3115 output tok/s AGGREGATE across 2x Gaudi 3 (batch 175; ~1558 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-2hpu-llama31-70b-fp8-2048-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 2HPU bs2048 128in/128out aggregate tok/s
5303 output tok/s AGGREGATE across 2x Gaudi 3 (batch 2048; ~2652 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-2hpu-llama31-70b-fp8-128-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 2HPU bs223 2048in/128out aggregate tok/s
650 output tok/s AGGREGATE across 2x Gaudi 3 (batch 223; ~325 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-2hpu-llama31-70b-fp8-2048-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 2HPU bs450 128in/2048out aggregate tok/s
6514 output tok/s AGGREGATE across 2x Gaudi 3 (batch 450; ~3257 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-2hpu-llama31-70b-fp8-128-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 8HPU bs4000 128in/128out aggregate tok/s
16929 output tok/s AGGREGATE across 8x Gaudi 3 (batch 4000; ~2116 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama31-70b-fp8-128-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 8HPU bs512 2048in/128out aggregate tok/s
2183 output tok/s AGGREGATE across 8x Gaudi 3 (batch 512; ~273 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama31-70b-fp8-2048-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 8HPU bs600 2048in/2048out aggregate tok/s
10767 output tok/s AGGREGATE across 8x Gaudi 3 (batch 600; ~1346 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama31-70b-fp8-2048-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-70B-Instruct FP8 8HPU bs768 128in/2048out aggregate tok/s
21044 output tok/s AGGREGATE across 8x Gaudi 3 (batch 768; ~2630 tok/s per HPU)Llama-3.1-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama31-70b-fp8-128-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-8B-Instruct FP8 1HPU bs1536 128in/128out aggregate tok/s
23922 output tok/s AGGREGATE across 1x Gaudi 3 (batch 1536; ~23922 tok/s per HPU)Llama-3.1-8B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-1hpu-llama31-8b-fp8-128-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-8B-Instruct FP8 1HPU bs256 2048in/128out aggregate tok/s
2751 output tok/s AGGREGATE across 1x Gaudi 3 (batch 256; ~2751 tok/s per HPU)Llama-3.1-8B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-1hpu-llama31-8b-fp8-2048-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-8B-Instruct FP8 1HPU bs256 2048in/2048out aggregate tok/s
9040 output tok/s AGGREGATE across 1x Gaudi 3 (batch 256; ~9040 tok/s per HPU)Llama-3.1-8B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-1hpu-llama31-8b-fp8-2048-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.1-8B-Instruct FP8 1HPU bs768 128in/2048out aggregate tok/s
19242 output tok/s AGGREGATE across 1x Gaudi 3 (batch 768; ~19242 tok/s per HPU)Llama-3.1-8B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-1hpu-llama31-8b-fp8-128-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.3-70B-Instruct FP8 8HPU bs4000 128in/128out aggregate tok/s
16883 output tok/s AGGREGATE across 8x Gaudi 3 (batch 4000; ~2110 tok/s per HPU)Llama-3.3-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama33-70b-fp8-128-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.3-70B-Instruct FP8 8HPU bs512 2048in/128out aggregate tok/s
2129 output tok/s AGGREGATE across 8x Gaudi 3 (batch 512; ~266 tok/s per HPU)Llama-3.3-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama33-70b-fp8-2048-128
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.3-70B-Instruct FP8 8HPU bs600 2048in/2048out aggregate tok/s
10747 output tok/s AGGREGATE across 8x Gaudi 3 (batch 600; ~1343 tok/s per HPU)Llama-3.3-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama33-70b-fp8-2048-2048
verifiedmedium · 0.80
Intel Gaudi 3, Optimum-Habana (sw 1.24): Llama-3.3-70B-Instruct FP8 8HPU bs768 128in/2048out aggregate tok/s
20980 output tok/s AGGREGATE across 8x Gaudi 3 (batch 768; ~2622 tok/s per HPU)Llama-3.3-70B-Instruct on intel-gaudi3 via vllm-gaudibenchmarks/gaudi3-8hpu-llama33-70b-fp8-128-2048
verifiedmedium · 0.80
Intel LLM-Scaler 1.2 on 4x Arc Pro B60 (TP=4): Qwen3-VL-30B decision-support 1K->2K TPOT (c=16)
44 ms/tokenQwen/Qwen3-VL-30B-A3B-Instruct on intel-arc-pro-b60 via vllmbenchmarks/embeddedllm-b60x4-llmscaler-qwen3vl30b-decision-tpot
verifiedmedium · 0.75
Intel perf table: ResNet-50 v1.5 INT8 on Flex 170, OpenVINO, 3297 img/s (b1)
3297.0 inferences/sResNet-50 v1.5 (ImageNet2013) on xpu-intel-dc-gpu-flex-170 via openvino-genaibenchmarks/intel-perf-flex170-resnet50-int8-b1
verifiedmedium · 0.75
Intel perf table: ResNet-50 v1.5 INT8 on Flex 170, OpenVINO, 9499 img/s (b256)
9499.0 inferences/sResNet-50 v1.5 (ImageNet2012) on xpu-intel-dc-gpu-flex-170 via openvino-genaibenchmarks/intel-perf-flex170-resnet50-int8-b256
verifiedmedium · 0.75
Intel perf table: Stable Diffusion FP16 on Flex 170, Intel TF 2.13, 6.45 its/s
6.45 iterations/sStable Diffusion (text prompts) on xpu-intel-dc-gpu-flex-170 via openvino-genaibenchmarks/intel-perf-flex170-stablediff-fp16-b1
verifiedmedium · 0.75
Intel perf table: YOLOv4 INT8 on Flex 170, OpenVINO, 1209 img/s (b256)
1209.0 inferences/sYOLOv4 (COCO2017_detection) on xpu-intel-dc-gpu-flex-170 via openvino-genaibenchmarks/intel-perf-flex170-yolov4-int8-b256
verifiedmedium · 0.75
Intel perf table: YOLOv4 INT8 on Flex 170, OpenVINO, 595 img/s (b1)
595.0 inferences/sYOLOv4 (COCO2017_detection) on xpu-intel-dc-gpu-flex-170 via openvino-genaibenchmarks/intel-perf-flex170-yolov4-int8-b1
verifiedmedium · 0.75
Jetson AGX Thor llama.cpp: Qwen2.5-1.5B Q4_K_M generate 128 (Multimodalflow)
106.8 tok/s (llama-bench tg, per-request)Qwen2.5-1.5B-Instruct Q4_K_M on nvidia-jetson-agx-thor via llama-cppbenchmarks/thor-llamacpp-qwen25-15b-q4km-tg128
verifiedlow · 0.60
Jetson AGX Thor llama.cpp: Qwen2.5-1.5B Q4_K_M prefill 512 (Multimodalflow)
4298.3 tok/s (llama-bench pp, per-request)Qwen2.5-1.5B-Instruct Q4_K_M on nvidia-jetson-agx-thor via llama-cppbenchmarks/thor-llamacpp-qwen25-15b-q4km-pp512
verifiedlow · 0.60
Jetson AGX Thor SGLang: Qwen3.6-35B-A3B FP8 decode (Multimodalflow)
14.6 tok/s (generation, per-request)Qwen3.6-35B-A3B-FP8 on nvidia-jetson-agx-thor via sglangbenchmarks/thor-sglang-qwen36-35b-fp8-decode
verifiedlow · 0.55
Jetson AGX Thor SGLang: Qwen3.6-35B-A3B FP8 TTFT warm (Multimodalflow)
101.0 ms (time to first token, warm, per-request)Qwen3.6-35B-A3B-FP8 on nvidia-jetson-agx-thor via sglangbenchmarks/thor-sglang-qwen36-35b-fp8-ttft
verifiedlow · 0.50
Kairos: P95 TTFT breakdown in disaggregated serving — prefill execution is 2-23%, queue+KV transfer is 77-98%
6278.8 ms (P95 TTFT, LPLD workload, 2P2D disaggregated, A100 80GB, cold cache)DeepSeek-V2-Lite (MoE) on nvidia-a100-80gb-sxm4 via vllmbenchmarks/kairos-ttft-breakdown-prefill-vs-queue-vs-kv-transfer
verifiedhigh · 0.85
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 6.4.4 32k pp2048
691.94 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 6.4.4 32k tg32
21.39 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 6.4.4 64k pp2048
470.28 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 6.4.4 64k tg32
20.66 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7 nightlies 32k pp2048
638.41 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7 nightlies 32k tg32
21.58 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7 nightlies 64k pp2048
421.88 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7 nightlies 64k tg32
20.88 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7.2.3 32k pp2048
667.44 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7.2.3 32k tg32
21.59 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7.2.3 64k pp2048
446.88 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 ROCm 7.2.3 64k tg32
20.79 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan AMDVLK 32k pp2048
60.07 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan AMDVLK 32k tg32
16.44 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan AMDVLK 64k pp2048
39.84 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan AMDVLK 64k tg32
15.1 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan RADV 32k pp2048
428.06 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan RADV 32k tg32
16.99 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan RADV 64k pp2048
331.82 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-BF16 Vulkan RADV 64k tg32
16.12 tok/sgemma-4-26B-A4B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-bf16-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 6.4.4 32k pp2048
761.5 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 6.4.4 32k tg32
37.29 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 6.4.4 64k pp2048
499.49 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 6.4.4 64k tg32
35.18 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 6.4.4 pp512
1334.91 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm644-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 6.4.4 tg128
47.74 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm644-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7 nightlies 32k pp2048
674.07 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7 nightlies 32k tg32
37.68 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7 nightlies 64k pp2048
437.2 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7 nightlies 64k tg32
35.52 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7.2.3 32k pp2048
725.46 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7.2.3 32k tg32
37.84 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7.2.3 64k pp2048
471.3 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL ROCm 7.2.3 64k tg32
35.61 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan AMDVLK 32k pp2048
101.11 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan AMDVLK 32k tg32
36.34 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan AMDVLK 64k pp2048
54.46 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan AMDVLK 64k tg32
30.34 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan AMDVLK pp512
765.18 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan AMDVLK tg128
52.04 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan RADV 32k pp2048
682.46 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan RADV 32k tg32
42.4 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan RADV 64k pp2048
469.95 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan RADV 64k tg32
37.46 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan RADV pp512
1324.66 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q4_K_XL Vulkan RADV tg128
54.67 tok/sgemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q4-k-xl-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 6.4.4 32k pp2048
769.66 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 6.4.4 32k tg32
33.66 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 6.4.4 64k pp2048
504.88 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 6.4.4 64k tg32
32.09 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7 nightlies 32k pp2048
692.34 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7 nightlies 32k tg32
34.0 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7 nightlies 64k pp2048
447.95 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7 nightlies 64k tg32
32.16 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7.2.3 32k pp2048
725.86 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7.2.3 32k tg32
34.01 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7.2.3 64k pp2048
471.81 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL ROCm 7.2.3 64k tg32
32.14 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan AMDVLK 32k pp2048
99.13 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan AMDVLK 32k tg32
32.25 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan AMDVLK 64k pp2048
53.86 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan AMDVLK 64k tg32
27.52 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan RADV 32k pp2048
672.93 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan RADV 32k tg32
36.81 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan RADV 64k pp2048
465.71 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-26B-A4B-it-UD-Q8_K_XL Vulkan RADV 64k tg32
32.9 tok/sgemma-4-26B-A4B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-26b-a4b-it-ud-q8-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 6.4.4 32k pp2048
172.49 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 6.4.4 32k tg32
3.21 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 6.4.4 64k pp2048
114.16 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 6.4.4 64k tg32
3.12 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7 nightlies 32k pp2048
158.77 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7 nightlies 32k tg32
3.25 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7 nightlies 64k pp2048
102.83 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7 nightlies 64k tg32
3.14 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7.2.3 32k pp2048
167.2 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7.2.3 32k tg32
3.23 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7.2.3 64k pp2048
108.99 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 ROCm 7.2.3 64k tg32
3.13 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 Vulkan RADV 32k pp2048
92.4 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 Vulkan RADV 64k pp2048
67.88 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-BF16 Vulkan RADV 64k tg32
3.08 tok/sgemma-4-31B-it-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-bf16-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 6.4.4 32k pp2048
154.73 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 6.4.4 32k tg32
7.95 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 6.4.4 64k pp2048
106.22 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 6.4.4 64k tg32
7.38 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7 nightlies 32k pp2048
142.53 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7 nightlies 32k tg32
8.21 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7 nightlies 64k pp2048
95.32 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7 nightlies 64k tg32
7.58 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7.2.3 32k pp2048
150.43 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7.2.3 32k tg32
8.13 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7.2.3 64k pp2048
101.59 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7.2.3 64k tg32
7.54 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7.2.3 pp512
322.69 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL ROCm 7.2.3 tg128
10.55 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL Vulkan RADV 32k pp2048
146.08 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL Vulkan RADV 32k tg32
9.09 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL Vulkan RADV 64k pp2048
86.61 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL Vulkan RADV 64k tg32
7.88 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL Vulkan RADV pp512
301.53 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q4_K_XL Vulkan RADV tg128
11.45 tok/sgemma-4-31B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q4-k-xl-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 6.4.4 32k pp2048
155.9 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 6.4.4 32k tg32
5.27 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 6.4.4 64k pp2048
106.02 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7 nightlies 32k pp2048
143.8 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7 nightlies 32k tg32
5.3 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7 nightlies 64k pp2048
96.12 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7.2.3 32k pp2048
150.02 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7.2.3 32k tg32
5.26 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7.2.3 64k pp2048
100.93 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL ROCm 7.2.3 64k tg32
5.01 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL Vulkan RADV 32k pp2048
141.06 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL Vulkan RADV 32k tg32
5.52 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL Vulkan RADV 64k pp2048
88.45 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gemma-4-31B-it-UD-Q8_K_XL Vulkan RADV 64k tg32
5.05 tok/sgemma-4-31B-it-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gemma-4-31b-it-ud-q8-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 6.4.4 32k pp2048
97.18 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 6.4.4 32k tg32
16.38 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 6.4.4 64k pp2048
51.9 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 6.4.4 64k tg32
12.82 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7 nightlies 32k pp2048
90.71 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7 nightlies 32k tg32
16.55 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7 nightlies 64k pp2048
47.61 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7 nightlies 64k tg32
12.92 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7.2.3 32k pp2048
91.33 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7.2.3 32k tg32
16.56 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7.2.3 64k pp2048
48.38 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 ROCm 7.2.3 64k tg32
12.9 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 Vulkan AMDVLK 32k pp2048
10.42 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 Vulkan AMDVLK 32k tg32
5.4 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 Vulkan RADV 32k pp2048
109.96 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 Vulkan RADV 32k tg32
7.96 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 Vulkan RADV 64k pp2048
65.21 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-BF16 Vulkan RADV 64k tg32
6.9 tok/sGLM-4.7-Flash-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-bf16-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 6.4.4 32k pp2048
102.84 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 6.4.4 32k tg32
21.41 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 6.4.4 64k pp2048
53.18 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 6.4.4 64k tg32
15.67 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7 nightlies 32k pp2048
94.94 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7 nightlies 32k tg32
22.0 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7 nightlies 64k pp2048
48.9 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7 nightlies 64k tg32
15.99 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7.2.3 32k pp2048
95.82 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7.2.3 32k tg32
21.96 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7.2.3 64k pp2048
49.39 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7.2.3 64k tg32
15.86 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7.2.3 pp512
913.36 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-47-flash-ud-q8-k-xl-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL ROCm 7.2.3 tg128
35.5 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-47-flash-ud-q8-k-xl-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL Vulkan AMDVLK pp512
543.44 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-47-flash-ud-q8-k-xl-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL Vulkan AMDVLK tg128
39.66 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-47-flash-ud-q8-k-xl-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL Vulkan RADV 32k pp2048
136.78 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL Vulkan RADV 32k tg32
22.55 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL Vulkan RADV 64k pp2048
74.06 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S GLM-4.7-Flash-UD-Q8_K_XL Vulkan RADV 64k tg32
15.7 tok/sGLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-glm-4-7-flash-ud-q8-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 6.4.4 32k pp2048
603.27 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 6.4.4 32k tg32
40.02 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 6.4.4 64k pp2048
379.11 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 6.4.4 64k tg32
32.99 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7 nightlies 32k pp2048
653.09 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7 nightlies 64k pp2048
412.83 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7 nightlies 64k tg32
28.15 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7.2.3 32k pp2048
589.25 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7.2.3 32k tg32
36.32 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7.2.3 64k pp2048
371.47 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 ROCm 7.2.3 64k tg32
26.11 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan AMDVLK 32k pp2048
234.02 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan AMDVLK 32k tg32
35.87 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan AMDVLK 64k pp2048
143.42 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan AMDVLK 64k tg32
27.87 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan RADV 32k pp2048
307.5 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan RADV 32k tg32
43.01 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan RADV 64k pp2048
145.92 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-120b-mxfp4 Vulkan RADV 64k tg32
34.97 tok/sgpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-120b-mxfp4-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 6.4.4 32k pp2048
1027.43 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 6.4.4 32k tg32
57.29 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 6.4.4 64k pp2048
613.02 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 6.4.4 64k tg32
47.67 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7 nightlies 32k pp2048
1124.14 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7 nightlies 32k tg32
52.23 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7 nightlies 64k pp2048
679.48 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7 nightlies 64k tg32
40.79 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7.2.3 32k pp2048
1003.11 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7.2.3 32k tg32
52.17 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7.2.3 64k pp2048
601.63 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7.2.3 64k tg32
40.68 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7.2.3 pp512
1786.17 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 ROCm 7.2.3 tg128
73.05 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan AMDVLK 32k pp2048
388.64 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan AMDVLK 32k tg32
51.77 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan AMDVLK 64k pp2048
228.5 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan AMDVLK 64k tg32
40.52 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan AMDVLK pp512
1376.47 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan AMDVLK tg128
74.15 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan RADV 32k pp2048
555.73 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan RADV 32k tg32
61.12 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan RADV 64k pp2048
237.74 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan RADV 64k tg32
50.13 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan RADV pp512
1692.49 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S gpt-oss-20b-mxfp4 Vulkan RADV tg128
79.78 tok/sgpt-oss-20b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-gpt-oss-20b-mxfp4-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 6.4.4 32k pp2048
187.91 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 6.4.4 32k tg32
6.94 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 6.4.4 64k pp2048
82.46 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 6.4.4 64k tg32
3.76 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7 nightlies 32k pp2048
204.17 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7 nightlies 32k tg32
7.0 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7 nightlies 64k pp2048
94.78 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7 nightlies 64k tg32
3.79 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7.2.3 32k pp2048
159.95 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7.2.3 64k pp2048
71.44 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7.2.3 64k tg32
3.0 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7.2.3 pp512
1545.36 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7bq4-0-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 ROCm 7.2.3 tg128
50.59 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7bq4-0-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan AMDVLK 32k tg32
9.42 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan AMDVLK 64k pp2048
102.05 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan AMDVLK 64k tg32
5.12 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan AMDVLK pp512
347.25 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7bq4-0-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan AMDVLK tg128
55.69 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7bq4-0-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan RADV 32k pp2048
216.79 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan RADV 32k tg32
8.47 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan RADV 64k pp2048
79.73 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan RADV 64k tg32
4.58 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7b-q4-0-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan RADV pp512
1337.7 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7bq4-0-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S llama-2-7b.Q4_0 Vulkan RADV tg128
55.73 tok/sllama-2-7b.Q4_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-llama-2-7bq4-0-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 6.4.4 32k pp2048
119.83 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 6.4.4 32k tg32
6.73 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 7 nightlies 32k pp2048
156.27 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 7 nightlies 32k tg32
6.45 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 7.2.3 32k pp2048
142.92 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 7.2.3 32k tg32
6.19 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 7.2.3 pp512
243.25 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m27-ud-q3-k-s-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S ROCm 7.2.3 tg128
23.11 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m27-ud-q3-k-s-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan AMDVLK 32k pp2048
30.23 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan AMDVLK 32k tg32
10.74 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan AMDVLK 64k pp2048
13.9 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan AMDVLK 64k tg32
6.67 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan RADV 32k pp2048
33.86 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan RADV 32k tg32
14.65 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan RADV 64k pp2048
13.79 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan RADV 64k tg32
8.84 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m2-7-ud-q3-k-s-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan RADV pp512
242.43 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m27-ud-q3-k-s-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S MiniMax-M2.7-UD-Q3_K_S Vulkan RADV tg128
31.08 tok/sMiniMax-M2.7-UD-Q3_K_S on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-minimax-m27-ud-q3-k-s-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 6.4.4 32k pp2048
318.28 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 6.4.4 32k tg32
7.11 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 6.4.4 64k pp2048
188.86 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7 nightlies 32k pp2048
395.07 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7 nightlies 64k pp2048
239.54 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7.2.3 32k pp2048
349.46 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7.2.3 32k tg32
7.12 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7.2.3 64k pp2048
209.94 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7.2.3 64k tg32
6.14 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7.2.3 pp512
801.06 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 ROCm 7.2.3 tg128
8.5 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 Vulkan RADV 32k pp2048
79.54 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 Vulkan RADV 32k tg32
6.62 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 Vulkan RADV 64k pp2048
35.49 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 Vulkan RADV 64k tg32
5.74 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 Vulkan RADV pp512
255.05 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Ministral-3-14B-Instruct-2512-BF16 Vulkan RADV tg128
7.83 tok/sMinistral-3-14B-Instruct-2512-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-ministral-3-14b-instruct-2512-bf16-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 6.4.4 32k pp2048
379.64 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 6.4.4 32k tg32
14.15 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 6.4.4 64k pp2048
333.18 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 6.4.4 64k tg32
13.88 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7 nightlies 32k pp2048
353.69 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7 nightlies 32k tg32
14.32 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7 nightlies 64k pp2048
326.04 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7 nightlies 64k tg32
14.05 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7.2.3 32k pp2048
382.38 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7.2.3 32k tg32
14.4 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7.2.3 64k pp2048
348.1 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7.2.3 64k tg32
13.96 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7.2.3 pp512
275.89 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL ROCm 7.2.3 tg128
14.85 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan AMDVLK 32k pp2048
120.39 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan AMDVLK 32k tg32
13.67 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan AMDVLK 64k pp2048
96.89 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan AMDVLK 64k tg32
13.3 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan RADV 32k pp2048
242.77 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan RADV 32k tg32
14.3 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan RADV 64k pp2048
216.34 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan RADV 64k tg32
13.69 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-ud-q4-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan RADV pp512
270.56 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL Vulkan RADV tg128
14.86 tok/sNVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-nvidia-nemotron-3-super-120b-a12b-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 6.4.4 32k pp2048
300.11 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 6.4.4 32k tg32
18.47 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 6.4.4 64k pp2048
202.62 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 6.4.4 64k tg32
17.3 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 6.4.4 pp512
336.02 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen35-122b-a10b-ud-q5-k-xl-rocm644-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 6.4.4 tg128
19.73 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen35-122b-a10b-ud-q5-k-xl-rocm644-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7 nightlies 32k pp2048
272.63 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7 nightlies 32k tg32
18.69 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7 nightlies 64k pp2048
184.62 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7 nightlies 64k tg32
17.52 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7.2.3 32k pp2048
276.15 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7.2.3 32k tg32
17.73 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7.2.3 64k pp2048
192.28 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL ROCm 7.2.3 64k tg32
15.74 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan AMDVLK 32k pp2048
115.36 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan AMDVLK 32k tg32
19.04 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan AMDVLK 64k pp2048
81.21 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan AMDVLK 64k tg32
17.29 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan AMDVLK pp512
202.91 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen35-122b-a10b-ud-q5-k-xl-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan AMDVLK tg128
21.42 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen35-122b-a10b-ud-q5-k-xl-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan RADV 32k pp2048
224.82 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan RADV 32k tg32
20.38 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan RADV 64k pp2048
165.14 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan RADV 64k tg32
18.99 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-5-122b-a10b-ud-q5-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan RADV pp512
308.78 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen35-122b-a10b-ud-q5-k-xl-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.5-122B-A10B-UD-Q5_K_XL Vulkan RADV tg128
22.3 tok/sQwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen35-122b-a10b-ud-q5-k-xl-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 6.4.4 32k pp2048
172.59 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 6.4.4 32k tg32
6.13 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 6.4.4 64k pp2048
93.78 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 6.4.4 64k tg32
5.81 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7 nightlies 32k pp2048
190.65 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7 nightlies 64k pp2048
105.08 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7 nightlies 64k tg32
5.82 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7.2.3 32k pp2048
197.7 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7.2.3 64k pp2048
106.3 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7.2.3 64k tg32
5.83 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7.2.3 pp512
367.47 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-27b-ud-q8-k-xl-rocm-723-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL ROCm 7.2.3 tg128
6.51 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-27b-ud-q8-k-xl-rocm-723-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan AMDVLK 32k pp2048
60.84 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan AMDVLK 32k tg32
5.42 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan AMDVLK 64k pp2048
48.97 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan AMDVLK 64k tg32
5.03 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan RADV 32k pp2048
123.12 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan RADV 32k tg32
5.92 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan RADV 64k pp2048
63.14 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan RADV 64k tg32
5.6 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-27b-ud-q8-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan RADV pp512
274.97 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-27b-ud-q8-k-xl-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-27B-UD-Q8_K_XL Vulkan RADV tg128
6.3 tok/sQwen3.6-27B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-27b-ud-q8-k-xl-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 6.4.4 32k pp2048
426.65 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 6.4.4 32k tg32
23.77 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 6.4.4 64k pp2048
343.79 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 6.4.4 64k tg32
22.22 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7 nightlies 32k pp2048
409.92 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7 nightlies 32k tg32
23.96 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7 nightlies 64k pp2048
314.48 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7 nightlies 64k tg32
22.6 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7.2.3 32k pp2048
417.86 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7.2.3 32k tg32
23.91 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7.2.3 64k pp2048
322.65 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 ROCm 7.2.3 64k tg32
22.2 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan AMDVLK 32k pp2048
100.51 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan AMDVLK 32k tg32
10.93 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan AMDVLK 64k pp2048
85.86 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan AMDVLK 64k tg32
10.46 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan RADV 32k pp2048
275.8 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan RADV 32k tg32
10.23 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan RADV 64k pp2048
241.42 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-BF16 Vulkan RADV 64k tg32
9.94 tok/sQwen3.6-35B-A3B-BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-bf16-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 6.4.4 32k pp2048
807.33 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 6.4.4 32k tg32
43.47 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 6.4.4 64k pp2048
547.71 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 6.4.4 64k tg32
38.47 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 6.4.4 pp512
1116.91 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q4-k-xl-rocm644-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 6.4.4 tg128
50.32 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q4-k-xl-rocm644-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7 nightlies 32k pp2048
716.94 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7 nightlies 32k tg32
44.45 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7 nightlies 64k pp2048
468.89 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7 nightlies 64k tg32
39.3 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7 nightlies pp512
1085.14 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q4-k-xl-rocm7-nightlies-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7 nightlies tg128
51.47 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q4-k-xl-rocm7-nightlies-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7.2.3 32k pp2048
746.6 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7.2.3 32k tg32
43.84 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7.2.3 64k pp2048
493.36 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL ROCm 7.2.3 64k tg32
38.39 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan AMDVLK 32k pp2048
319.31 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan AMDVLK 32k tg32
45.3 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan AMDVLK 64k pp2048
207.54 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan AMDVLK 64k tg32
36.74 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan AMDVLK pp512
696.54 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q4-k-xl-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan AMDVLK tg128
59.57 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q4-k-xl-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan RADV 32k pp2048
715.52 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan RADV 32k tg32
49.21 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan RADV 64k pp2048
524.7 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q4_K_XL Vulkan RADV 64k tg32
43.23 tok/sQwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q4-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 6.4.4 32k pp2048
787.54 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm644-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 6.4.4 32k tg32
39.88 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm644-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 6.4.4 64k pp2048
537.23 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm644-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 6.4.4 64k tg32
35.72 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm644-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 6.4.4 pp512
1095.53 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-rocm644-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 6.4.4 tg128
45.13 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-rocm644-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7 nightlies 32k pp2048
704.46 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm7n-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7 nightlies 32k tg32
40.54 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm7n-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7 nightlies 64k pp2048
462.61 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm7n-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7 nightlies 64k tg32
36.31 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm7n-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7 nightlies pp512
1088.56 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-rocm7-nightlies-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7 nightlies tg128
46.21 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-rocm7-nightlies-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7.2.3 32k pp2048
728.92 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm723-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7.2.3 32k tg32
40.0 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm723-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7.2.3 64k pp2048
486.17 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm723-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL ROCm 7.2.3 64k tg32
35.45 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-rocm723-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan AMDVLK 32k pp2048
307.07 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-amdvlk-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan AMDVLK 32k tg32
37.93 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-amdvlk-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan AMDVLK 64k pp2048
202.18 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-amdvlk-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan AMDVLK 64k tg32
32.35 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-amdvlk-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan AMDVLK pp512
640.78 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-vulkanamdvlk-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan AMDVLK tg128
46.53 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-vulkanamdvlk-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan RADV 32k pp2048
682.8 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-radv-32k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan RADV 32k tg32
40.08 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-radv-32k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan RADV 64k pp2048
504.87 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-radv-64k-pp2048
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan RADV 64k tg32
35.8 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen3-6-35b-a3b-ud-q8-k-xl-radv-64k-tg32
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan RADV pp512
1045.0 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-vulkanradv-pp512
verifiedmedium · 0.70
kyuz0 Strix Halo 8060S Qwen3.6-35B-A3B-UD-Q8_K_XL Vulkan RADV tg128
46.33 tok/sQwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-qwen36-35b-a3b-ud-q8-k-xl-vulkanradv-tg128
verifiedmedium · 0.70
kyuz0 Strix Halo sweep: gemma-4-26B-A4B-it-UD-Q4_K_XL rocm-7_2_3 decode_tok_s on Radeon 8060S (gfx1151)
48.46 tok/s (tg128, batch 1)gemma-4-26B-A4B-it-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-gemma4-26b-a4b-q4kxl-rocm723-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: GLM-4.7-Flash-UD-Q8_K_XL vulkan_radv decode_tok_s on Radeon 8060S (gfx1151)
41.17 tok/s (tg128, batch 1)GLM-4.7-Flash-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-glm47-flash-q8kxl-vulkan-radv-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: gpt-oss-120b-mxfp4 rocm-7_2_3 decode_tok_s on Radeon 8060S (gfx1151)
51.81 tok/s (tg128, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-gpt-oss-120b-mxfp4-rocm723-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: gpt-oss-120b-mxfp4 rocm-7_2_3 prefill_tok_s on Radeon 8060S (gfx1151)
625.58 tok/s (pp512, batch 512)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-gpt-oss-120b-mxfp4-rocm723-pp512
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: gpt-oss-120b-mxfp4 vulkan_radv decode_tok_s on Radeon 8060S (gfx1151)
56.61 tok/s (tg128, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-gpt-oss-120b-mxfp4-vulkan-radv-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: Qwen3.5-122B-A10B-UD-Q5_K_XL rocm-7_2_3 decode_tok_s on Radeon 8060S (gfx1151)
20.12 tok/s (tg128, batch 1)Qwen3.5-122B-A10B-UD-Q5_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-qwen35-122b-a10b-q5kxl-rocm723-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: Qwen3.6-35B-A3B-UD-Q4_K_XL rocm-7_2_3 decode_tok_s on Radeon 8060S (gfx1151)
51.53 tok/s (tg128, batch 1)Qwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-qwen36-35b-a3b-q4kxl-rocm723-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: Qwen3.6-35B-A3B-UD-Q4_K_XL vulkan_radv decode_tok_s on Radeon 8060S (gfx1151)
60.43 tok/s (tg128, batch 1)Qwen3.6-35B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-qwen36-35b-a3b-q4kxl-vulkan-radv-tg128
verifiedmedium · 0.75
kyuz0 Strix Halo sweep: Qwen3.6-35B-A3B-UD-Q8_K_XL rocm-7_2_3 decode_tok_s on Radeon 8060S (gfx1151)
46.18 tok/s (tg128, batch 1)Qwen3.6-35B-A3B-UD-Q8_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-strix-8060s-qwen36-35b-a3b-q8kxl-rocm723-tg128
verifiedmedium · 0.75
LBNL 2014 US PUE by space type - closet: 2.0 (component sum: 1 | 0.05 | - | 0.93 | 0.02 -> IT 1, no transformer, UPS 0.93, cooling 0.02, lighting 0.02)
2.0 dimensionless PUE ratio (total facility energy / IT equipment energy), 2014 baseline. Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-lbnl-2016-us-closet-2014-2-0
verifiedhigh · 0.85
LBNL 2014 US PUE by space type - high-end: 1.7 (component sum: 1 | 0.03 | 0.1 | 0.55 | 0.02 -> IT 1, transformer 0.03, UPS 0.10, cooling 0.55, lighting 0.02)
1.7 dimensionless PUE ratio (total facility energy / IT equipment energy), 2014 baseline. Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-lbnl-2016-us-high-end-2014-1-7
verifiedhigh · 0.85
LBNL 2014 US PUE by space type - hyperscale: 1.2 (component sum: 1 | 0.02 | - | 0.16 | 0.02 -> IT 1, transformer 0.02, no UPS term, cooling 0.16, lighting 0.02)
1.2 dimensionless PUE ratio (total facility energy / IT equipment energy), 2014 baseline. Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-lbnl-2016-us-hyperscale-2014-1-2
verifiedhigh · 0.85
LBNL 2014 US PUE by space type - localized: 2.0 (component sum: 1 | 0.05 | 0.2 | 0.73 | 0.02 -> IT 1, transformer 0.05, UPS 0.20, cooling 0.73, lighting 0.02)
2.0 dimensionless PUE ratio (total facility energy / IT equipment energy), 2014 baseline. Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-lbnl-2016-us-localized-2014-2-0
verifiedhigh · 0.85
LBNL 2014 US PUE by space type - midtier: 1.9 (component sum: 1 | 0.05 | 0.2 | 0.63 | 0.02 -> IT 1, transformer 0.05, UPS 0.20, cooling 0.63, lighting 0.02)
1.9 dimensionless PUE ratio (total facility energy / IT equipment energy), 2014 baseline. Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-lbnl-2016-us-midtier-2014-1-9
verifiedhigh · 0.85
LBNL 2014 US PUE by space type - room: 2.5 (component sum: 1 | 0.05 | 0.2 | 1.23 | 0.02 -> IT 1, transformer 0.05, UPS 0.20, cooling 1.23, lighting 0.02)
2.5 dimensionless PUE ratio (total facility energy / IT equipment energy), 2014 baseline. Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-lbnl-2016-us-room-2014-2-5
verifiedhigh · 0.85
LeaseWeb amd-epyc-9334-genoa-dual DeepSeek-R1-0528-Qwen3-8B FP16 decode tok/s FP16
8.1 tok/s (decode, single-request)DeepSeek-R1-0528-Qwen3-8B on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/lw-epyc9334-dual-deepseek8b-fp16-tg
verifiedmedium · 0.65
LeaseWeb amd-epyc-9334-genoa-dual DeepSeek-R1-0528-Qwen3-8B FP16 TTFT s
8100.0 msDeepSeek-R1-0528-Qwen3-8B on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/lw-epyc9334-dual-deepseek8b-fp16-ttft
verifiedmedium · 0.65
LeaseWeb amd-epyc-9334-genoa-dual DeepSeek-R1-0528-Qwen3-8B Q4_K_M decode tok/s Q4_K_M
27.8 tok/s (decode, single-request)DeepSeek-R1-0528-Qwen3-8B on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/lw-epyc9334-dual-deepseek8b-q4-tg
verifiedmedium · 0.65
LeaseWeb amd-epyc-9334-genoa-dual Gpt-oss-20b FP16 decode tok/s FP16
26.2 tok/s (decode, single-request)Gpt-oss-20b on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/lw-epyc9334-dual-gptoss20b-fp16-tg
verifiedmedium · 0.65
LeaseWeb amd-epyc-9334-genoa-dual Gpt-oss-20b Q4_K_M TTFT s
3600.0 msGpt-oss-20b on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/lw-epyc9334-dual-gptoss20b-q4-ttft
verifiedmedium · 0.65
LeaseWeb amd-epyc-9334-genoa-dual Llama-2-7b Q4_K_M TTFT s
4800.0 msLlama-2-7b on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/lw-epyc9334-dual-llama27b-q4-ttft
verifiedmedium · 0.65
LeaseWeb nvidia-l4 DeepSeek-R1-8B FP16 decode tok/s FP16 (L4 GPU reference)
16.7 tok/s (decode, single-request)DeepSeek-R1-8B on nvidia-l4 via llama-cppbenchmarks/lw-l4-deepseek8b-fp16-tg
verifiedmedium · 0.65
LeaseWeb nvidia-l4 Gpt-oss-20b FP16 decode tok/s FP16 (L4 GPU reference)
58.6 tok/s (decode, single-request)Gpt-oss-20b on nvidia-l4 via llama-cppbenchmarks/lw-l4-gptoss20b-fp16-tg
verifiedmedium · 0.65
LeaseWeb: DeepSeek-R1-0528-Qwen3-8B Q4_K_M on dual EPYC 9334, 27.8 tok/s decode
27.8 tok/s (batch-1 decode, 24 threads)DeepSeek-R1-0528-Qwen3-8B Q4_K_M on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-deepseek8b-q4-decode
verifiedlow · 0.60
LeaseWeb: DeepSeek-R1-8B FP16 on dual EPYC 9334, 8.1 tok/s decode
8.1 tok/s (batch-1 decode, 24 threads)DeepSeek-R1-0528-Qwen3-8B FP16 on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-deepseek8b-fp16-decode
verifiedlow · 0.60
LeaseWeb: DeepSeek-R1-8B FP16 on NVIDIA L4, 16.7 tok/s decode
16.7 tok/s (batch-1 decode)DeepSeek-R1-0528-Qwen3-8B FP16 on nvidia-l4 via llama-cppbenchmarks/leaseweb-l4-deepseek8b-fp16-decode
verifiedlow · 0.60
LeaseWeb: DeepSeek-R1-8B Q4_K_M on dual EPYC 9334, TTFT ~4.2s (512-tok prompt)
4200.0 ms (time-to-first-token, 512-token prompt)DeepSeek-R1-0528-Qwen3-8B Q4_K_M on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-deepseek8b-q4-ttft
verifiedlow · 0.60
LeaseWeb: GPT-OSS-20B FP16 on dual EPYC 9334, 26.2 tok/s decode
26.2 tok/s (batch-1 decode, 24 threads)gpt-oss-20b FP16 on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-gptoss20b-fp16-decode
verifiedlow · 0.60
LeaseWeb: GPT-OSS-20B FP16 on NVIDIA L4, 58.6 tok/s decode
58.6 tok/s (batch-1 decode)gpt-oss-20b FP16 on nvidia-l4 via llama-cppbenchmarks/leaseweb-l4-gptoss20b-fp16-decode
verifiedlow · 0.60
LeaseWeb: GPT-OSS-20B Q4 on dual EPYC 9334, ~20 tok/s decode (24t)
20.0 tok/s (batch-1 decode, 24 threads)gpt-oss-20b Q4 on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-gptoss20b-q4-decode
verifiedlow · 0.60
LeaseWeb: GPT-OSS-20B Q4 on dual EPYC 9334, TTFT ~3.6s (512-tok prompt)
3600.0 ms (time-to-first-token, 512-token prompt)gpt-oss-20b Q4 on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-gptoss20b-q4-ttft
verifiedlow · 0.60
LeaseWeb: Llama-2-7B Q4_K_M on dual EPYC 9334, ~28 tok/s decode (24t)
28.0 tok/s (batch-1 decode, 24 threads)Llama-2-7B Q4_K_M on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-llama2-7b-q4km-decode
verifiedlow · 0.60
LeaseWeb: Llama-2-7B Q4_K_M on dual EPYC 9334, TTFT ~4.8s (512-tok prompt)
4800.0 ms (time-to-first-token, 512-token prompt)Llama-2-7B Q4_K_M on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-llama2-7b-q4-ttft
verifiedlow · 0.60
LeaseWeb: Mistral-7B-Instruct-v0.2 Q4_K_M on dual EPYC 9334, ~24 tok/s decode
24.0 tok/s (batch-1 decode, 24 threads)Mistral-7B-Instruct-v0.2 Q4_K_M on amd-epyc-9334-genoa-dual via llama-cppbenchmarks/leaseweb-epyc9334x2-mistral-7b-q4km-decode
verifiedlow · 0.60
lhl 8060S GLM-4.5-Air-UD-Q4_K_XL Vulkan tg1 decode
22.61 tok/sGLM-4.5-Air-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-glm-45-air-ud-q4-k-xl-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S GLM-4.5-Air-UD-Q4_K_XL Vulkan tg1 decode
22.89 tok/sGLM-4.5-Air-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-glm-45-air-ud-q4-k-xl-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S gpt-oss-120b-F16 ROCm tg1 decode
33.06 tok/sgpt-oss-120b-F16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-gpt-oss-120b-f16-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S gpt-oss-120b-F16 ROCm tg1 decode
33.92 tok/sgpt-oss-120b-F16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-gpt-oss-120b-f16-rocm-tg1
verifiedmedium · 0.70
lhl 8060S gpt-oss-120b-F16 Vulkan tg1 decode
33.72 tok/sgpt-oss-120b-F16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-gpt-oss-120b-f16-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S gpt-oss-120b-F16 Vulkan,RPC tg1 decode
33.34 tok/sgpt-oss-120b-F16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-gpt-oss-120b-f16-vulkanrpc-fa-tg1
verifiedmedium · 0.70
lhl 8060S gpt-oss-120b-F16 Vulkan,RPC tg1 decode
33.49 tok/sgpt-oss-120b-F16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-gpt-oss-120b-f16-vulkanrpc-tg1
verifiedmedium · 0.70
lhl 8060S llama-2-7b.Q4_K_M ROCm tg1 decode
37.14 tok/sllama-2-7b.Q4_K_M on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-2-7bq4-k-m-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S llama-2-7b.Q4_K_M ROCm tg1 decode
41.29 tok/sllama-2-7b.Q4_K_M on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-2-7bq4-k-m-rocm-tg1
verifiedmedium · 0.70
lhl 8060S llama-2-7b.Q4_K_M Vulkan tg1 decode
47.49 tok/sllama-2-7b.Q4_K_M on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-2-7bq4-k-m-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S llama-2-7b.Q4_K_M Vulkan tg1 decode
48.0 tok/sllama-2-7b.Q4_K_M on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-2-7bq4-k-m-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S Llama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL ROCm tg1 decode
16.27 tok/sLlama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-4-scout-17b-16e-instruct-ud-q4-k-x-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S Llama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL ROCm tg1 decode
17.2 tok/sLlama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-4-scout-17b-16e-instruct-ud-q4-k-x-rocm-tg1
verifiedmedium · 0.70
lhl 8060S Llama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL Vulkan tg1 decode
19.42 tok/sLlama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-4-scout-17b-16e-instruct-ud-q4-k-x-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S Llama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL Vulkan tg1 decode
19.9 tok/sLlama-4-Scout-17B-16E-Instruct-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-llama-4-scout-17b-16e-instruct-ud-q4-k-x-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S Mistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL ROCm tg1 decode
13.16 tok/sMistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-mistral-small-31-24b-instruct-2503-ud-q4-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S Mistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL ROCm tg1 decode
13.62 tok/sMistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-mistral-small-31-24b-instruct-2503-ud-q4-rocm-tg1
verifiedmedium · 0.70
lhl 8060S Mistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL Vulkan tg1 decode
14.24 tok/sMistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-mistral-small-31-24b-instruct-2503-ud-q4-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S Mistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL Vulkan tg1 decode
14.42 tok/sMistral-Small-3.1-24B-Instruct-2503-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-mistral-small-31-24b-instruct-2503-ud-q4-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL ROCm tg1 decode
13.67 tok/sQwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-235b-a22b-instruct-2507-ud-q3-k-xl-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL ROCm tg1 decode
13.55 tok/sQwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-235b-a22b-instruct-2507-ud-q3-k-xl-rocm-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL Vulkan tg1 decode
14.95 tok/sQwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-235b-a22b-instruct-2507-ud-q3-k-xl-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL Vulkan tg1 decode
15.16 tok/sQwen3-235B-A22B-Instruct-2507-UD-Q3_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-235b-a22b-instruct-2507-ud-q3-k-xl-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-30B-A3B-UD-Q4_K_XL ROCm tg1 decode
47.88 tok/sQwen3-30B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-30b-a3b-ud-q4-k-xl-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-30B-A3B-UD-Q4_K_XL ROCm tg1 decode
54.88 tok/sQwen3-30B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-30b-a3b-ud-q4-k-xl-rocm-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-30B-A3B-UD-Q4_K_XL Vulkan tg1 decode
67.2 tok/sQwen3-30B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-30b-a3b-ud-q4-k-xl-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-30B-A3B-UD-Q4_K_XL Vulkan tg1 decode
72.35 tok/sQwen3-30B-A3B-UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-30b-a3b-ud-q4-k-xl-vulkan-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-32B-Q8_0 ROCm tg1 decode
6.25 tok/sQwen3-32B-Q8_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-32b-q8-0-rocm-fa-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-32B-Q8_0 ROCm tg1 decode
6.4 tok/sQwen3-32B-Q8_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-32b-q8-0-rocm-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-32B-Q8_0 Vulkan tg1 decode
6.39 tok/sQwen3-32B-Q8_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-32b-q8-0-vulkan-fa-tg1
verifiedmedium · 0.70
lhl 8060S Qwen3-32B-Q8_0 Vulkan tg1 decode
6.43 tok/sQwen3-32B-Q8_0 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-8060s-qwen3-32b-q8-0-vulkan-tg1
verifiedmedium · 0.70
lhl Strix Halo: gpt-oss-120b MXFP4 tg32 on Radeon 8060S (lhl tuned rocWMMA @ d32768)
36.43 tok/s (tg32, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-rocm-tuned-tg32-d32k
verifiedmedium · 0.75
lhl Strix Halo: gpt-oss-120b MXFP4 tg32 on Radeon 8060S (ROCm rocWMMA (TheRock nightly 7.10.0a20251017))
46.56 tok/s (tg32, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-rocm-wmma-tg32-d0
verifiedmedium · 0.75
lhl Strix Halo: gpt-oss-120b MXFP4 tg32 on Radeon 8060S (ROCm rocWMMA @ d32768)
17.82 tok/s (tg32, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-rocm-wmma-tg32-d32k
verifiedmedium · 0.75
lhl Strix Halo: gpt-oss-120b MXFP4 tg32 on Radeon 8060S (Vulkan AMDVLK @ d32768)
31.54 tok/s (tg32, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-vulkan-amdvlk-tg32-d32k
verifiedmedium · 0.75
lhl Strix Halo: gpt-oss-120b MXFP4 tg32 on Radeon 8060S (Vulkan AMDVLK 2025.Q2.1 (-ub 512))
50.05 tok/s (tg32, batch 1)gpt-oss-120b-mxfp4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-vulkan-amdvlk-tg32-d0
verifiedmedium · 0.75
lhl/ggerganov: NVIDIA DGX Spark GB10 CUDA CUDA gpt-oss-120b pp2048 1221.23 t/s @32768
1221.23 tok/s per-request prefill (batch 1, depth 32768)gpt-oss-120b MXFP4/Q8 GGUF on nvidia-gb10 via llama-cppbenchmarks/lhl-dgx-spark-gpt-oss-120b-cuda-pp2048-d32k
verifiedmedium · 0.70
lhl/ggerganov: NVIDIA DGX Spark GB10 CUDA CUDA gpt-oss-120b pp2048 1689.47 t/s @0
1689.47 tok/s per-request prefill (batch 1, depth 0)gpt-oss-120b MXFP4/Q8 GGUF on nvidia-gb10 via llama-cppbenchmarks/lhl-dgx-spark-gpt-oss-120b-cuda-pp2048-d0
verifiedmedium · 0.70
lhl/ggerganov: NVIDIA DGX Spark GB10 CUDA CUDA gpt-oss-120b tg32 38.76 t/s @32768
38.76 tok/s per-request decode (batch 1, depth 32768)gpt-oss-120b MXFP4/Q8 GGUF on nvidia-gb10 via llama-cppbenchmarks/lhl-dgx-spark-gpt-oss-120b-cuda-tg32-d32k
verifiedmedium · 0.70
lhl/ggerganov: NVIDIA DGX Spark GB10 CUDA CUDA gpt-oss-120b tg32 52.87 t/s @0
52.87 tok/s per-request decode (batch 1, depth 0)gpt-oss-120b MXFP4/Q8 GGUF on nvidia-gb10 via llama-cppbenchmarks/lhl-dgx-spark-gpt-oss-120b-cuda-tg32-d0
verifiedmedium · 0.70
lhl/ggerganov: Strix Halo 8060S ROCm stock rocWMMA (-ub2048) ROCm-rocWMMA gpt-oss-120b pp2048 1006.65 t/s @0
1006.65 tok/s per-request prefill (batch 1, depth 0)gpt-oss-120b MXFP4/Q8 GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-rocm-wmma-pp2048-d0
verifiedmedium · 0.70
lhl/ggerganov: Strix Halo 8060S ROCm tuned-rocWMMA ROCm-rocWMMA-tuned gpt-oss-120b pp2048 977.22 t/s @0
977.22 tok/s per-request prefill (batch 1, depth 0)gpt-oss-120b MXFP4/Q8 GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-rocm-tuned-pp2048-d0
verifiedmedium · 0.70
lhl/ggerganov: Strix Halo 8060S Vulkan AMDVLK (-ub512) Vulkan-AMDVLK gpt-oss-120b pp2048 137.08 t/s @32768
137.08 tok/s per-request prefill (batch 1, depth 32768)gpt-oss-120b MXFP4/Q8 GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-vulkan-amdvlk-pp2048-d32k
verifiedmedium · 0.70
lhl/ggerganov: Strix Halo 8060S Vulkan AMDVLK (-ub512) Vulkan-AMDVLK gpt-oss-120b pp2048 729.1 t/s @0
729.1 tok/s per-request prefill (batch 1, depth 0)gpt-oss-120b MXFP4/Q8 GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/lhl-strix-8060s-gpt-oss-120b-vulkan-amdvlk-pp2048-d0
verifiedmedium · 0.70
llama-cpp on intel-arc-pro-b70: Gemma 4 31B gguf-q3-k-m pp512 tok/s (bentech, single B70 SYCL)
174.7 tok/s (pp512, batch 512)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q3km-sycl-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B gguf-q3-k-m tg128 tok/s (bentech, single B70 SYCL)
7.0 tok/s (tg128, batch-1 per-request)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q3km-sycl-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B gguf-q4-k-m pp512 tok/s (bentech, single B70 SYCL)
204.7 tok/s (pp512, batch 512)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q4km-sycl-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B gguf-q4-k-m tg128 tok/s (bentech, single B70 SYCL)
18.6 tok/s (tg128, batch-1 per-request)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q4km-sycl-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B gguf-q5-k-m pp512 tok/s (bentech, single B70 SYCL)
201.7 tok/s (pp512, batch 512)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q5km-sycl-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B gguf-q5-k-m tg128 tok/s (bentech, single B70 SYCL)
10.2 tok/s (tg128, batch-1 per-request)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q5km-sycl-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B None pp512 tok/s (bentech, dual B70+B60 SYCL layer-split 31:23)
163.5 tok/s (pp512, batch 512)Gemma 4 31B on intel-arc-pro-b70, intel-arc-pro-b60 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q4xl-sycl-dual-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B None pp512 tok/s (bentech, single B70 Vulkan)
119.6 tok/s (pp512, batch 512)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q4xl-vulkan-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B None tg128 tok/s (bentech, dual B70+B60 SYCL layer-split 31:23)
13.3 tok/s (tg128, batch-1 per-request)Gemma 4 31B on intel-arc-pro-b70, intel-arc-pro-b60 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q4xl-sycl-dual-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Gemma 4 31B None tg128 tok/s (bentech, single B70 Vulkan)
8.1 tok/s (tg128, batch-1 per-request)Gemma 4 31B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-gemma4-31b-q4xl-vulkan-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q3-k-m pp512 tok/s (bentech, single B70 Vulkan)
230.0 tok/s (pp512, batch 512)Qwen3.5-27B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q3km-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q3-k-m tg128 tok/s (bentech, single B70 Vulkan)
9.3 tok/s (tg128, batch-1 per-request)Qwen3.5-27B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q3km-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q4-k-m pp512 tok/s (bentech, dual B70+B60 Vulkan row-split 32:24)
153.0 tok/s (pp512, batch 512)Qwen3.5-27B on intel-arc-pro-b70, intel-arc-pro-b60 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q4km-dual-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q4-k-m pp512 tok/s (bentech, single B70 Vulkan)
238.2 tok/s (pp512, batch 512)Qwen3.5-27B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q4km-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q4-k-m tg128 tok/s (bentech, dual B70+B60 Vulkan row-split 32:24)
4.9 tok/s (tg128, batch-1 per-request)Qwen3.5-27B on intel-arc-pro-b70, intel-arc-pro-b60 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q4km-dual-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q4-k-m tg128 tok/s (bentech, single B70 Vulkan)
10.1 tok/s (tg128, batch-1 per-request)Qwen3.5-27B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q4km-tg
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q5-k-m pp512 tok/s (bentech, single B70 Vulkan)
233.5 tok/s (pp512, batch 512)Qwen3.5-27B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q5km-pp
verifiedmedium · 0.80
llama-cpp on intel-arc-pro-b70: Qwen3.5-27B gguf-q5-k-m tg128 tok/s (bentech, single B70 Vulkan)
8.5 tok/s (tg128, batch-1 per-request)Qwen3.5-27B on intel-arc-pro-b70 via llama-cppbenchmarks/bentech-b70-qwen35-27b-q5km-tg
verifiedmedium · 0.80
llama.cpp b7083 Vulkan: DeepSeek-R1-Distill-Llama-8B-Q8_0 decode 128 tok on amd-radeon-rx-9060-xt
37.04 tok/s (single-request decode, 128 generated tokens, batch 1)DeepSeek-R1-Distill-Llama-8B-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-dsr1-llama-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: DeepSeek-R1-Distill-Llama-8B-Q8_0 decode 128 tok on amd-radeon-rx-9070
69.23 tok/s (single-request decode, 128 generated tokens, batch 1)DeepSeek-R1-Distill-Llama-8B-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-dsr1-llama-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: DeepSeek-R1-Distill-Llama-8B-Q8_0 decode 128 tok on intel-arc-b580
13.69 tok/s (single-request decode, 128 generated tokens, batch 1)DeepSeek-R1-Distill-Llama-8B-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-dsr1-llama-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: DeepSeek-R1-Distill-Llama-8B-Q8_0 prefill 1024 tok on amd-radeon-rx-9060-xt
1787.9 tok/s (single-request prefill, 1024 prompt tokens)DeepSeek-R1-Distill-Llama-8B-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-dsr1-llama-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: DeepSeek-R1-Distill-Llama-8B-Q8_0 prefill 1024 tok on amd-radeon-rx-9070
2821.94 tok/s (single-request prefill, 1024 prompt tokens)DeepSeek-R1-Distill-Llama-8B-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-dsr1-llama-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: DeepSeek-R1-Distill-Llama-8B-Q8_0 prefill 1024 tok on intel-arc-b580
508.54 tok/s (single-request prefill, 1024 prompt tokens)DeepSeek-R1-Distill-Llama-8B-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-dsr1-llama-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: gpt-oss-20b-Q8_0 decode 128 tok on amd-radeon-rx-9060-xt
102.7 tok/s (single-request decode, 128 generated tokens, batch 1)gpt-oss-20b-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-gptoss20b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: gpt-oss-20b-Q8_0 decode 128 tok on amd-radeon-rx-9070
170.89 tok/s (single-request decode, 128 generated tokens, batch 1)gpt-oss-20b-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-gptoss20b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: gpt-oss-20b-Q8_0 decode 128 tok on intel-arc-b580
26.64 tok/s (single-request decode, 128 generated tokens, batch 1)gpt-oss-20b-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-gptoss20b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: gpt-oss-20b-Q8_0 prefill 1024 tok on amd-radeon-rx-9060-xt
2319.17 tok/s (single-request prefill, 1024 prompt tokens)gpt-oss-20b-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-gptoss20b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: gpt-oss-20b-Q8_0 prefill 1024 tok on amd-radeon-rx-9070
3531.8 tok/s (single-request prefill, 1024 prompt tokens)gpt-oss-20b-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-gptoss20b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: gpt-oss-20b-Q8_0 prefill 1024 tok on intel-arc-b580
1041.18 tok/s (single-request prefill, 1024 prompt tokens)gpt-oss-20b-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-gptoss20b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Llama-3.1-Tulu-3-8B-Q8_0 decode 128 tok on amd-radeon-rx-9060-xt
37.04 tok/s (single-request decode, 128 generated tokens, batch 1)Llama-3.1-Tulu-3-8B-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-tulu3-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Llama-3.1-Tulu-3-8B-Q8_0 decode 128 tok on amd-radeon-rx-9070
69.23 tok/s (single-request decode, 128 generated tokens, batch 1)Llama-3.1-Tulu-3-8B-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-tulu3-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Llama-3.1-Tulu-3-8B-Q8_0 decode 128 tok on intel-arc-b580
12.85 tok/s (single-request decode, 128 generated tokens, batch 1)Llama-3.1-Tulu-3-8B-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-tulu3-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Llama-3.1-Tulu-3-8B-Q8_0 prefill 1024 tok on amd-radeon-rx-9060-xt
1788.06 tok/s (single-request prefill, 1024 prompt tokens)Llama-3.1-Tulu-3-8B-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-tulu3-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Llama-3.1-Tulu-3-8B-Q8_0 prefill 1024 tok on amd-radeon-rx-9070
2823.06 tok/s (single-request prefill, 1024 prompt tokens)Llama-3.1-Tulu-3-8B-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-tulu3-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Llama-3.1-Tulu-3-8B-Q8_0 prefill 1024 tok on intel-arc-b580
507.15 tok/s (single-request prefill, 1024 prompt tokens)Llama-3.1-Tulu-3-8B-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-tulu3-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Mistral-7B-Instruct-v0.3-Q8_0 decode 128 tok on amd-radeon-rx-9060-xt
38.95 tok/s (single-request decode, 128 generated tokens, batch 1)Mistral-7B-Instruct-v0.3-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-mistral-7b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Mistral-7B-Instruct-v0.3-Q8_0 decode 128 tok on amd-radeon-rx-9070
72.72 tok/s (single-request decode, 128 generated tokens, batch 1)Mistral-7B-Instruct-v0.3-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-mistral-7b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Mistral-7B-Instruct-v0.3-Q8_0 decode 128 tok on intel-arc-b580
14.37 tok/s (single-request decode, 128 generated tokens, batch 1)Mistral-7B-Instruct-v0.3-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-mistral-7b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Mistral-7B-Instruct-v0.3-Q8_0 prefill 1024 tok on amd-radeon-rx-9060-xt
1792.13 tok/s (single-request prefill, 1024 prompt tokens)Mistral-7B-Instruct-v0.3-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-mistral-7b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Mistral-7B-Instruct-v0.3-Q8_0 prefill 1024 tok on amd-radeon-rx-9070
2829.08 tok/s (single-request prefill, 1024 prompt tokens)Mistral-7B-Instruct-v0.3-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-mistral-7b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Mistral-7B-Instruct-v0.3-Q8_0 prefill 1024 tok on intel-arc-b580
509.09 tok/s (single-request prefill, 1024 prompt tokens)Mistral-7B-Instruct-v0.3-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-mistral-7b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Qwen3-8B-Q8_0 decode 128 tok on amd-radeon-rx-9060-xt
36.53 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3-8B-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-qwen3-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Qwen3-8B-Q8_0 decode 128 tok on amd-radeon-rx-9070
67.8 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3-8B-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-qwen3-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Qwen3-8B-Q8_0 decode 128 tok on intel-arc-b580
13.28 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3-8B-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-qwen3-8b-tg128
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Qwen3-8B-Q8_0 prefill 1024 tok on amd-radeon-rx-9060-xt
1745.08 tok/s (single-request prefill, 1024 prompt tokens)Qwen3-8B-Q8_0 on amd-radeon-rx-9060-xt via llama-cppbenchmarks/phx-vk-b7083-9060xt-qwen3-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Qwen3-8B-Q8_0 prefill 1024 tok on amd-radeon-rx-9070
2754.84 tok/s (single-request prefill, 1024 prompt tokens)Qwen3-8B-Q8_0 on amd-radeon-rx-9070 via llama-cppbenchmarks/phx-vk-b7083-9070-qwen3-8b-pp1024
verifiedmedium · 0.70
llama.cpp b7083 Vulkan: Qwen3-8B-Q8_0 prefill 1024 tok on intel-arc-b580
463.79 tok/s (single-request prefill, 1024 prompt tokens)Qwen3-8B-Q8_0 on intel-arc-b580 via llama-cppbenchmarks/phx-vk-b7083-b580-qwen3-8b-pp1024
verifiedmedium · 0.70
llama.cpp CUDA A100 80 GB llama-2-7b Q4_0 pp512 FA
5285.96 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/llamacpp-scoreboard-a100-80gb-sxm4-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA A100 80 GB llama-2-7b Q4_0 pp512 noFA
4849.53 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/llamacpp-scoreboard-a100-80gb-sxm4-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA A100 80 GB llama-2-7b Q4_0 tg128 FA
200.9 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/llamacpp-scoreboard-a100-80gb-sxm4-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA A100 80 GB llama-2-7b Q4_0 tg128 noFA
190.88 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/llamacpp-scoreboard-a100-80gb-sxm4-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA DGX Spark llama-2-7b Q4_0 pp512 FA
3661.37 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-scoreboard-gb10-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA DGX Spark llama-2-7b Q4_0 pp512 noFA
3062.31 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-scoreboard-gb10-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA DGX Spark llama-2-7b Q4_0 tg128 FA
56.74 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-scoreboard-gb10-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA DGX Spark llama-2-7b Q4_0 tg128 noFA
57.21 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-scoreboard-gb10-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1060 llama-2-7b Q4_0 pp512 FA
446.19 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1060-6gb via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1060-6gb-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1060 llama-2-7b Q4_0 pp512 noFA
416.85 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1060-6gb via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1060-6gb-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1060 llama-2-7b Q4_0 tg128 FA
28.18 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1060-6gb via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1060-6gb-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1060 llama-2-7b Q4_0 tg128 noFA
27.79 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1060-6gb via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1060-6gb-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1070 Ti llama-2-7b Q4_0 pp512 FA
790.52 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1070-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1070-ti-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1070 Ti llama-2-7b Q4_0 pp512 noFA
714.44 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1070-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1070-ti-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1070 Ti llama-2-7b Q4_0 tg128 FA
37.87 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1070-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1070-ti-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1070 Ti llama-2-7b Q4_0 tg128 noFA
37.82 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1070-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1070-ti-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1080 Ti llama-2-7b Q4_0 pp512 FA
1138.14 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1080-ti-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1080 Ti llama-2-7b Q4_0 pp512 noFA
1084.41 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1080-ti-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1080 Ti llama-2-7b Q4_0 tg128 FA
61.38 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1080-ti-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1080 Ti llama-2-7b Q4_0 tg128 noFA
62.49 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1080-ti-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1660 llama-2-7b Q4_0 pp512 FA
154.45 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1660 via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1660-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1660 llama-2-7b Q4_0 pp512 noFA
148.91 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1660 via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1660-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA GTX 1660 llama-2-7b Q4_0 tg128 FA
41.43 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1660 via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1660-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA GTX 1660 llama-2-7b Q4_0 tg128 noFA
41.35 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1660 via llama-cppbenchmarks/llamacpp-scoreboard-gtx-1660-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA H100 80 GB llama-2-7b Q4_0 pp512 FA
11263.29 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-h100-sxm via llama-cppbenchmarks/llamacpp-scoreboard-h100-sxm-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA H100 80 GB llama-2-7b Q4_0 pp512 noFA
9918.34 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-h100-sxm via llama-cppbenchmarks/llamacpp-scoreboard-h100-sxm-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA H100 80 GB llama-2-7b Q4_0 tg128 FA
280.74 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-h100-sxm via llama-cppbenchmarks/llamacpp-scoreboard-h100-sxm-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA H100 80 GB llama-2-7b Q4_0 tg128 noFA
267.81 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-h100-sxm via llama-cppbenchmarks/llamacpp-scoreboard-h100-sxm-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Jetson AGX Orin llama-2-7b Q4_0 pp512 FA
1171.96 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-jetson-agx-orin-64gb via llama-cppbenchmarks/llamacpp-scoreboard-jetson-agx-orin-64gb-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Jetson AGX Orin llama-2-7b Q4_0 pp512 noFA
991.31 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-jetson-agx-orin-64gb via llama-cppbenchmarks/llamacpp-scoreboard-jetson-agx-orin-64gb-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Jetson AGX Orin llama-2-7b Q4_0 tg128 FA
35.88 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-jetson-agx-orin-64gb via llama-cppbenchmarks/llamacpp-scoreboard-jetson-agx-orin-64gb-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Jetson AGX Orin llama-2-7b Q4_0 tg128 noFA
33.58 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-jetson-agx-orin-64gb via llama-cppbenchmarks/llamacpp-scoreboard-jetson-agx-orin-64gb-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA on L40 48GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
8870.49 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40 via llama-cppbenchmarks/llamacpp-cuda-l40-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on L40 48GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
152.01 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40 via llama-cppbenchmarks/llamacpp-cuda-l40-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on L40 48GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
10097.64 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40 via llama-cppbenchmarks/llamacpp-cuda-l40-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on L40 48GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
153.76 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40 via llama-cppbenchmarks/llamacpp-cuda-l40-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on L40: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
10097.64 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40s via llama-cppbenchmarks/llamacpp-cuda-l40s-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on L40: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
153.76 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40s via llama-cppbenchmarks/llamacpp-cuda-l40s-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on L40: Llama-2-7B Q4_0 noFA pp512 prefill tok/s (scoreboard #15013)
8870.49 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40s via llama-cppbenchmarks/llamacpp-cuda-l40s-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on L40: Llama-2-7B Q4_0 noFA tg128 decode tok/s (scoreboard #15013)
152.01 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-l40s via llama-cppbenchmarks/llamacpp-cuda-l40s-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on NVIDIA A10G (A10 AWS variant): Llama-2-7B Q4_0 noFA pp512 t/s (#15013 comment)
2354.23 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a10 via llama-cppbenchmarks/llamacpp-cuda-a10g-llama27b-q40-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA on NVIDIA A10G (A10 AWS variant): Llama-2-7B Q4_0 noFA tg128 t/s (#15013 comment)
108.29 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a10 via llama-cppbenchmarks/llamacpp-cuda-a10g-llama27b-q40-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA on NVIDIA A30: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
2767.1 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-cuda-a30-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on NVIDIA A30: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
124.81 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-cuda-a30-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on NVIDIA A30: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
3068.72 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-cuda-a30-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on NVIDIA A30: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
131.93 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-cuda-a30-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on NVIDIA A40: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
4609.01 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a40 via llama-cppbenchmarks/llamacpp-cuda-a40-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on NVIDIA A40: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
124.11 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a40 via llama-cppbenchmarks/llamacpp-cuda-a40-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on NVIDIA A40: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
5256.38 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a40 via llama-cppbenchmarks/llamacpp-cuda-a40-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on NVIDIA A40: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
126.24 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a40 via llama-cppbenchmarks/llamacpp-cuda-a40-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on nvidia-quadro-rtx-6000: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
2941.85 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-6000 via llama-cppbenchmarks/llamacpp-cuda-quadro-rtx-6000-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-quadro-rtx-6000: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
93.11 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-6000 via llama-cppbenchmarks/llamacpp-cuda-quadro-rtx-6000-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-quadro-rtx-8000: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
2934.33 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-cuda-quadro-rtx-8000-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-quadro-rtx-8000: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
91.88 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-cuda-quadro-rtx-8000-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-rtx-4090-d: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
12506.97 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4090-d via llama-cppbenchmarks/llamacpp-cuda-rtx-4090-d-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-rtx-4090-d: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
191.57 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4090-d via llama-cppbenchmarks/llamacpp-cuda-rtx-4090-d-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-rtx-5090-laptop: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
7641.89 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-cuda-rtx-5090-laptop-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-rtx-5090-laptop: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
158.14 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-cuda-rtx-5090-laptop-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on nvidia-rtx-pro-4500-blackwell: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
168.9 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-4500-blackwell via llama-cppbenchmarks/llamacpp-cuda-rtx-pro-4500-blackwell-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 2000 Ada 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
1956.22 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-2000-ada-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2000 Ada 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
50.62 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-2000-ada-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2000 Ada: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
2250.14 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2000-ada via llama-cppbenchmarks/llamacpp-cuda-ada2000-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2000 Ada: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
50.71 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2000-ada via llama-cppbenchmarks/llamacpp-cuda-ada2000-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2080 Ti 11GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
2890.66 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-2080-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2080 Ti 11GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
107.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-2080-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2080 Ti 11GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
3107.61 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-2080-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 2080 Ti 11GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
109.17 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-2080-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3060 12GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
2137.5 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-cuda-rtx-3060-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3060 12GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
75.57 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-cuda-rtx-3060-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3060 12GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
2407.67 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-cuda-rtx-3060-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3060 12GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
76.92 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-cuda-rtx-3060-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3080 10GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
5013.86 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-cuda-rtx-3080-10gb-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3080 10GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
139.65 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-cuda-rtx-3080-10gb-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3080 10GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
5569.56 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-cuda-rtx-3080-10gb-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3080 10GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
139.95 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-cuda-rtx-3080-10gb-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 3090 Ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
6567.49 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090 Ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
171.19 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090 Ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
6924.01 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090 Ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
172.26 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
5174.69 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
158.16 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
5560.06 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 3090: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
161.89 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-cuda-rtx-3090-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 4000 Ada 20GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
2779.77 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-4000-ada-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4000 Ada 20GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
61.83 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-4000-ada-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4000 Ada 20GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
3171.86 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-4000-ada-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4000 Ada 20GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
61.37 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-4000-ada-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4060 Ti 8GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
3394.63 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4060-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-4060-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4060 Ti 8GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
63.86 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4060-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-4060-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4060 Ti 8GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
3803.45 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4060-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-4060-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4060 Ti 8GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
64.03 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4060-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-4060-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4070 Ti SUPER 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
6924.53 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4070-ti-super-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4070 Ti SUPER 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
132.26 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4070-ti-super-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4070 Ti SUPER 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
7612.32 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4070-ti-super-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4070 Ti SUPER 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
132.85 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4070-ti-super-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4080 SUPER 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
8125.15 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-super-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4080 SUPER 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
148.33 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-super-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4080 SUPER 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
9439.01 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-super-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4080 SUPER 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
147.48 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-super-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX 4080: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
8031.64 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080 via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 4080: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
142.49 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080 via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 4080: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
9205.93 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080 via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 4080: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
143.47 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080 via llama-cppbenchmarks/llamacpp-cuda-rtx-4080-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070 Ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
6952.38 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070 Ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
176.85 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070 Ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
8419.56 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070 Ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
182.43 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
5184.75 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on local-nvidia-rtx-5070 via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
127.54 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on local-nvidia-rtx-5070 via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
5783.44 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on local-nvidia-rtx-5070 via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5070: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
128.21 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on local-nvidia-rtx-5070 via llama-cppbenchmarks/llamacpp-cuda-rtx-5070-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5080: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
8297.36 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-cuda-rtx-5080-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5080: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
181.99 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-cuda-rtx-5080-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5080: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
9487.7 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-cuda-rtx-5080-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 5080: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
184.68 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-cuda-rtx-5080-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 6000 Ada: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
9229.23 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-6000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-6000-ada-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 6000 Ada: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
176.07 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-6000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-6000-ada-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX 6000 Ada: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
10576.85 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-6000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-6000-ada-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on RTX 6000 Ada: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
179.47 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-6000-ada via llama-cppbenchmarks/llamacpp-cuda-rtx-6000-ada-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on RTX A4000 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
2684.06 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4000-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4000 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
83.77 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4000-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4000 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
2807.83 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4000-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4000 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
85.17 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4000-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4500 20GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
2827.2 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4500 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4500-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4500 20GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
97.32 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4500 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4500-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4500 20GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
3453.1 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4500 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4500-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX A4500 20GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
103.0 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4500 via llama-cppbenchmarks/llamacpp-cuda-rtx-a4500-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX A5000 24GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
4028.16 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a5000-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX A5000 24GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
130.07 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a5000-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX A5000 24GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
4552.15 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a5000-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX A5000 24GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
135.83 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-cuda-rtx-a5000-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX PRO 4000 Blackwell 24GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
4992.83 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-4000-blackwell via llama-cppbenchmarks/llamacpp-cuda-rtx-pro-4000-blackwell-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX PRO 4000 Blackwell 24GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
131.66 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-4000-blackwell via llama-cppbenchmarks/llamacpp-cuda-rtx-pro-4000-blackwell-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX PRO 4000 Blackwell 24GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
5674.44 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-4000-blackwell via llama-cppbenchmarks/llamacpp-cuda-rtx-pro-4000-blackwell-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on RTX PRO 4000 Blackwell 24GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
136.38 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-4000-blackwell via llama-cppbenchmarks/llamacpp-cuda-rtx-pro-4000-blackwell-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on RTX PRO 4500 Blackwell: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
7251.66 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-4500-blackwell via llama-cppbenchmarks/llamacpp-cuda-rtx-pro-4500-blackwell-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla M40: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
290.17 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-cuda-tesla-m40-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla M40: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
39.98 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-cuda-tesla-m40-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla M40: Llama-2-7B Q4_0 noFA pp512 prefill tok/s (scoreboard #15013)
282.65 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-cuda-tesla-m40-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla M40: Llama-2-7B Q4_0 noFA tg128 decode tok/s (scoreboard #15013)
38.04 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-cuda-tesla-m40-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P100: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
760.8 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-cuda-tesla-p100-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P100: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
58.35 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-cuda-tesla-p100-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P100: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
787.36 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-cuda-tesla-p100-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P100: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
61.99 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-cuda-tesla-p100-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P40: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
1079.66 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-cuda-tesla-p40-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P40: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
53.73 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-cuda-tesla-p40-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P40: Llama-2-7B Q4_0 noFA pp512 prefill tok/s (scoreboard #15013)
1007.42 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-cuda-tesla-p40-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla P40: Llama-2-7B Q4_0 noFA tg128 decode tok/s (scoreboard #15013)
54.74 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-cuda-tesla-p40-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla T4 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
1219.06 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-t4 via llama-cppbenchmarks/llamacpp-cuda-tesla-t4-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on Tesla T4 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
46.38 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-t4 via llama-cppbenchmarks/llamacpp-cuda-tesla-t4-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on Tesla T4: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
1309.73 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-t4 via llama-cppbenchmarks/llamacpp-cuda-t4-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp CUDA on Tesla T4: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
44.03 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-t4 via llama-cppbenchmarks/llamacpp-cuda-t4-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp CUDA on Tesla V100 (PCIe 32GB): Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
2973.78 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-pcie-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-pcie-32gb-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 (PCIe 32GB): Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
134.76 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-pcie-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-pcie-32gb-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 (PCIe 32GB): Llama-2-7B Q4_0 noFA pp512 prefill tok/s (scoreboard #15013)
3042.64 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-pcie-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-pcie-32gb-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 (PCIe 32GB): Llama-2-7B Q4_0 noFA tg128 decode tok/s (scoreboard #15013)
129.08 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-pcie-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-pcie-32gb-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 32GB SXM2: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15013)
3042.64 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-sxm2-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-sxm2-32gb-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 32GB SXM2: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15013)
129.08 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-sxm2-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-sxm2-32gb-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 32GB SXM2: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15013)
2973.78 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-sxm2-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-sxm2-32gb-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp CUDA on Tesla V100 32GB SXM2: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15013)
134.76 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-sxm2-32gb via llama-cppbenchmarks/llamacpp-cuda-v100-sxm2-32gb-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp CUDA P106-100 llama-2-7b Q4_0 pp512 FA
438.49 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-p106-100 via llama-cppbenchmarks/llamacpp-scoreboard-p106-100-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA P106-100 llama-2-7b Q4_0 pp512 noFA
406.94 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-p106-100 via llama-cppbenchmarks/llamacpp-scoreboard-p106-100-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA P106-100 llama-2-7b Q4_0 tg128 FA
30.64 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-p106-100 via llama-cppbenchmarks/llamacpp-scoreboard-p106-100-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA P106-100 llama-2-7b Q4_0 tg128 noFA
30.4 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-p106-100 via llama-cppbenchmarks/llamacpp-scoreboard-p106-100-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro P1000 llama-2-7b Q4_0 pp512 FA
173.82 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p1000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro P1000 llama-2-7b Q4_0 pp512 noFA
183.4 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p1000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro P1000 llama-2-7b Q4_0 tg128 FA
13.65 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p1000-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro P1000 llama-2-7b Q4_0 tg128 noFA
13.99 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p1000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro P2000 llama-2-7b Q4_0 pp512 FA
311.55 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p2000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p2000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro P2000 llama-2-7b Q4_0 pp512 noFA
309.3 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p2000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p2000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro P2000 llama-2-7b Q4_0 tg128 FA
23.76 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p2000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p2000-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro P2000 llama-2-7b Q4_0 tg128 noFA
23.63 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-p2000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-p2000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 4000 llama-2-7b Q4_0 pp512 FA
1662.8 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-4000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-4000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 4000 llama-2-7b Q4_0 pp512 noFA
1536.89 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-4000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-4000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 4000 llama-2-7b Q4_0 tg128 FA
67.62 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-4000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-4000-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 4000 llama-2-7b Q4_0 tg128 noFA
65.62 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-4000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-4000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 6000 llama-2-7b Q4_0 pp512 FA
3053.96 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-6000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-6000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 6000 llama-2-7b Q4_0 pp512 noFA
2751.18 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-6000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-6000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 6000 llama-2-7b Q4_0 tg128 FA
104.38 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-6000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-6000-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 6000 llama-2-7b Q4_0 tg128 noFA
102.77 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-6000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-6000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 8000 llama-2-7b Q4_0 pp512 FA
3052.35 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-8000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 8000 llama-2-7b Q4_0 pp512 noFA
2709.95 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-8000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 8000 llama-2-7b Q4_0 tg128 FA
103.63 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-8000-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro RTX 8000 llama-2-7b Q4_0 tg128 noFA
102.68 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-rtx-8000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-rtx-8000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Quadro T1000 llama-2-7b Q4_0 pp512 FA
27.46 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-t1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-t1000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro T1000 llama-2-7b Q4_0 pp512 noFA
79.44 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-t1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-t1000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Quadro T1000 llama-2-7b Q4_0 tg128 noFA
27.82 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-quadro-t1000 via llama-cppbenchmarks/llamacpp-scoreboard-quadro-t1000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 2060 SUPER llama-2-7b Q4_0 pp512 FA
1563.77 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2060-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2060-super-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 2060 SUPER llama-2-7b Q4_0 pp512 noFA
1420.24 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2060-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2060-super-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 2060 SUPER llama-2-7b Q4_0 tg128 FA
61.13 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2060-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2060-super-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 2060 SUPER llama-2-7b Q4_0 tg128 noFA
60.04 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2060-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2060-super-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 2070 SUPER llama-2-7b Q4_0 pp512 FA
2293.29 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2070-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2070-super-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 2070 SUPER llama-2-7b Q4_0 pp512 noFA
2088.34 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2070-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2070-super-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 2070 SUPER llama-2-7b Q4_0 tg128 FA
87.71 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2070-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2070-super-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 2070 SUPER llama-2-7b Q4_0 tg128 noFA
88.06 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2070-super via llama-cppbenchmarks/llamacpp-scoreboard-rtx-2070-super-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 4050 Laptop llama-2-7b Q4_0 pp512 noFA
1725.85 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4050-laptop via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4050-laptop-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 4050 Laptop llama-2-7b Q4_0 tg128 noFA
43.72 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4050-laptop via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4050-laptop-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 4090 D llama-2-7b Q4_0 pp512 noFA
10293.86 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090-d via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4090-d-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 4090 D llama-2-7b Q4_0 tg128 noFA
189.33 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090-d via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4090-d-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 4090 llama-2-7b Q4_0 pp512 FA
14770.63 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4090-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 4090 llama-2-7b Q4_0 pp512 noFA
11992.7 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4090-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 4090 llama-2-7b Q4_0 tg128 FA
188.96 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4090-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 4090 llama-2-7b Q4_0 tg128 noFA
186.21 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-4090-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 5060 Ti llama-2-7b Q4_0 pp512 FA
4195.53 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5060-ti-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 5060 Ti llama-2-7b Q4_0 pp512 noFA
3737.25 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5060-ti-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 5060 Ti llama-2-7b Q4_0 tg128 FA
93.46 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5060-ti-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 5060 Ti llama-2-7b Q4_0 tg128 noFA
90.94 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5060-ti-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 5090 Laptop llama-2-7b Q4_0 pp512 noFA
6667.06 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5090-laptop-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 5090 Laptop llama-2-7b Q4_0 tg128 noFA
156.49 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5090-laptop-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 5090 llama-2-7b Q4_0 pp512 FA
14970.15 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5090-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 5090 llama-2-7b Q4_0 pp512 noFA
14073.41 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5090-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX 5090 llama-2-7b Q4_0 tg128 FA
300.4 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5090-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX 5090 llama-2-7b Q4_0 tg128 noFA
290.02 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-5090-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX A6000 llama-2-7b Q4_0 pp512 FA
5662.39 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a6000 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-a6000-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX A6000 llama-2-7b Q4_0 pp512 noFA
4913.93 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a6000 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-a6000-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX A6000 llama-2-7b Q4_0 tg128 FA
144.87 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a6000 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-a6000-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX A6000 llama-2-7b Q4_0 tg128 noFA
138.73 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a6000 via llama-cppbenchmarks/llamacpp-scoreboard-rtx-a6000-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX PRO 6000 Blackwell llama-2-7b Q4_0 pp512 FA
16618.98 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/llamacpp-scoreboard-rtx-pro-6000-blackwell-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX PRO 6000 Blackwell llama-2-7b Q4_0 pp512 noFA
14854.63 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/llamacpp-scoreboard-rtx-pro-6000-blackwell-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA RTX PRO 6000 Blackwell llama-2-7b Q4_0 tg128 FA
281.11 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/llamacpp-scoreboard-rtx-pro-6000-blackwell-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA RTX PRO 6000 Blackwell llama-2-7b Q4_0 tg128 noFA
274.2 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/llamacpp-scoreboard-rtx-pro-6000-blackwell-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Tesla K80 llama-2-7b Q4_0 pp512 FA
133.36 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-k80 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-k80-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Tesla K80 llama-2-7b Q4_0 pp512 noFA
133.14 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-k80 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-k80-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Tesla K80 llama-2-7b Q4_0 tg128 FA
14.27 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-k80 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-k80-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Tesla K80 llama-2-7b Q4_0 tg128 noFA
13.8 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-k80 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-k80-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Tesla P4 llama-2-7b Q4_0 pp512 FA
529.53 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p4 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-p4-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Tesla P4 llama-2-7b Q4_0 pp512 noFA
514.53 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p4 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-p4-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Tesla P4 llama-2-7b Q4_0 tg128 FA
33.12 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p4 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-p4-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Tesla P4 llama-2-7b Q4_0 tg128 noFA
33.29 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p4 via llama-cppbenchmarks/llamacpp-scoreboard-tesla-p4-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Titan V llama-2-7b Q4_0 pp512 FA
2481.25 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-scoreboard-titan-v-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Titan V llama-2-7b Q4_0 pp512 noFA
2617.46 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-scoreboard-titan-v-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Titan V llama-2-7b Q4_0 tg128 FA
112.17 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-scoreboard-titan-v-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Titan V llama-2-7b Q4_0 tg128 noFA
108.79 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-scoreboard-titan-v-nofa-tg128
verifiedlow · 0.60
llama.cpp CUDA Titan Xp llama-2-7b Q4_0 pp512 FA
1218.12 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-xp via llama-cppbenchmarks/llamacpp-scoreboard-titan-xp-fa-pp512
verifiedlow · 0.60
llama.cpp CUDA Titan Xp llama-2-7b Q4_0 pp512 noFA
1154.96 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-xp via llama-cppbenchmarks/llamacpp-scoreboard-titan-xp-nofa-pp512
verifiedlow · 0.60
llama.cpp CUDA Titan Xp llama-2-7b Q4_0 tg128 FA
73.84 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-xp via llama-cppbenchmarks/llamacpp-scoreboard-titan-xp-fa-tg128
verifiedlow · 0.60
llama.cpp CUDA Titan Xp llama-2-7b Q4_0 tg128 noFA
76.08 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-titan-xp via llama-cppbenchmarks/llamacpp-scoreboard-titan-xp-nofa-tg128
verifiedlow · 0.60
llama.cpp GPT-OSS 120B MXFP4 MoE batch-1 decode on AMD Radeon 8060S (Strix Halo)
49.0 tok/sgpt-oss-120b on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-strix-halo-8060s-gpt-oss-120b-mxfp4-decode
verifiedhigh · 0.85
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (7 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
107.81 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-7core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (7 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
14.19 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-7core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (7 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
108.21 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-7core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (7 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
7.92 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-7core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (8 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
117.96 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-8core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (8 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
14.15 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-8core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (8 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
117.25 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-8core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 (base, 7- or 8-core GPU) (8 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
7.91 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1 via llama-cppbenchmarks/llamacpp-metal-m1-8core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (14 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
262.65 tok/s (pp512, batch 512, per-request)Llama-2-7B F16 on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-14core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (14 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
12.75 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-14core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (14 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
232.55 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-14core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (14 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
35.52 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-14core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (14 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
235.16 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-14core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (14 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
21.95 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-14core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
302.14 tok/s (pp512, batch 512, per-request)Llama-2-7B F16 on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-16core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
12.75 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-16core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
266.25 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-16core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
36.41 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-16core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
270.37 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-16core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Pro (14- or 16-core GPU) (16 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
22.34 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m1-pro via llama-cppbenchmarks/llamacpp-metal-m1-pro-16core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M1 Ultra (64-core GPU, 128 GB): llama 7B F16 pp512 (discussion #4167, c1d0e7a)
1294.31 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-64core-f16-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M1 Ultra (64-core GPU, 128 GB): llama 7B F16 tg128 (discussion #4167, c1d0e7a)
47.82 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-64core-f16-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M1 Ultra (64-core GPU, 128 GB): llama 7B Q4_0 pp512 (discussion #4167, c1d0e7a)
1179.89 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-64core-q4-0-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M1 Ultra (64-core GPU, 128 GB): llama 7B Q4_0 tg128 (discussion #4167, c1d0e7a)
113.79 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-64core-q4-0-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M1 Ultra (64-core GPU, 128 GB): llama 7B Q8_0 pp512 (discussion #4167, c1d0e7a)
1167.67 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-64core-q8-0-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M1 Ultra (64-core GPU, 128 GB): llama 7B Q8_0 tg128 (discussion #4167, c1d0e7a)
79.13 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-64core-q8-0-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
201.34 tok/s (pp512, batch 512, per-request)Llama-2-7B F16 on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-10core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
6.72 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-10core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
179.57 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-10core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
21.91 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-10core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
181.4 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-10core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
12.21 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-10core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (8 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
145.91 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-8core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (8 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
21.7 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-8core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (8 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
147.27 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-8core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 (base, 8- or 10-core GPU) (8 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
12.18 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2 via llama-cppbenchmarks/llamacpp-metal-m2-8core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Max (30-core GPU, 32 GB): Qwen3.6-35B-A3B Q4_K_M pp4096
671.4 tok/s (pp4096, per-request, batch 1)Qwen3.6-35B-A3B Q4_K_M (20.22 GiB, MoE, 3B active) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-qwen36-35b-a3b-q4km-pp4096
verifiedmedium · 0.70
llama.cpp Metal on Apple M2 Max (30-core GPU, 32 GB): Qwen3.6-35B-A3B Q4_K_M tg128
58.56 tok/s (tg128, per-request, batch 1)Qwen3.6-35B-A3B Q4_K_M (20.22 GiB, MoE, 3B active) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-qwen36-35b-a3b-q4km-tg128
verifiedmedium · 0.70
llama.cpp Metal on Apple M2 Max (30-core GPU, 32 GB): Qwen3.8-27B IQ3_S pp4096
106.78 tok/s (pp4096, per-request, batch 1)Qwen3.8-27B (dense) IQ3_S (11.20 GiB) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-qwen38-27b-iq3s-pp4096
verifiedmedium · 0.70
llama.cpp Metal on Apple M2 Max (30-core GPU, 32 GB): Qwen3.8-27B IQ3_S tg128
10.82 tok/s (tg128, per-request, batch 1)Qwen3.8-27B (dense) IQ3_S (11.20 GiB) on apple-m2-max via llama-cppbenchmarks/llamacpp-metal-m2-max-30core-qwen38-27b-iq3s-tg128
verifiedmedium · 0.70
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (16 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
312.65 tok/s (pp512, batch 512, per-request)Llama-2-7B F16 on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-16core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (16 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
12.47 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-16core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (16 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
294.24 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-16core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (16 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
37.87 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-16core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (16 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
288.46 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-16core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (16 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
22.7 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-16core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (19 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
384.38 tok/s (pp512, batch 512, per-request)Llama-2-7B F16 on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-19core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (19 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
13.06 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-19core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (19 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
341.19 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-19core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (19 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
38.86 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-19core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (19 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
344.5 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-19core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M2 Pro (16- or 19-core GPU) (19 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
23.01 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m2-pro via llama-cppbenchmarks/llamacpp-metal-m2-pro-19core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
186.75 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m3 via llama-cppbenchmarks/llamacpp-metal-m3-10core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
21.34 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m3 via llama-cppbenchmarks/llamacpp-metal-m3-10core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
187.52 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m3 via llama-cppbenchmarks/llamacpp-metal-m3-10core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 (base, 8- or 10-core GPU) (10 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
12.27 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m3 via llama-cppbenchmarks/llamacpp-metal-m3-10core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU, Low Power Mode): llama 7B F16 pp512 (discussion #4167, c1d0e7a)
678.18 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-lpm-f16-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU, Low Power Mode): llama 7B F16 tg128 (discussion #4167, c1d0e7a)
19.89 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-lpm-f16-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU, Low Power Mode): llama 7B Q4_0 pp512 (discussion #4167, c1d0e7a)
691.12 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-lpm-q4-0-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU, Low Power Mode): llama 7B Q4_0 tg128 (discussion #4167, c1d0e7a)
49.95 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-lpm-q4-0-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU, Low Power Mode): llama 7B Q8_0 pp512 (discussion #4167, c1d0e7a)
659.98 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-lpm-q8-0-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU, Low Power Mode): llama 7B Q8_0 tg128 (discussion #4167, c1d0e7a)
32.11 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-lpm-q8-0-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU): llama 7B F16 pp512 (discussion #4167, c1d0e7a)
934.98 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-f16-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU): llama 7B F16 tg128 (discussion #4167, c1d0e7a)
27.82 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-f16-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU): llama 7B Q4_0 pp512 (discussion #4167, c1d0e7a)
889.62 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-q4-0-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU): llama 7B Q4_0 tg128 (discussion #4167, c1d0e7a)
81.57 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-q4-0-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU): llama 7B Q8_0 pp512 (discussion #4167, c1d0e7a)
873.59 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-q8-0-pp512-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Max (MBP 16, 40-core GPU): llama 7B Q8_0 tg128 (discussion #4167, c1d0e7a)
48.94 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-40core-q8-0-tg128-c1d0e7a
verifiedmedium · 0.70
llama.cpp Metal on Apple M3 Pro (14-core GPU bin) (14 GPU): Llama-2-7B Q4_0 pp512 tok/s (scoreboard #4167, 8e672ef)
269.49 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m3-pro-14core via llama-cppbenchmarks/llamacpp-metal-m3-pro-14core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 Pro (14-core GPU bin) (14 GPU): Llama-2-7B Q4_0 tg128 tok/s (scoreboard #4167, 8e672ef)
30.65 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m3-pro-14core via llama-cppbenchmarks/llamacpp-metal-m3-pro-14core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 Pro (14-core GPU bin) (14 GPU): Llama-2-7B Q8_0 pp512 tok/s (scoreboard #4167, 8e672ef)
272.11 tok/s (pp512, batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m3-pro-14core via llama-cppbenchmarks/llamacpp-metal-m3-pro-14core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M3 Pro (14-core GPU bin) (14 GPU): Llama-2-7B Q8_0 tg128 tok/s (scoreboard #4167, 8e672ef)
17.44 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m3-pro-14core via llama-cppbenchmarks/llamacpp-metal-m3-pro-14core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on Apple M4 (Mac mini, 10 CPU cores, 24 GB): llama 3.2 3B Q4_K_M pp512
530.01 tok/s (pp512, per-request, batch 1)llama 3.2 3B Q4_K_M (1.87 GiB, 3.21B) on local-apple-m4 via llama-cppbenchmarks/llamacpp-metal-m4-base-llama32-3b-q4km-pp512-2e97c5f
verifiedmedium · 0.70
llama.cpp Metal on Apple M4 (Mac mini, 10 CPU cores, 24 GB): llama 3.2 3B Q4_K_M tg128
45.95 tok/s (tg128, per-request, batch 1)llama 3.2 3B Q4_K_M (1.87 GiB, 3.21B) on local-apple-m4 via llama-cppbenchmarks/llamacpp-metal-m4-base-llama32-3b-q4km-tg128-2e97c5f
verifiedmedium · 0.70
llama.cpp Metal on Apple M4 (Mac mini, 10 CPU cores, 24 GB): Qwen2.5-Coder 7B Q4_K_M pp512
239.73 tok/s (pp512, per-request, batch 1)Qwen2.5-Coder 7B Q4_K_M (4.36 GiB, 7.62B) on local-apple-m4 via llama-cppbenchmarks/llamacpp-metal-m4-base-qwen25-coder-7b-q4km-pp512-2e97c5f
verifiedmedium · 0.70
llama.cpp Metal on Apple M4 (Mac mini, 10 CPU cores, 24 GB): Qwen2.5-Coder 7B Q4_K_M tg128
22.6 tok/s (tg128, per-request, batch 1)Qwen2.5-Coder 7B Q4_K_M (4.36 GiB, 7.62B) on local-apple-m4 via llama-cppbenchmarks/llamacpp-metal-m4-base-qwen25-coder-7b-q4km-tg128-2e97c5f
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 128 GB): llama 7B F16 pp512 (discussion #4167, 8e672ef)
1032.18 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-128gb-f16-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 128 GB): llama 7B F16 tg128 (discussion #4167, 8e672ef)
38.06 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-128gb-f16-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 128 GB): llama 7B Q4_0 pp512 (discussion #4167, 8e672ef)
990.53 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-128gb-q4-0-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 128 GB): llama 7B Q4_0 tg128 (discussion #4167, 8e672ef)
104.03 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-128gb-q4-0-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 128 GB): llama 7B Q8_0 pp512 (discussion #4167, 8e672ef)
1054.83 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-128gb-q8-0-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 128 GB): llama 7B Q8_0 tg128 (discussion #4167, 8e672ef)
65.11 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-128gb-q8-0-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 64 GB): llama 7B F16 pp512 (discussion #4167, 8e672ef)
1018.3 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-64gb-f16-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 64 GB): llama 7B F16 tg128 (discussion #4167, 8e672ef)
37.58 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-64gb-f16-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 64 GB): llama 7B Q4_0 pp512 (discussion #4167, 8e672ef)
987.1 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-64gb-q4-0-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 64 GB): llama 7B Q4_0 tg128 (discussion #4167, 8e672ef)
102.93 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-64gb-q4-0-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 64 GB): llama 7B Q8_0 pp512 (discussion #4167, 8e672ef)
1051.59 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-64gb-q8-0-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Max (MBP 16, 40-core GPU, 64 GB): llama 7B Q8_0 tg128 (discussion #4167, 8e672ef)
64.61 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-40core-64gb-q8-0-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B F16 pp512 (discussion #4167, 073bb2c)
695.51 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-f16-pp512-073bb2c
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B F16 pp512 (discussion #4167, 8e672ef)
423.3 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-f16-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B F16 pp512 (discussion #4167, c3a1128)
1310.28 tok/s (pp512, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-f16-pp512-c3a1128
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B F16 tg128 (discussion #4167, 073bb2c)
21.37 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-f16-tg128-073bb2c
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B F16 tg128 (discussion #4167, 8e672ef)
20.22 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-f16-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B F16 tg128 (discussion #4167, c3a1128)
21.11 tok/s (tg128, per-request, batch 1)llama 7B F16 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-f16-tg128-c3a1128
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q4_0 pp512 (discussion #4167, 073bb2c)
1135.71 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q4-0-pp512-073bb2c
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q4_0 pp512 (discussion #4167, 8e672ef)
403.19 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q4-0-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q4_0 pp512 (discussion #4167, c3a1128)
1350.43 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q4-0-pp512-c3a1128
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q4_0 tg128 (discussion #4167, 073bb2c)
63.74 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q4-0-tg128-073bb2c
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q4_0 tg128 (discussion #4167, 8e672ef)
60.04 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q4-0-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q4_0 tg128 (discussion #4167, c3a1128)
63.86 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q4-0-tg128-c3a1128
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q8_0 pp512 (discussion #4167, 073bb2c)
964.18 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q8-0-pp512-073bb2c
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q8_0 pp512 (discussion #4167, 8e672ef)
431.14 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q8-0-pp512-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q8_0 pp512 (discussion #4167, c3a1128)
1304.66 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q8-0-pp512-c3a1128
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q8_0 tg128 (discussion #4167, 073bb2c)
37.7 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q8-0-tg128-073bb2c
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q8_0 tg128 (discussion #4167, 8e672ef)
35.86 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q8-0-tg128-8e672ef
verifiedmedium · 0.70
llama.cpp Metal on Apple M5 Pro (MBP 14, 16-core GPU, 24 GB): llama 7B Q8_0 tg128 (discussion #4167, c3a1128)
37.78 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.74B) on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-24gb-q8-0-tg128-c3a1128
verifiedmedium · 0.70
llama.cpp Metal on Apple M6 (Mac mini, 12-core GPU, 32 GB): llama 7B F16 (12.55 GiB, 6.74B) pp512 (discussion #4167)
952.53 tok/s (pp512, per-request, batch 1)llama 7B F16 (12.55 GiB, 6.74B) on local-apple-m6 via llama-cppbenchmarks/llamacpp-metal-m6-f16-pp512
verifiedmedium · 0.70
llama.cpp Metal on Apple M6 (Mac mini, 12-core GPU, 32 GB): llama 7B F16 (12.55 GiB, 6.74B) tg128 (discussion #4167)
11.59 tok/s (tg128, per-request, batch 1)llama 7B F16 (12.55 GiB, 6.74B) on local-apple-m6 via llama-cppbenchmarks/llamacpp-metal-m6-f16-tg128
verifiedmedium · 0.70
llama.cpp Metal on Apple M6 (Mac mini, 12-core GPU, 32 GB): llama 7B Q4_0 (3.53 GiB, 6.74B) pp512 (discussion #4167)
988.93 tok/s (pp512, per-request, batch 1)llama 7B Q4_0 (3.53 GiB, 6.74B) on local-apple-m6 via llama-cppbenchmarks/llamacpp-metal-m6-q4-0-pp512
verifiedmedium · 0.70
llama.cpp Metal on Apple M6 (Mac mini, 12-core GPU, 32 GB): llama 7B Q4_0 (3.53 GiB, 6.74B) tg128 (discussion #4167)
35.74 tok/s (tg128, per-request, batch 1)llama 7B Q4_0 (3.53 GiB, 6.74B) on local-apple-m6 via llama-cppbenchmarks/llamacpp-metal-m6-q4-0-tg128
verifiedmedium · 0.70
llama.cpp Metal on Apple M6 (Mac mini, 12-core GPU, 32 GB): llama 7B Q8_0 (6.67 GiB, 6.74B) pp512 (discussion #4167)
970.46 tok/s (pp512, per-request, batch 1)llama 7B Q8_0 (6.67 GiB, 6.74B) on local-apple-m6 via llama-cppbenchmarks/llamacpp-metal-m6-q8-0-pp512
verifiedmedium · 0.70
llama.cpp Metal on Apple M6 (Mac mini, 12-core GPU, 32 GB): llama 7B Q8_0 (6.67 GiB, 6.74B) tg128 (discussion #4167)
21.01 tok/s (tg128, per-request, batch 1)llama 7B Q8_0 (6.67 GiB, 6.74B) on local-apple-m6 via llama-cppbenchmarks/llamacpp-metal-m6-q8-0-tg128
verifiedmedium · 0.70
llama.cpp Metal on apple-m1-ultra (48 GPU cores): Llama-2-7B F16 tg128 (scoreboard #4167, 8e672ef)
33.92 tok/s (decode, per-request, batch 1, tg128)Llama-2-7B on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-48core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m1-ultra (48 GPU cores): Llama-2-7B Q4_0 pp512 (scoreboard #4167, 8e672ef)
772.24 tok/s (prefill, per-request, pp512)Llama-2-7B on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-48core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m1-ultra (48 GPU cores): Llama-2-7B Q4_0 tg128 (scoreboard #4167, 8e672ef)
74.93 tok/s (decode, per-request, batch 1, tg128)Llama-2-7B on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-48core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m1-ultra (48 GPU cores): Llama-2-7B Q8_0 tg128 (scoreboard #4167, 8e672ef)
55.69 tok/s (decode, per-request, batch 1, tg128)Llama-2-7B on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-48core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m3-ultra (60 GPU cores): Llama-2-7B F16 tg128 (scoreboard #4167, 8e672ef)
42.24 tok/s (decode, per-request, batch 1, tg128)Llama-2-7B on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-60core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m3-ultra (60 GPU cores): Llama-2-7B Q4_0 pp512 (scoreboard #4167, 8e672ef)
1073.09 tok/s (prefill, per-request, pp512)Llama-2-7B on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-60core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m3-ultra (60 GPU cores): Llama-2-7B Q4_0 tg128 (scoreboard #4167, 8e672ef)
88.4 tok/s (decode, per-request, batch 1, tg128)Llama-2-7B on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-60core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on apple-m3-ultra (60 GPU cores): Llama-2-7B Q8_0 tg128 (scoreboard #4167, 8e672ef)
63.55 tok/s (decode, per-request, batch 1, tg128)Llama-2-7B on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-60core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.75
llama.cpp Metal on M1 Ultra (64 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
1168.89 tok/s (pp512, batch 512)Llama-2-7B F16 on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M1 Ultra (64 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
37.01 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M1 Ultra (64 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
1030.04 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M1 Ultra (64 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
83.73 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M1 Ultra (64 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
59.87 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m1-ultra via llama-cppbenchmarks/llamacpp-metal-m1-ultra-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
1401.85 tok/s (pp512, batch 512)Llama-2-7B F16 on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, c1d0e7a)
1627.82 tok/s (pp512, batch 512)Llama-2-7B F16 on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-f16-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
41.02 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, c1d0e7a)
49.55 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-f16-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
1238.48 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
1488.81 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-q4-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
94.27 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
125.21 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-q4-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
66.64 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M2 Ultra (76 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
82.83 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m2-ultra via llama-cppbenchmarks/llamacpp-metal-m2-ultra-llama27b-q8-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M3 Pro (18 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
357.45 tok/s (pp512, batch 512)Llama-2-7B F16 on local-apple-m3-pro via llama-cppbenchmarks/llamacpp-metal-m3-pro-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Pro (18 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
9.89 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m3-pro via llama-cppbenchmarks/llamacpp-metal-m3-pro-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Pro (18 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
341.67 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on local-apple-m3-pro via llama-cppbenchmarks/llamacpp-metal-m3-pro-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Pro (18 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
30.74 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m3-pro via llama-cppbenchmarks/llamacpp-metal-m3-pro-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Pro (18 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
17.53 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m3-pro via llama-cppbenchmarks/llamacpp-metal-m3-pro-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Ultra (80 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
1538.34 tok/s (pp512, batch 512)Llama-2-7B F16 on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Ultra (80 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
39.78 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Ultra (80 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
1471.24 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Ultra (80 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
92.14 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M3 Ultra (80 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
63.93 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m3-ultra via llama-cppbenchmarks/llamacpp-metal-m3-ultra-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (32 GPU): Llama-2-7B gguf-f16 pp512 tok/s (scoreboard #4167, 8e672ef)
736.25 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-f16.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-32core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (32 GPU): Llama-2-7B gguf-f16 tg128 tok/s (scoreboard #4167, 8e672ef)
24.29 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-f16.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-32core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (32 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
718.56 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-32core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (32 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
43.87 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-32core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (32 GPU): Llama-2-7B gguf-q8-0 pp512 tok/s (scoreboard #4167, 8e672ef)
713.93 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-32core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (32 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
69.95 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-32core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-f16 pp512 tok/s (scoreboard #4167, 8e672ef)
922.83 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-f16.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-40core-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-f16 tg128 tok/s (scoreboard #4167, 8e672ef)
31.64 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-f16.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-40core-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
891.94 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-40core-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
54.05 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-40core-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-q8-0 pp512 tok/s (scoreboard #4167, 8e672ef)
885.68 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-40core-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Max (40 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
83.06 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on apple-m4-max via llama-cppbenchmarks/llamacpp-metal-m4-max-40core-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Pro (20 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, 8e672ef)
464.48 tok/s (pp512, batch 512)Llama-2-7B F16 on local-apple-m4-pro via llama-cppbenchmarks/llamacpp-metal-m4-pro-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Pro (20 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, 8e672ef)
17.18 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m4-pro via llama-cppbenchmarks/llamacpp-metal-m4-pro-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Pro (20 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 8e672ef)
439.78 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on local-apple-m4-pro via llama-cppbenchmarks/llamacpp-metal-m4-pro-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Pro (20 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 8e672ef)
50.74 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m4-pro via llama-cppbenchmarks/llamacpp-metal-m4-pro-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M4 Pro (20 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 8e672ef)
30.69 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m4-pro via llama-cppbenchmarks/llamacpp-metal-m4-pro-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.80
llama.cpp Metal on M5 (10 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
722.79 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on local-apple-m5 via llama-cppbenchmarks/llamacpp-metal-m5-llama27b-q4-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 (10 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
31.88 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5 via llama-cppbenchmarks/llamacpp-metal-m5-llama27b-q4-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 (10 GPU): Llama-2-7B gguf-q8-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
715.42 tok/s (pp512, batch 512)Llama-2-7B gguf-q8-0.gguf on local-apple-m5 via llama-cppbenchmarks/llamacpp-metal-m5-llama27b-q8-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 (10 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
18.42 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5 via llama-cppbenchmarks/llamacpp-metal-m5-llama27b-q8-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Max (40 GPU): Llama-2-7B f16 pp512 tok/s (scoreboard #4167, c1d0e7a)
3158.49 tok/s (pp512, prefill batch-512, per-request)Llama-2-7B gguf-f16.gguf on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-llama27b-f16-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Max (40 GPU): Llama-2-7B f16 tg128 tok/s (scoreboard #4167, c1d0e7a)
37.11 tok/s (tg128, batch-1 decode, per-request)Llama-2-7B gguf-f16.gguf on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-llama27b-f16-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Max (40 GPU): Llama-2-7B q4-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
3219.99 tok/s (pp512, prefill batch-512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-llama27b-q4-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Max (40 GPU): Llama-2-7B q4-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
119.92 tok/s (tg128, batch-1 decode, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-llama27b-q4-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Max (40 GPU): Llama-2-7B q8-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
3143.81 tok/s (pp512, prefill batch-512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-llama27b-q8-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Max (40 GPU): Llama-2-7B q8-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
72.42 tok/s (tg128, batch-1 decode, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-max via llama-cppbenchmarks/llamacpp-metal-m5-max-llama27b-q8-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-f16 pp512 tok/s (scoreboard #4167, 60081bb)
1310.78 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-f16.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-llama27b-f16-pp512-60081bb
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-f16 tg128 tok/s (scoreboard #4167, 60081bb)
21.39 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-f16.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-llama27b-f16-tg128-60081bb
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, 60081bb)
1302.59 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-llama27b-q4-0-pp512-60081bb
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, 60081bb)
38.94 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-llama27b-q4-0-tg128-60081bb
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-q8-0 pp512 tok/s (scoreboard #4167, 60081bb)
1340.44 tok/s (pp512, prefill batch 512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-llama27b-q8-0-pp512-60081bb
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (16 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, 60081bb)
67.59 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-16core-llama27b-q8-0-tg128-60081bb
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (20 GPU): Llama-2-7B F16 pp512 tok/s (scoreboard #4167, c1d0e7a)
1588.78 tok/s (pp512, batch 512)Llama-2-7B F16 on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-llama27b-f16-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (20 GPU): Llama-2-7B F16 tg128 tok/s (scoreboard #4167, c1d0e7a)
21.55 tok/s (tg128, batch 1, per-request)Llama-2-7B F16 on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-llama27b-f16-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (20 GPU): Llama-2-7B gguf-q4-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
1620.64 tok/s (pp512, batch 512)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-llama27b-q4-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (20 GPU): Llama-2-7B gguf-q4-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
66.33 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-llama27b-q4-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Pro (20 GPU): Llama-2-7B gguf-q8-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
38.92 tok/s (tg128, batch 1, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-pro via llama-cppbenchmarks/llamacpp-metal-m5-pro-llama27b-q8-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Ultra (80 GPU): Llama-2-7B f16 pp512 tok/s (scoreboard #4167, c1d0e7a)
5207.14 tok/s (pp512, prefill batch-512, per-request)Llama-2-7B gguf-f16.gguf on local-apple-m5-ultra via llama-cppbenchmarks/llamacpp-metal-m5-ultra-llama27b-f16-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Ultra (80 GPU): Llama-2-7B f16 tg128 tok/s (scoreboard #4167, c1d0e7a)
73.6 tok/s (tg128, batch-1 decode, per-request)Llama-2-7B gguf-f16.gguf on local-apple-m5-ultra via llama-cppbenchmarks/llamacpp-metal-m5-ultra-llama27b-f16-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Ultra (80 GPU): Llama-2-7B q4-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
4944.66 tok/s (pp512, prefill batch-512, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-ultra via llama-cppbenchmarks/llamacpp-metal-m5-ultra-llama27b-q4-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Ultra (80 GPU): Llama-2-7B q4-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
179.1 tok/s (tg128, batch-1 decode, per-request)Llama-2-7B gguf-q4-0.gguf on local-apple-m5-ultra via llama-cppbenchmarks/llamacpp-metal-m5-ultra-llama27b-q4-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Ultra (80 GPU): Llama-2-7B q8-0 pp512 tok/s (scoreboard #4167, c1d0e7a)
4910.55 tok/s (pp512, prefill batch-512, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-ultra via llama-cppbenchmarks/llamacpp-metal-m5-ultra-llama27b-q8-0-pp512-c1d0e7a
verifiedmedium · 0.80
llama.cpp Metal on M5 Ultra (80 GPU): Llama-2-7B q8-0 tg128 tok/s (scoreboard #4167, c1d0e7a)
119.01 tok/s (tg128, batch-1 decode, per-request)Llama-2-7B gguf-q8-0.gguf on local-apple-m5-ultra via llama-cppbenchmarks/llamacpp-metal-m5-ultra-llama27b-q8-0-tg128-c1d0e7a
verifiedmedium · 0.80
llama.cpp on 4x NVIDIA A100 PCIe 80GB: LLaMA-3-70B F16 tg1024 tok/s (XiongjieDai)
7.38 tok/s (tg1024, batch-1)LLaMA-3-70B F16 on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80x4-llama3-70b-f16-tg1024
verifiedmedium · 0.65
llama.cpp on 4x NVIDIA A100 PCIe 80GB: LLaMA-3-8B Q4_K_M pp1024 tok/s (XiongjieDai)
8889.35 tok/s (pp1024, 4-GPU parallel split)LLaMA-3-8B Q4_K_M GGUF on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80x4-llama3-8b-q4km-pp1024
verifiedmedium · 0.65
llama.cpp on 8xMI300X: DeepSeek-V3 671B Q4_K_M decode throughput at batch 1, 256 generated tokens
36.53 tok/s (tg256, per-request, batch 1, all layers on GPU)DeepSeek-V3 671B (Q4_K_M, 376.65 GiB) on amd-instinct-mi300x via llama-cppbenchmarks/llama-cpp-mi300x-deepseek-v3-671b-q4-decode-tok-s
verifiedhigh · 0.85
llama.cpp on 8xMI300X: DeepSeek-V3 671B Q4_K_M prompt processing at 512-token prompts
439.14 tok/s (pp512, aggregate prompt processing)DeepSeek-V3 671B (Q4_K_M, 376.65 GiB) on amd-instinct-mi300x via llama-cppbenchmarks/llama-cpp-mi300x-deepseek-v3-671b-q4-prefill-tok-s
verifiedhigh · 0.85
llama.cpp on MI250: Llama-2-7b FP16 b1 128io throughput
44.4 tok/s (per-request b1)Llama-2-7b-hf on amd-instinct-mi250 via llama-cppbenchmarks/argonne-mi250-llamacpp-l27b-fp16-1x-b1-128io-tps
verifiedmedium · 0.80
llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-70B F16 tg1024 tok/s (XiongjieDai)
7.38 tok/s (tg1024, batch-1 per-request)LLaMA-3-70B F16 on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80-llama3-70b-f16-tg1024
verifiedmedium · 0.70
llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-70B Q4_K_M pp1024 tok/s (XiongjieDai)
726.65 tok/s (pp1024)LLaMA-3-70B Q4_K_M GGUF on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80-llama3-70b-q4km-pp1024
verifiedmedium · 0.70
llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-70B Q4_K_M tg1024 tok/s (XiongjieDai)
22.11 tok/s (tg1024, batch-1 per-request)LLaMA-3-70B Q4_K_M GGUF on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80-llama3-70b-q4km-tg1024
verifiedmedium · 0.70
llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-8B F16 tg1024 tok/s (XiongjieDai)
54.56 tok/s (tg1024, batch-1 per-request)LLaMA-3-8B F16 on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80-llama3-8b-f16-tg1024
verifiedmedium · 0.70
llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-8B Q4_K_M pp1024 tok/s (XiongjieDai)
5800.48 tok/s (pp1024)LLaMA-3-8B Q4_K_M GGUF on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80-llama3-8b-q4km-pp1024
verifiedmedium · 0.70
llama.cpp on NVIDIA A100 PCIe 80GB: LLaMA-3-8B Q4_K_M tg1024 tok/s (XiongjieDai)
138.31 tok/s (tg1024, batch-1 per-request)LLaMA-3-8B Q4_K_M GGUF on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xjd-a100pcie80-llama3-8b-q4km-tg1024
verifiedmedium · 0.70
llama.cpp Qwen 2.5 72B Q5_K_M batch-1 decode on AMD Radeon 8060S (Strix Halo)
4.1 tok/sqwen2-5-72b on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-strix-halo-8060s-qwen2-5-72b-q5-k-m-decode
verifiedhigh · 0.85
llama.cpp ROCm AI Max+ 395 llama-2-7b Q4_0 pp512 FA
1003.53 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/llamacpp-scoreboard-ryzen-ai-max-395-framework-desktop-fa-pp512
verifiedlow · 0.60
llama.cpp ROCm AI Max+ 395 llama-2-7b Q4_0 tg128 FA
49.87 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/llamacpp-scoreboard-ryzen-ai-max-395-framework-desktop-fa-tg128
verifiedlow · 0.60
llama.cpp ROCm AI PRO R9700 llama-2-7b Q4_0 pp512 FA
4773.07 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-rocm-r9700-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm AI PRO R9700 llama-2-7b Q4_0 pp512 noFA
4443.54 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-rocm-r9700-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm AI PRO R9700 llama-2-7b Q4_0 tg128 FA
97.98 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-rocm-r9700-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm AI PRO R9700 llama-2-7b Q4_0 tg128 noFA
93.84 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-rocm-r9700-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm Instinct MI25 llama-2-7b Q4_0 pp512 noFA
409.83 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-instinct-mi25 via llama-cppbenchmarks/llamacpp-scoreboard-instinct-mi25-nofa-pp512
verifiedlow · 0.60
llama.cpp ROCm Instinct MI25 llama-2-7b Q4_0 tg128 noFA
63.94 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-instinct-mi25 via llama-cppbenchmarks/llamacpp-scoreboard-instinct-mi25-nofa-tg128
verifiedlow · 0.60
llama.cpp ROCm Instinct MI300X llama-2-7b Q4_0 pp512 FA
11945.97 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi300x via llama-cppbenchmarks/llamacpp-rocm-mi300x-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm Instinct MI300X llama-2-7b Q4_0 pp512 noFA
11476.4 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi300x via llama-cppbenchmarks/llamacpp-rocm-mi300x-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm Instinct MI300X llama-2-7b Q4_0 tg128 FA
218.53 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi300x via llama-cppbenchmarks/llamacpp-rocm-mi300x-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm Instinct MI300X llama-2-7b Q4_0 tg128 noFA
232.92 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi300x via llama-cppbenchmarks/llamacpp-rocm-mi300x-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm Instinct MI60 llama-2-7b Q4_0 pp512 FA
1289.11 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi60 via llama-cppbenchmarks/llamacpp-rocm-mi60-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm Instinct MI60 llama-2-7b Q4_0 tg128 FA
91.46 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi60 via llama-cppbenchmarks/llamacpp-rocm-mi60-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on AI Max+ 395 (Strix Halo): Llama-2-7B Q4_0 noFA pp512 prefill tok/s (scoreboard #15021)
911.36 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-rocm-8060s-run1-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on AI Max+ 395 (Strix Halo): Llama-2-7B Q4_0 noFA tg128 decode tok/s (scoreboard #15021)
50.01 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-rocm-8060s-run1-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI100: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
2732.83 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi100 via llama-cppbenchmarks/llamacpp-rocm-mi100-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI100: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
110.48 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi100 via llama-cppbenchmarks/llamacpp-rocm-mi100-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI100: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15021)
2755.0 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi100 via llama-cppbenchmarks/llamacpp-rocm-mi100-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI100: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15021)
104.71 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi100 via llama-cppbenchmarks/llamacpp-rocm-mi100-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI210: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
2486.22 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi210 via llama-cppbenchmarks/llamacpp-rocm-mi210-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI210: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
124.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi210 via llama-cppbenchmarks/llamacpp-rocm-mi210-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI210: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15021)
2571.82 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi210 via llama-cppbenchmarks/llamacpp-rocm-mi210-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI210: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15021)
130.18 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-instinct-mi210 via llama-cppbenchmarks/llamacpp-rocm-mi210-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI50 32GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1057.24 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-rocm-mi50-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI50 32GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
98.95 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-rocm-mi50-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI50 32GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15021)
1129.43 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-rocm-mi50-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI50 32GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15021)
105.82 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-rocm-mi50-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI60: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1289.11 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi60 via llama-cppbenchmarks/llamacpp-rocm-mi60-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Instinct MI60: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
91.46 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi60 via llama-cppbenchmarks/llamacpp-rocm-mi60-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Radeon 8060S (System Shared): Llama-2-7B Q4_0 noFA pp512 prefill tok/s (scoreboard #15021)
351.36 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-rocm-8060s-run2-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Radeon 8060S (System Shared): Llama-2-7B Q4_0 noFA tg128 decode tok/s (scoreboard #15021)
47.97 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-rocm-8060s-run2-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Radeon PRO V620 32GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
1256.86 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-rocm-prov620-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon PRO V620 32GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
70.83 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-rocm-prov620-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon Pro V620: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1803.65 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-rocm-prov620-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon Pro V620: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
74.66 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-rocm-prov620-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon Pro VII: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1064.99 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-rocm-pro-vii-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Radeon Pro VII: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
87.45 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-rocm-pro-vii-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Radeon PRO W7900: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
3213.17 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-w7900 via llama-cppbenchmarks/llamacpp-rocm-w7900-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Radeon PRO W7900: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
121.18 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-w7900 via llama-cppbenchmarks/llamacpp-rocm-w7900-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Radeon PRO W7900: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #15021)
3472.86 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-w7900 via llama-cppbenchmarks/llamacpp-rocm-w7900-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on Radeon PRO W7900: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #15021)
127.43 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-w7900 via llama-cppbenchmarks/llamacpp-rocm-w7900-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on Radeon RX 5700 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
354.17 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-rocm-rx5700xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 5700 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
67.55 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-rocm-rx5700xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 6900 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1889.84 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-rocm-rx6900xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 6900 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
88.49 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-rocm-rx6900xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7600 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1099.64 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-rocm-rx7600xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7600 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
48.58 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-rocm-rx7600xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7800 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
2151.81 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-rocm-rx7800xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7800 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
100.94 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-rocm-rx7800xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7900 GRE: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
1456.98 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-rocm-rx7900gre-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7900 GRE: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
96.07 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-rocm-rx7900gre-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7900 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
3098.38 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-rocm-rx7900xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 7900 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
116.15 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-rocm-rx7900xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 9070: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #15021)
2381.77 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-rocm-rx9070-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on Radeon RX 9070: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #15021)
114.48 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-rocm-rx9070-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 5700 XT 8GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
314.17 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-rocm-rx5700xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 5700 XT 8GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
62.02 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-rocm-rx5700xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 6800 XT 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
1447.07 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800-xt via llama-cppbenchmarks/llamacpp-rocm-rx6800xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 6800 XT 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
83.92 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800-xt via llama-cppbenchmarks/llamacpp-rocm-rx6800xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 6900 XT 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
1948.31 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-rocm-rx6900xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 6900 XT 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
85.04 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-rocm-rx6900xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 7600 XT 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
1199.16 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-rocm-rx7600xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 7600 XT 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
47.65 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-rocm-rx7600xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 7800 XT 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
2304.63 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-rocm-rx7800xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 7800 XT 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
95.99 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-rocm-rx7800xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 7900 GRE 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
1598.79 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-rocm-rx7900gre-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 7900 GRE 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
97.53 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-rocm-rx7900gre-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 7900 XT 20GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
3261.75 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-rocm-rx7900xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 7900 XT 20GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
112.3 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-rocm-rx7900xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 7900 XTX: Llama-2-7B Q4_0 FA-off pp512 (#15021)
3552.27 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-7900xtx-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on RX 7900 XTX: Llama-2-7B Q4_0 FA-off tg128 (#15021)
167.11 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-7900xtx-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on RX 7900 XTX: Llama-2-7B Q4_0 FA-on pp512 (#15021)
3874.25 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-7900xtx-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on RX 7900 XTX: Llama-2-7B Q4_0 FA-on tg128 (#15021)
170.12 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-7900xtx-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on RX 9060 XT 16GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s
1419.67 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-rocm-rx9060xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 9060 XT 16GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s
67.58 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-rocm-rx9060xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 9060 XT 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
1479.27 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-rocm-rx9060xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 9060 XT 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
65.42 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-rocm-rx9060xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 9070 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
2452.68 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-rocm-rx9070-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 9070 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
115.32 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-rocm-rx9070-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 9070 XT 16GB: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s
4903.51 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-rx9070xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm on RX 9070 XT 16GB: Llama-2-7B Q4_0 FA-on tg128 decode tok/s
97.28 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-rx9070xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm on RX 9070 XT: Llama-2-7B Q4_0 FA-off pp512
5055.19 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-9070xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on RX 9070 XT: Llama-2-7B Q4_0 FA-off tg128
101.27 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-9070xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.80
llama.cpp ROCm on RX 9070 XT: Llama-2-7B Q4_0 FA-on pp512
4903.51 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-9070xt-llama27b-q40-fa-pp512
verifiedmedium · 0.80
llama.cpp ROCm on RX 9070 XT: Llama-2-7B Q4_0 FA-on tg128
97.28 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-9070xt-llama27b-q40-fa-tg128
verifiedmedium · 0.80
llama.cpp ROCm Pro VII llama-2-7b Q4_0 pp512 noFA
1064.99 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-rocm-vii-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm Pro VII llama-2-7b Q4_0 tg128 noFA
87.45 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-rocm-vii-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm Radeon 8060S llama-2-7b Q4_0 pp512 FA
366.08 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-scoreboard-radeon-8060s-ryzen-ai-max-fa-pp512
verifiedlow · 0.60
llama.cpp ROCm Radeon 8060S llama-2-7b Q4_0 tg128 FA
48.97 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-scoreboard-radeon-8060s-ryzen-ai-max-fa-tg128
verifiedlow · 0.60
llama.cpp ROCm Radeon 880M llama-2-7b Q4_0 pp512 FA
213.31 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-880m via llama-cppbenchmarks/llamacpp-scoreboard-radeon-880m-fa-pp512
verifiedlow · 0.60
llama.cpp ROCm Radeon 880M llama-2-7b Q4_0 pp512 noFA
163.25 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-880m via llama-cppbenchmarks/llamacpp-scoreboard-radeon-880m-nofa-pp512
verifiedlow · 0.60
llama.cpp ROCm Radeon 880M llama-2-7b Q4_0 tg128 FA
16.16 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-880m via llama-cppbenchmarks/llamacpp-scoreboard-radeon-880m-fa-tg128
verifiedlow · 0.60
llama.cpp ROCm Radeon 880M llama-2-7b Q4_0 tg128 noFA
12.97 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-880m via llama-cppbenchmarks/llamacpp-scoreboard-radeon-880m-nofa-tg128
verifiedlow · 0.60
llama.cpp ROCm RX 7900 XTX llama-2-7b Q4_0 pp512 FA
3874.25 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-xtx-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm RX 7900 XTX llama-2-7b Q4_0 pp512 noFA
3552.27 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-xtx-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm RX 7900 XTX llama-2-7b Q4_0 tg128 FA
170.12 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-xtx-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm RX 7900 XTX llama-2-7b Q4_0 tg128 noFA
167.11 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-rocm-xtx-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm RX 9070 XT llama-2-7b Q4_0 pp512 FA
4903.51 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp ROCm RX 9070 XT llama-2-7b Q4_0 pp512 noFA
5055.19 tok/s (pp512, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp ROCm RX 9070 XT llama-2-7b Q4_0 tg128 FA
97.28 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp ROCm RX 9070 XT llama-2-7b Q4_0 tg128 noFA
101.27 tok/s (tg128, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-rocm-xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp ROCm RX Vega 64 llama-2-7b Q4_0 pp512 FA
153.17 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-scoreboard-radeon-rx-vega-64-fa-pp512
verifiedlow · 0.60
llama.cpp ROCm RX Vega 64 llama-2-7b Q4_0 pp512 noFA
240.68 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-scoreboard-radeon-rx-vega-64-nofa-pp512
verifiedlow · 0.60
llama.cpp ROCm RX Vega 64 llama-2-7b Q4_0 tg128 FA
42.46 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-scoreboard-radeon-rx-vega-64-fa-tg128
verifiedlow · 0.60
llama.cpp ROCm RX Vega 64 llama-2-7b Q4_0 tg128 noFA
48.46 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-scoreboard-radeon-rx-vega-64-nofa-tg128
verifiedlow · 0.60
llama.cpp speculative decoding: a repeated-prompt benchmark protocol reads 9.5x-16.9x HIGH tok/s versus varied-prompt traffic on identical settings
13.4 value 13.4 with metric speedup_ratio: an INFLATION FACTOR (how many times too fast a benchmark reads when it replays one prompt instead of varying them; reported range 9.5x-16.9x), NOT 13.4 output tokens/s. BASELINE: the same llama.cpp b8400 build, same settings, 5 DISTINCT prompts of similar length instead of the same prompt 5 times. This was the most dangerous of the three, because 13.4 reads as a respectable per-GPU tok/s and would have been quoted as one.Qwen3-8B Q4_K_M + Qwen3-0.6B Q8_0 draft model on ? via llama-cppbenchmarks/ev-llamacpp-spec-bench-replay-inflates
verifiedhigh · 0.85
llama.cpp Vulkan AMD Radeon RX Vega 56 llama-2-7b Q4_0 pp512 FA
439.87 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-56 via llama-cppbenchmarks/llamacpp-vulkan-vega56-llama27b-q40-fa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 56 llama-2-7b Q4_0 pp512 noFA
428.54 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-56 via llama-cppbenchmarks/llamacpp-vulkan-vega56-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 56 llama-2-7b Q4_0 tg128 FA
50.23 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-56 via llama-cppbenchmarks/llamacpp-vulkan-vega56-llama27b-q40-fa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 56 llama-2-7b Q4_0 tg128 noFA
52.66 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-56 via llama-cppbenchmarks/llamacpp-vulkan-vega56-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 64 llama-2-7b Q4_0 pp512 FA
575.06 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-vulkan-vega64-llama27b-q40-fa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 64 llama-2-7b Q4_0 pp512 noFA
584.98 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-vulkan-vega64-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 64 llama-2-7b Q4_0 tg128 FA
63.43 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-vulkan-vega64-llama27b-q40-fa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon RX Vega 64 llama-2-7b Q4_0 tg128 noFA
67.7 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-vega-64 via llama-cppbenchmarks/llamacpp-vulkan-vega64-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon VII llama-2-7b Q4_0 pp512 FA
1059.14 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-vii via llama-cppbenchmarks/llamacpp-vulkan-radeonvii-llama27b-q40-fa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon VII llama-2-7b Q4_0 pp512 noFA
1007.19 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-vii via llama-cppbenchmarks/llamacpp-vulkan-radeonvii-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon VII llama-2-7b Q4_0 tg128 FA
101.19 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-vii via llama-cppbenchmarks/llamacpp-vulkan-radeonvii-llama27b-q40-fa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan AMD Radeon VII llama-2-7b Q4_0 tg128 noFA
110.86 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-vii via llama-cppbenchmarks/llamacpp-vulkan-radeonvii-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan Apple M2 Ultra (Asahi) llama-2-7b Q4_0 pp512 FA
198.83 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m2-ultra via llama-cppbenchmarks/llamacpp-vulkan-ultra-llama27b-q40-fa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Apple M2 Ultra (Asahi) llama-2-7b Q4_0 pp512 noFA
205.98 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m2-ultra via llama-cppbenchmarks/llamacpp-vulkan-ultra-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Apple M2 Ultra (Asahi) llama-2-7b Q4_0 tg128 FA
36.07 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m2-ultra via llama-cppbenchmarks/llamacpp-vulkan-ultra-llama27b-q40-fa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Apple M2 Ultra (Asahi) llama-2-7b Q4_0 tg128 noFA
34.34 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m2-ultra via llama-cppbenchmarks/llamacpp-vulkan-ultra-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc A750 llama-2-7b Q4_0 pp512 FA
303.37 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-a750 via llama-cppbenchmarks/llamacpp-vulkan-arc-a750-llama27b-q40-fa-pp512
verifiedlow · 0.60
llama.cpp Vulkan Intel Arc A750 llama-2-7b Q4_0 pp512 noFA
1075.94 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-a750 via llama-cppbenchmarks/llamacpp-vulkan-arc-a750-llama27b-q40-nofa-pp512
verifiedlow · 0.60
llama.cpp Vulkan Intel Arc A750 llama-2-7b Q4_0 tg128 FA
43.96 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-a750 via llama-cppbenchmarks/llamacpp-vulkan-arc-a750-llama27b-q40-fa-tg128
verifiedlow · 0.60
llama.cpp Vulkan Intel Arc A750 llama-2-7b Q4_0 tg128 noFA
42.66 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-a750 via llama-cppbenchmarks/llamacpp-vulkan-arc-a750-llama27b-q40-nofa-tg128
verifiedlow · 0.60
llama.cpp Vulkan Intel Arc A770 16GB llama-2-7b Q4_0 pp512 FA
1073.85 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-a770 via llama-cppbenchmarks/llamacpp-vulkan-a770-llama27b-q40-fa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan Intel Arc A770 16GB llama-2-7b Q4_0 pp512 noFA
1119.68 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-a770 via llama-cppbenchmarks/llamacpp-vulkan-a770-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan Intel Arc A770 16GB llama-2-7b Q4_0 tg128 FA
52.56 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-a770 via llama-cppbenchmarks/llamacpp-vulkan-a770-llama27b-q40-fa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan Intel Arc A770 16GB llama-2-7b Q4_0 tg128 noFA
53.07 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-a770 via llama-cppbenchmarks/llamacpp-vulkan-a770-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan Intel Arc B580 llama-2-7b Q4_0 pp512 FA
419.49 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-b580 via llama-cppbenchmarks/llamacpp-vulkan-b580-llama27b-q40-fa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc B580 llama-2-7b Q4_0 pp512 noFA
620.94 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-b580 via llama-cppbenchmarks/llamacpp-vulkan-b580-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc B580 llama-2-7b Q4_0 tg128 FA
72.0 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-b580 via llama-cppbenchmarks/llamacpp-vulkan-b580-llama27b-q40-fa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc B580 llama-2-7b Q4_0 tg128 noFA
70.14 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-b580 via llama-cppbenchmarks/llamacpp-vulkan-b580-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc Pro B60 llama-2-7b Q4_0 pp512 FA
274.76 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-pro-b60 via llama-cppbenchmarks/llamacpp-vulkan-b60-llama27b-q40-fa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc Pro B60 llama-2-7b Q4_0 pp512 noFA
522.36 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-pro-b60 via llama-cppbenchmarks/llamacpp-vulkan-b60-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc Pro B60 llama-2-7b Q4_0 tg128 FA
70.54 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-pro-b60 via llama-cppbenchmarks/llamacpp-vulkan-b60-llama27b-q40-fa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Intel Arc Pro B60 llama-2-7b Q4_0 tg128 noFA
68.55 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-pro-b60 via llama-cppbenchmarks/llamacpp-vulkan-b60-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Nvidia A30 llama-2-7b Q4_0 pp512 FA
3158.25 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-vulkan-a30-llama27b-q40-fa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Nvidia A30 llama-2-7b Q4_0 pp512 noFA
3290.7 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-vulkan-a30-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan Nvidia A30 llama-2-7b Q4_0 tg128 FA
117.43 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-vulkan-a30-llama27b-q40-fa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Nvidia A30 llama-2-7b Q4_0 tg128 noFA
111.48 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a30 via llama-cppbenchmarks/llamacpp-vulkan-a30-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan NVIDIA TITAN V llama-2-7b Q4_0 pp512 FA
780.48 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-vulkan-titanv-llama27b-q40-fa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan NVIDIA TITAN V llama-2-7b Q4_0 pp512 noFA
792.74 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-vulkan-titanv-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan NVIDIA TITAN V llama-2-7b Q4_0 tg128 FA
100.69 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-vulkan-titanv-llama27b-q40-fa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan NVIDIA TITAN V llama-2-7b Q4_0 tg128 noFA
109.21 tok/s (tg128)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-titan-v via llama-cppbenchmarks/llamacpp-vulkan-titanv-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on A100 80GB PCIe: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
6389.86 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-pcie via llama-cppbenchmarks/llamacpp-vulkan-a100-80gb-pcie-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on A100 80GB PCIe: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
160.78 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-pcie via llama-cppbenchmarks/llamacpp-vulkan-a100-80gb-pcie-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on amd-radeon-8060s-ryzen-ai-max: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1704.3 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-vulkan-strix-halo-8060s-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-8060s-ryzen-ai-max: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
54.38 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llamacpp-vulkan-strix-halo-8060s-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-ai-pro-r9700: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
5907.66 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-vulkan-r9700-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-ai-pro-r9700: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
152.7 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-vulkan-r9700-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-instinct-mi50x: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1127.37 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-vulkan-mi50x-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-instinct-mi50x: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
117.94 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-vulkan-mi50x-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-pro-v620: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1556.31 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-v620-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-pro-v620: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
79.24 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-v620-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-pro-vii: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
783.94 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-vii-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-pro-vii: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
108.45 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-vii-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-5700-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
538.31 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-5700-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-5700-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
74.43 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-5700-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6600-xt: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
574.65 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6600-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6600-xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6600-xt: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
53.92 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6600-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6600-xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6600: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
758.5 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6600 via llama-cppbenchmarks/llamacpp-vulkan-rx-6600-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6600: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
50.57 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6600 via llama-cppbenchmarks/llamacpp-vulkan-rx-6600-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6600: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
804.64 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6600 via llama-cppbenchmarks/llamacpp-vulkan-rx-6600-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6600: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
53.17 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6600 via llama-cppbenchmarks/llamacpp-vulkan-rx-6600-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6650-xt: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1029.52 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6650-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6650-xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6650-xt: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
62.14 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6650-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6650-xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6650-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1088.9 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6650-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6650-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6650-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
64.53 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6650-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6650-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6700-xt: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1051.2 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6700-xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6700-xt: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
83.88 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6700-xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6700-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1010.9 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6700-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6700-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
81.86 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6700-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6750-xt: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1040.58 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6750-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6750-xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6750-xt: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
81.98 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6750-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6750-xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6750-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
997.05 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6750-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6750-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6750-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
82.29 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6750-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6750-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6800-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1704.79 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6800-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6800-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
100.5 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6800-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6800: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1593.97 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800 via llama-cppbenchmarks/llamacpp-vulkan-rx-6800-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6800: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
101.47 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800 via llama-cppbenchmarks/llamacpp-vulkan-rx-6800-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6800: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1704.79 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800 via llama-cppbenchmarks/llamacpp-vulkan-rx-6800-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6800: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
100.5 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800 via llama-cppbenchmarks/llamacpp-vulkan-rx-6800-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6900-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1761.93 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6900-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-6900-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
106.15 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-6900-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7600-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1024.38 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-7600-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7600-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
56.11 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-7600-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7600: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1167.2 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600 via llama-cppbenchmarks/llamacpp-vulkan-rx-7600-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7600: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
58.03 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600 via llama-cppbenchmarks/llamacpp-vulkan-rx-7600-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7600: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1353.47 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600 via llama-cppbenchmarks/llamacpp-vulkan-rx-7600-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7600: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
60.88 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600 via llama-cppbenchmarks/llamacpp-vulkan-rx-7600-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7800-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2197.05 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-7800-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7800-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
124.86 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-7800-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7900-gre: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2251.37 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-gre-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7900-gre: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
110.5 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-gre-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7900-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2701.13 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7900-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
120.62 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7900-xtx: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
3889.54 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-xtx-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-7900-xtx: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
190.92 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-xtx-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-9060-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
3228.15 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-9060-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-9060-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
74.01 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-9060-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-9070-xt: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
5048.07 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-9070-xt-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-9070-xt: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
131.54 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-9070-xt-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-9070: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2859.98 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-vulkan-rx-9070-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on amd-radeon-rx-9070: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
119.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-vulkan-rx-9070-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on Apple M4 Max: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
724.77 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m4-max via llama-cppbenchmarks/llamacpp-vulkan-m4-max-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Apple M4 Max: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
75.02 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m4-max via llama-cppbenchmarks/llamacpp-vulkan-m4-max-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on apple-m4-max: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
557.46 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on apple-m4-max via llama-cppbenchmarks/llamacpp-vulkan-m4-max-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on apple-m4-max: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
71.79 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on apple-m4-max via llama-cppbenchmarks/llamacpp-vulkan-m4-max-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on Arc Pro B70: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
3379.0 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-pro-b70 via llama-cppbenchmarks/llamacpp-vulkan-b70-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on GB10: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2789.51 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-vulkan-gb10-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on GB10: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
58.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-vulkan-gb10-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Intel Arc Pro B70: Llama-2-7B Q4_0 noFA tg128 decode tok/s
112.02 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on intel-arc-pro-b70 via llama-cppbenchmarks/llamacpp-vulkan-arc-pro-b70-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on intel-arc-b570: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
913.95 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-b570 via llama-cppbenchmarks/llamacpp-vulkan-arc-b570-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on intel-arc-b570: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
49.64 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-b570 via llama-cppbenchmarks/llamacpp-vulkan-arc-b570-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on intel-arc-b570: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
288.51 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-b570 via llama-cppbenchmarks/llamacpp-vulkan-arc-b570-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on intel-arc-b570: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
50.49 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-b570 via llama-cppbenchmarks/llamacpp-vulkan-arc-b570-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on intel-arc-pro-b70: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
3150.55 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-pro-b70 via llama-cppbenchmarks/llamacpp-vulkan-arc-pro-b70-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on intel-arc-pro-b70: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
114.19 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on intel-arc-pro-b70 via llama-cppbenchmarks/llamacpp-vulkan-arc-pro-b70-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-a100-80gb-sxm4: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
7064.4 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/llamacpp-vulkan-a100-80gb-sxm4-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-a100-80gb-sxm4: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
170.56 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/llamacpp-vulkan-a100-80gb-sxm4-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-gb10: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
3321.91 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-vulkan-gb10-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-gb10: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
60.39 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gb10 via llama-cppbenchmarks/llamacpp-vulkan-gb10-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-gtx-1080-ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
585.54 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-vulkan-gtx-1080-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-gtx-1080-ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
67.75 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-vulkan-gtx-1080-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-gtx-1080-ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
647.81 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-vulkan-gtx-1080-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-gtx-1080-ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
71.58 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-gtx-1080-ti via llama-cppbenchmarks/llamacpp-vulkan-gtx-1080-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-2070-super: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1199.13 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2070-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-2070-super-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-2070-super: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
64.64 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2070-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-2070-super-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-2080-ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1936.25 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-2080-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-2080-ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
100.99 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-2080-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3060-ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2361.43 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3060-ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
86.74 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3060-ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2644.95 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3060-ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
90.35 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3060: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2012.88 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3060: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
80.59 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3080-10gb: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
4913.83 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-vulkan-rtx-3080-10gb-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3080-10gb: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
145.74 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-vulkan-rtx-3080-10gb-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3090: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
5200.68 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3090-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-3090: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
171.61 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3090-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070-super: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
5198.91 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-super-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070-super: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
112.0 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-super-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070-ti-super: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
6801.18 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-super-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070-ti-super: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
135.81 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-super-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070-ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
5515.26 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070-ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
113.52 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
4293.57 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4070: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
91.49 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4070 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4080-super: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
8007.37 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4080-super-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4080-super: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
150.2 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4080-super-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4090: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
10830.41 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4090-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-4090: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
190.1 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4090-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5060-ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
3912.65 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5060-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5060-ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
97.01 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5060-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5070-ti: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
7567.94 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5070-ti-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5070-ti: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
176.18 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5070-ti-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5080: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
8054.59 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5080-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5080: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
192.17 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5080-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5090-laptop: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
6011.18 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-laptop-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5090-laptop: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
154.31 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-laptop-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5090: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
11796.38 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-5090: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
273.68 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a2000: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1245.19 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a2000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a2000-llama27b-q40-nofa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a2000: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
45.52 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a2000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a2000-llama27b-q40-nofa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a2000: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1361.85 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a2000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a2000-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a2000: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
45.69 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a2000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a2000-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a4000: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
2482.74 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a4000-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a4000: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
76.07 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a4000-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a5000: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
4071.22 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a5000-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-rtx-a5000: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
140.43 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-vulkan-rtx-a5000-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-tesla-m40: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
93.35 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-m40-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-tesla-m40: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
41.68 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-m40-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-tesla-p100: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
685.51 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p100-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-tesla-p100: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
66.48 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p100-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-tesla-p40: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
484.37 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p40-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-tesla-p40: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
59.22 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p40-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-v100-sxm2-32gb: Llama-2-7B Q4_0 FA-on pp512 prefill tok/s (scoreboard #10879)
1411.25 tok/s (pp512, batch 512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-v100-sxm2-32gb via llama-cppbenchmarks/llamacpp-vulkan-v100-sxm2-32gb-llama27b-q40-fa-pp512
verifiedmedium · 0.75
llama.cpp Vulkan on nvidia-v100-sxm2-32gb: Llama-2-7B Q4_0 FA-on tg128 decode tok/s (scoreboard #10879)
142.13 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on nvidia-v100-sxm2-32gb via llama-cppbenchmarks/llamacpp-vulkan-v100-sxm2-32gb-llama27b-q40-fa-tg128
verifiedmedium · 0.75
llama.cpp Vulkan on Radeon AI Pro R9700: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
5609.82 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-vulkan-r9700-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon AI Pro R9700: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
145.67 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llamacpp-vulkan-r9700-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon Instinct MI50: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1119.55 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-vulkan-mi50-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon Instinct MI50: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
108.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi50x via llama-cppbenchmarks/llamacpp-vulkan-mi50-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon Instinct MI60: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
369.26 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi60 via llama-cppbenchmarks/llamacpp-vulkan-mi60-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon Instinct MI60: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
78.16 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-instinct-mi60 via llama-cppbenchmarks/llamacpp-vulkan-mi60-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon Pro V620: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1595.32 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-vulkan-prov620-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon Pro V620: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
81.78 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-v620 via llama-cppbenchmarks/llamacpp-vulkan-prov620-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon Pro VII: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
912.47 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-vulkan-provii-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon Pro VII: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
106.03 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-pro-vii via llama-cppbenchmarks/llamacpp-vulkan-provii-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 5700 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
529.69 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx5700xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 5700 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
70.73 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-5700-xt via llama-cppbenchmarks/llamacpp-vulkan-rx5700xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 6800 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1752.92 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx6800xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 6800 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
100.32 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx6800xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 6900 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1901.2 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx6900xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 6900 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
108.0 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-6900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx6900xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7600 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
840.85 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-vulkan-rx7600xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7600 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
53.02 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7600-xt via llama-cppbenchmarks/llamacpp-vulkan-rx7600xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7800 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2017.33 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx7800xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7800 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
118.27 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7800-xt via llama-cppbenchmarks/llamacpp-vulkan-rx7800xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7900 GRE: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2336.31 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-vulkan-rx7900gre-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7900 GRE: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
116.11 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-gre via llama-cppbenchmarks/llamacpp-vulkan-rx7900gre-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7900 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2941.58 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx7900xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7900 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
123.18 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xt via llama-cppbenchmarks/llamacpp-vulkan-rx7900xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 7900 XTX: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
3726.99 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-xtx-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon RX 7900 XTX: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
182.63 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/llamacpp-vulkan-rx-7900-xtx-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon RX 9060 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2844.8 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-vulkan-rx9060xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 9060 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
71.07 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9060-xt via llama-cppbenchmarks/llamacpp-vulkan-rx9060xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 9070 XT: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
5036.04 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-9070-xt-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon RX 9070 XT: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
137.11 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070-xt via llama-cppbenchmarks/llamacpp-vulkan-rx-9070-xt-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Radeon RX 9070: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
3164.1 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-vulkan-rx9070-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Radeon RX 9070: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
119.71 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on amd-radeon-rx-9070 via llama-cppbenchmarks/llamacpp-vulkan-rx9070-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on RTX 2080 Ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1888.24 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-vulkan-2080ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on RTX 2080 Ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
97.58 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-2080-ti via llama-cppbenchmarks/llamacpp-vulkan-2080ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on RTX 3060: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1815.7 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 3060: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
75.94 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3060 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3060-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 3070: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2113.02 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3070 via llama-cppbenchmarks/llamacpp-vulkan-3070-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on RTX 3070: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
78.71 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3070 via llama-cppbenchmarks/llamacpp-vulkan-3070-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on RTX 3080: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
4287.11 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-vulkan-rtx-3080-10gb-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 3080: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
139.15 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3080-10gb via llama-cppbenchmarks/llamacpp-vulkan-rtx-3080-10gb-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 3090: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
4666.15 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3090-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 3090: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
164.05 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-3090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-3090-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070 Super: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
4608.2 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-super-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070 Super: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
108.74 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-super-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070 Ti Super: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
6099.18 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-super-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070 Ti Super: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
129.45 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-super-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070 Ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
4981.44 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070 Ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
110.53 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
3179.37 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4070: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
92.29 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4070 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4070-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4080 Super: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
7101.18 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4080-super-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4080 Super: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
147.13 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4080-super via llama-cppbenchmarks/llamacpp-vulkan-rtx-4080-super-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4090: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
9452.03 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4090-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 4090: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
187.97 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-4090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-4090-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5060 Ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
3460.92 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5060-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5060 Ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
93.51 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5060-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5060-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5070 Ti: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
6213.63 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5070-ti-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5070 Ti: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
135.63 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5070-ti via llama-cppbenchmarks/llamacpp-vulkan-rtx-5070-ti-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5080: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
7444.99 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5080-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5080: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
185.1 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5080 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5080-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5090 Mobile: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
5546.39 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-laptop-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5090 Mobile: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
149.41 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5090-laptop via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-laptop-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5090: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
10381.64 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on RTX 5090: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
263.63 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-5090 via llama-cppbenchmarks/llamacpp-vulkan-rtx-5090-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on RTX A4000: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
2248.14 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-vulkan-a4000-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on RTX A4000: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
73.74 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a4000 via llama-cppbenchmarks/llamacpp-vulkan-a4000-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on RTX A5000: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
3641.55 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-vulkan-a5000-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on RTX A5000: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
139.89 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-rtx-a5000 via llama-cppbenchmarks/llamacpp-vulkan-a5000-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan on Tesla M40: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
92.48 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-m40-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Tesla M40: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
39.35 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-m40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-m40-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Tesla P100: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
678.14 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p100-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Tesla P100: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
63.16 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p100 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p100-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Tesla P40: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
488.06 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p40-llama27b-q40-nofa-pp512
verifiedmedium · 0.65
llama.cpp Vulkan on Tesla P40: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
59.36 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-tesla-p40 via llama-cppbenchmarks/llamacpp-vulkan-tesla-p40-llama27b-q40-nofa-tg128
verifiedmedium · 0.65
llama.cpp Vulkan on Tesla V100 32GB: Llama-2-7B Q4_0 FA-off pp512 prefill tok/s (scoreboard #10879)
1391.39 tok/s (pp512, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-pcie-32gb via llama-cppbenchmarks/llamacpp-vulkan-v100-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan on Tesla V100 32GB: Llama-2-7B Q4_0 FA-off tg128 decode tok/s (scoreboard #10879)
129.58 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on nvidia-v100-pcie-32gb via llama-cppbenchmarks/llamacpp-vulkan-v100-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp Vulkan Radeon Pro W5700 llama-2-7b Q4_0 pp512 FA
446.98 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-w5700 via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-w5700-llama27b-q40-fa-pp512
verifiedlow · 0.60
llama.cpp Vulkan Radeon Pro W5700 llama-2-7b Q4_0 pp512 noFA
449.85 tok/s (pp512)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-w5700 via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-w5700-llama27b-q40-nofa-pp512
verifiedlow · 0.60
llama.cpp Vulkan Radeon Pro W5700 llama-2-7b Q4_0 tg128 FA
71.3 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-w5700 via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-w5700-llama27b-q40-fa-tg128
verifiedlow · 0.60
llama.cpp Vulkan Radeon Pro W5700 llama-2-7b Q4_0 tg128 noFA
68.55 tok/s (tg128)llama-2-7b.Q4_0.gguf (ggml-org, 3.56 GiB, 6.74B params) on amd-radeon-pro-w5700 via llama-cppbenchmarks/llamacpp-vulkan-radeon-pro-w5700-llama27b-q40-nofa-tg128
verifiedlow · 0.60
llama.cpp Vulkan/MoltenVK on Apple M3 Ultra: Llama-2-7B Q4_0 noFA pp512 tok/s
1116.83 tok/s (pp512)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m3-ultra via llama-cppbenchmarks/llamacpp-vulkan-m3-ultra-llama27b-q40-nofa-pp512
verifiedmedium · 0.70
llama.cpp Vulkan/MoltenVK on Apple M3 Ultra: Llama-2-7B Q4_0 noFA tg128 decode tok/s
115.54 tok/s (tg128, per-request, batch 1)llama-2-7b.Q4_0.gguf (TheBloke, 3.56 GiB, 6.74B params) on apple-m3-ultra via llama-cppbenchmarks/llamacpp-vulkan-m3-ultra-llama27b-q40-nofa-tg128
verifiedmedium · 0.70
llama.cpp+ZenDNN on EPYC 9755 128c: gemma-4-12B-it gguf-q8-0 end-to-end tok/s (512in/128out/128seq)
695.38 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)gemma-4-12B-it on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gemma4-12b-q8-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gemma-4-12B-it gguf-q8-0 prefill tok/s (512in/128seq)
986.31 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)gemma-4-12B-it on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gemma4-12b-q8-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gemma-4-31B-it gguf-q8-0 end-to-end tok/s (512in/128out/128seq)
286.03 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)gemma-4-31B-it on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gemma4-31b-q8-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gemma-4-31B-it gguf-q8-0 prefill tok/s (512in/128seq)
404.54 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)gemma-4-31B-it on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gemma4-31b-q8-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gemma-4-E4B None end-to-end tok/s (512in/128out/128seq)
1550.29 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)gemma-4-E4B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gemma4-e4b-bf16-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gemma-4-E4B None prefill tok/s (512in/128seq)
2341.72 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)gemma-4-E4B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gemma4-e4b-bf16-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gpt-oss-20B None end-to-end tok/s (512in/128out/128seq)
1558.54 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)gpt-oss-20B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gptoss20b-bf16-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: gpt-oss-20B None prefill tok/s (512in/128seq)
2589.4 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)gpt-oss-20B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-gptoss20b-bf16-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Llama-3.1-8B-Instruct None end-to-end tok/s (512in/128out/128seq)
914.85 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Llama-3.1-8B-Instruct on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-llama31-8b-bf16-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Llama-3.1-8B-Instruct None prefill tok/s (512in/128seq)
1233.85 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Llama-3.1-8B-Instruct on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-llama31-8b-bf16-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Mistral-Small-3.2-24B-Instruct-2506 gguf-q8-0 end-to-end tok/s (512in/128out/128seq)
446.51 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Mistral-Small-3.2-24B-Instruct-2506 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-mistralsmall-24b-q8-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Mistral-Small-3.2-24B-Instruct-2506 gguf-q8-0 prefill tok/s (512in/128seq)
620.85 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Mistral-Small-3.2-24B-Instruct-2506 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-mistralsmall-24b-q8-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Mixtral-8x7B-Instruct-v0.1 gguf-q8-0 end-to-end tok/s (512in/128out/128seq)
573.59 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Mixtral-8x7B-Instruct-v0.1 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-mixtral8x7b-q8-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Mixtral-8x7B-Instruct-v0.1 gguf-q8-0 prefill tok/s (512in/128seq)
954.14 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Mixtral-8x7B-Instruct-v0.1 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-mixtral8x7b-q8-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Mixtral-8x7B-Instruct-v0.1 None end-to-end tok/s (512in/128out/128seq)
433.45 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Mixtral-8x7B-Instruct-v0.1 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-mixtral8x7b-bf16-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Mixtral-8x7B-Instruct-v0.1 None prefill tok/s (512in/128seq)
771.73 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Mixtral-8x7B-Instruct-v0.1 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-mixtral8x7b-bf16-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: phi-4 None end-to-end tok/s (512in/128out/128seq)
568.7 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)phi-4 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-phi4-bf16-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: phi-4 None prefill tok/s (512in/128seq)
751.24 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)phi-4 on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-phi4-bf16-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Qwen3.5-4B None end-to-end tok/s (512in/128out/128seq)
924.28 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Qwen3.5-4B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-qwen35-4b-bf16-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Qwen3.5-4B None prefill tok/s (512in/128seq)
2307.04 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Qwen3.5-4B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-qwen35-4b-bf16-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Qwen3.5-9B gguf-q8-0 end-to-end tok/s (512in/128out/128seq)
723.37 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Qwen3.5-9B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-qwen35-9b-q8-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Qwen3.5-9B gguf-q8-0 prefill tok/s (512in/128seq)
1557.65 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Qwen3.5-9B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-qwen35-9b-q8-pp512
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Qwen3.6-27B gguf-q8-0 end-to-end tok/s (512in/128out/128seq)
260.09 tok/s end-to-end (llama-batched-bench 512in/128out/128seq, aggregate)Qwen3.6-27B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-qwen36-27b-q8-e2e
verifiedlow · 0.60
llama.cpp+ZenDNN on EPYC 9755 128c: Qwen3.6-27B gguf-q8-0 prefill tok/s (512in/128seq)
517.55 tok/s prefill (llama-batched-bench pp512, 128 parallel seq, aggregate)Qwen3.6-27B on ? via llama-cppbenchmarks/amd-epyc9755-zendnn-qwen36-27b-q8-pp512
verifiedlow · 0.60
LLMCheck: Apple M3 Ultra (512GB), LM Studio, GLM 5.2 753B-A40B IQ1_S decode
22.0 tok/s (decode, per-request)GLM 5.2 753B-A40B on apple-m3-ultra via llama-cppbenchmarks/llmcheck-m3-ultra-glm5-753b-iq1s-lmstudio-decode
verifiedmedium · 0.80
LLMCheck: Apple M3 Ultra (512GB), MLX, GLM 5.2 753B-A40B Q4_K_M decode
15.0 tok/s (decode, per-request)GLM 5.2 753B-A40B on apple-m3-ultra via mlx-lmbenchmarks/llmcheck-m3-ultra-glm5-753b-q4km-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M4 Max, MLX, Qwen3.6-35B-A3B Q4_K_M decode
48.0 tok/s (decode, per-request)Qwen3.6-35B-A3B on apple-m4-max via mlx-lmbenchmarks/llmcheck-m4-max-qwen36-35b-a3b-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Max (128GB), MLX, Llama 3.3 70B Q4_K_M decode
12.0 tok/s (decode, per-request)Llama 3.3 70B on local-apple-m5-max via mlx-lmbenchmarks/llmcheck-m5-max-llama33-70b-q4km-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Max (128GB), MLX, Qwen3.5-35B-A3B Q4_K_M decode
52.0 tok/s (decode, per-request)Qwen3.5-35B-A3B on local-apple-m5-max via mlx-lmbenchmarks/llmcheck-m5-max-qwen35-35b-a3b-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Max (128GB), MLX, Qwen3.6-35B-A3B Q4_K_M decode
55.0 tok/s (decode, per-request)Qwen3.6-35B-A3B on local-apple-m5-max via mlx-lmbenchmarks/llmcheck-m5-max-qwen36-35b-a3b-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Max (128GB), Ollama, Llama 3.3 70B Q4_K_M decode
12.0 tok/s (decode, per-request)Llama 3.3 70B on local-apple-m5-max via ollamabenchmarks/llmcheck-m5-max-llama33-70b-q4km-ollama-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Max (64GB), Ollama, Qwen3.6-35B-A3B Q4_K_M decode
48.0 tok/s (decode, per-request)Qwen3.6-35B-A3B on local-apple-m5-max via ollamabenchmarks/llmcheck-m5-max-qwen36-35b-a3b-ollama-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Max, MLX, GLM-4.5-Air 106B-A?B Q4_K_M decode
34.0 tok/s (decode, per-request)GLM-4.5-Air on local-apple-m5-max via mlx-lmbenchmarks/llmcheck-m5-max-glm45-air-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Ultra (256GB), MLX, Qwen3-235B-A22B Q4_K_M decode
37.0 tok/s (decode, per-request)Qwen3-235B-A22B on local-apple-m5-ultra via mlx-lmbenchmarks/llmcheck-m5-ultra-qwen3-235b-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Ultra (512GB), MLX, Llama 3.1 405B Q4_K_M decode
4.0 tok/s (decode, per-request)Llama 3.1 405B on local-apple-m5-ultra via mlx-lmbenchmarks/llmcheck-m5-ultra-llama31-405b-q4km-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Ultra (96GB), MLX, Llama 3.3 70B Q4_K_M decode
24.0 tok/s (decode, per-request)Llama 3.3 70B on local-apple-m5-ultra via mlx-lmbenchmarks/llmcheck-m5-ultra-llama33-70b-q4km-mlx-decode
verifiedmedium · 0.80
LLMCheck: Apple M5 Ultra (96GB), MLX, Qwen3-Coder-Next 80B Q4_K_M decode
72.0 tok/s (decode, per-request)Qwen3-Coder-Next on local-apple-m5-ultra via mlx-lmbenchmarks/llmcheck-m5-ultra-qwen3-coder-next-mlx-decode
verifiedmedium · 0.80
Lookahead decoding acceptance length and speedup on Vicuna-13B (temperature=0)
1.62 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-lookahead-acceptance-vicuna13b-t0
verifiedhigh · 0.90
M1 Max 64GB llama-server b8139 GLM-4.7-Flash Q4_K_XL generation average
36.8 tok/s (average over the author's prompt set, ctx capped at 4096)GLM-4.7-Flash Q4_K_XL (unsloth, 16 GB on disk) on apple-m1-max via llama-cppbenchmarks/llamacpp-m1max64-glm47-flash-q4kxl-tg-avg
verifiedlow · 0.50
M1 Max 64GB llama-server b8139 GLM-4.7-Flash Q4_K_XL prompt-eval average
99.4 tok/s (average over the author's prompt set, ctx capped at 4096)GLM-4.7-Flash Q4_K_XL (unsloth, 16 GB on disk) on apple-m1-max via llama-cppbenchmarks/llamacpp-m1max64-glm47-flash-q4kxl-pp-avg
verifiedlow · 0.50
M1 Max 64GB llama-server b8139 Nemotron-3-Nano-30B-A3B Q4_K_XL generation average
43.7 tok/s (average over the author's prompt set, ctx capped at 4096)Nemotron-3-Nano-30B-A3B Q4_K_XL (unsloth, 22 GB on disk) on apple-m1-max via llama-cppbenchmarks/llamacpp-m1max64-nemotron3-nano-30b-q4kxl-tg-avg
verifiedlow · 0.50
M1 Max 64GB llama-server b8139 Nemotron-3-Nano-30B-A3B Q4_K_XL prompt-eval average
136.9 tok/s (average over the author's prompt set, ctx capped at 4096)Nemotron-3-Nano-30B-A3B Q4_K_XL (unsloth, 22 GB on disk) on apple-m1-max via llama-cppbenchmarks/llamacpp-m1max64-nemotron3-nano-30b-q4kxl-pp-avg
verifiedlow · 0.50
M1 Max 64GB llama-server b8139 Qwen3-Coder-30B-A3B IQ4_XS generation average
58.5 tok/s (average over the author's prompt set, ctx capped at 4096)Qwen3-Coder-30B-A3B IQ4_XS (unsloth, 15 GB on disk) on apple-m1-max via llama-cppbenchmarks/llamacpp-m1max64-qwen3-coder-30b-iq4xs-tg-avg
verifiedlow · 0.50
M1 Max 64GB llama-server b8139 Qwen3-Coder-30B-A3B IQ4_XS prompt-eval average
132.1 tok/s (average over the author's prompt set, ctx capped at 4096)Qwen3-Coder-30B-A3B IQ4_XS (unsloth, 15 GB on disk) on apple-m1-max via llama-cppbenchmarks/llamacpp-m1max64-qwen3-coder-30b-iq4xs-pp-avg
verifiedlow · 0.50
M3 Max DeepSeek R1 Distill Qwen 32B Q4_K - Medium decode (Ollama, siliconscore ref)
14.0 tok/s (single-request decode, per-request)DeepSeek R1 Distill Qwen 32B on apple-m3-max via ollamabenchmarks/m3max-deepseek-r1-distill-qwen-32b-q4-k-medium-ollama-ctxnp-tg
verifiedlow · 0.60
M3 Max DeepSeek R1 Distill Qwen 32B Q4_K - Medium TTFT (Ollama, siliconscore ref)
2000.0 ms (time to first token)DeepSeek R1 Distill Qwen 32B on apple-m3-max via ollamabenchmarks/m3max-deepseek-r1-distill-qwen-32b-q4-k-medium-ollama-ctxnp-ttft
verifiedlow · 0.60
M3 Max Gemma 3 27B Q4_K - Medium decode (MLX, siliconscore ref)
28.0 tok/s (single-request decode, per-request)Gemma 3 27B on apple-m3-max via mlx-lmbenchmarks/m3max-gemma-3-27b-q4-k-medium-mlx-ctxnp-tg
verifiedlow · 0.60
M3 Max Gemma 3 27B Q4_K - Medium TTFT (MLX, siliconscore ref)
1300.0 ms (time to first token)Gemma 3 27B on apple-m3-max via mlx-lmbenchmarks/m3max-gemma-3-27b-q4-k-medium-mlx-ctxnp-ttft
verifiedlow · 0.60
M3 Max Llama 2 7B Q4_0 decode (llama.cpp, siliconscore ref)
65.85 tok/s (single-request decode, per-request)Llama 2 7B on apple-m3-max via llama-cppbenchmarks/m3max-llama-2-7b-q4-0-llama-cpp-ctx512-tg
verifiedlow · 0.60
M3 Max Llama 2 7B Q4_0 prefill (llama.cpp, siliconscore ref)
690.99 tok/s (prompt eval)Llama 2 7B on apple-m3-max via llama-cppbenchmarks/m3max-llama-2-7b-q4-0-llama-cpp-ctx512-pp
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium decode (llama.cpp, siliconscore ref)
7.0 tok/s (single-request decode, per-request)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx16001-tg
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium decode (llama.cpp, siliconscore ref)
8.15 tok/s (single-request decode, per-request)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx258-tg
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium decode (llama.cpp, siliconscore ref)
6.13 tok/s (single-request decode, per-request)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx32170-tg
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium decode (llama.cpp, siliconscore ref)
7.48 tok/s (single-request decode, per-request)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx8013-tg
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium prefill (llama.cpp, siliconscore ref)
59.5 tok/s (prompt eval)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx16001-pp
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium prefill (llama.cpp, siliconscore ref)
67.86 tok/s (prompt eval)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx258-pp
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium prefill (llama.cpp, siliconscore ref)
50.32 tok/s (prompt eval)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx32170-pp
verifiedlow · 0.60
M3 Max Llama 3.3 70B Q4_K - Medium prefill (llama.cpp, siliconscore ref)
65.17 tok/s (prompt eval)Llama 3.3 70B on apple-m3-max via llama-cppbenchmarks/m3max-llama-3-3-70b-q4-k-medium-llama-cpp-ctx8013-pp
verifiedlow · 0.60
M3 Max Llama-2-7B F16 pp512 (llama.cpp #4167)
779.17 tok/s (pp512, batch 512)Llama-2-7B on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-llama27b-f16-pp512-8e672ef
verifiedmedium · 0.70
M3 Max Llama-2-7B F16 tg128 (llama.cpp #4167)
25.09 tok/s (tg128, batch 1, per-request)Llama-2-7B on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-llama27b-f16-tg128-8e672ef
verifiedmedium · 0.70
M3 Max Llama-2-7B Q4_0 pp512 (llama.cpp #4167)
759.7 tok/s (pp512, batch 512)Llama-2-7B on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-llama27b-q4-0-pp512-8e672ef
verifiedmedium · 0.70
M3 Max Llama-2-7B Q4_0 tg128 (llama.cpp #4167)
66.31 tok/s (tg128, batch 1, per-request)Llama-2-7B on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-llama27b-q4-0-tg128-8e672ef
verifiedmedium · 0.70
M3 Max Llama-2-7B Q8_0 pp512 (llama.cpp #4167)
757.64 tok/s (pp512, batch 512)Llama-2-7B on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-llama27b-q8-0-pp512-8e672ef
verifiedmedium · 0.70
M3 Max Llama-2-7B Q8_0 tg128 (llama.cpp #4167)
42.75 tok/s (tg128, batch 1, per-request)Llama-2-7B on apple-m3-max via llama-cppbenchmarks/llamacpp-metal-m3-max-llama27b-q8-0-tg128-8e672ef
verifiedmedium · 0.70
M3 Max llama-3-1-8b-instruct Q4_K - Medium decode (llamafile, siliconscore ref)
37.47979072222222 tok/s (single-request decode, per-request)llama-3-1-8b-instruct on apple-m3-max via llamafilebenchmarks/m3max-llama-3-1-8b-instruct-q4-k-medium-llamafile-ctxnp-tg
verifiedlow · 0.60
M3 Max llama-3-1-8b-instruct Q4_K - Medium prefill (llamafile, siliconscore ref)
443.4985221666666 tok/s (prompt eval)llama-3-1-8b-instruct on apple-m3-max via llamafilebenchmarks/m3max-llama-3-1-8b-instruct-q4-k-medium-llamafile-ctxnp-pp
verifiedlow · 0.60
M3 Max llama-3-1-8b-instruct Q4_K - Medium TTFT (llamafile, siliconscore ref)
2725.7188864444443 ms (time to first token)llama-3-1-8b-instruct on apple-m3-max via llamafilebenchmarks/m3max-llama-3-1-8b-instruct-q4-k-medium-llamafile-ctxnp-ttft
verifiedlow · 0.60
M3 Max llama-3-2-1b-instruct Q4_K - Medium decode (llamafile, siliconscore ref)
132.9925948888889 tok/s (single-request decode, per-request)llama-3-2-1b-instruct on apple-m3-max via llamafilebenchmarks/m3max-llama-3-2-1b-instruct-q4-k-medium-llamafile-ctxnp-tg
verifiedlow · 0.60
M3 Max llama-3-2-1b-instruct Q4_K - Medium prefill (llamafile, siliconscore ref)
2553.2959846666668 tok/s (prompt eval)llama-3-2-1b-instruct on apple-m3-max via llamafilebenchmarks/m3max-llama-3-2-1b-instruct-q4-k-medium-llamafile-ctxnp-pp
verifiedlow · 0.60
M3 Max llama-3-2-1b-instruct Q4_K - Medium TTFT (llamafile, siliconscore ref)
466.9405787777778 ms (time to first token)llama-3-2-1b-instruct on apple-m3-max via llamafilebenchmarks/m3max-llama-3-2-1b-instruct-q4-k-medium-llamafile-ctxnp-ttft
verifiedlow · 0.60
M3 Max Qwen 3 30B-A3B Q4_K - Medium decode (Ollama, siliconscore ref)
28.0 tok/s (single-request decode, per-request)Qwen 3 30B-A3B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-30b-a3b-q4-k-medium-ollama-ctxnp-tg
verifiedlow · 0.60
M3 Max Qwen 3 30B-A3B Q4_K - Medium TTFT (Ollama, siliconscore ref)
1300.0 ms (time to first token)Qwen 3 30B-A3B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-30b-a3b-q4-k-medium-ollama-ctxnp-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
10.19 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx1219-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
8.6 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx12416-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
10.09 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx1858-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
7.58 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx20172-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
10.41 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx264-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
9.89 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx2979-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
10.28 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx450-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
9.67 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx4669-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
10.29 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx723-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (llama.cpp, siliconscore ref)
9.2 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx7948-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
10.11 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx1219-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
8.57 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx12416-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
10.1 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx1858-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
7.53 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx20172-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
10.35 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx264-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
9.88 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx2979-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
10.33 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx450-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
9.66 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx4669-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
10.27 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx723-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 decode (Ollama, siliconscore ref)
9.18 tok/s (single-request decode, per-request)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx7948-tg
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
169.15 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx1219-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
127.96 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx12416-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
166.81 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx1858-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
111.18 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx20172-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
153.63 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx264-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
162.22 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx2979-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
171.37 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx450-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
154.16 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx4669-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
164.83 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx723-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (llama.cpp, siliconscore ref)
140.11 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx7948-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
168.32 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx1219-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
127.08 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx12416-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
166.96 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx1858-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
111.8 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx20172-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
152.12 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx264-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
161.46 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx2979-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
169.53 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx450-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
153.03 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx4669-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
163.79 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx723-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 prefill (Ollama, siliconscore ref)
138.99 tok/s (prompt eval)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx7948-pp
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
7210.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx1219-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
97030.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx12416-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
11140.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx1858-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
181440.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx20172-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
1720.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx264-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
18360.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx2979-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
2630.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx450-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
30290.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx4669-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
4390.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx723-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (llama.cpp, siliconscore ref)
56730.0 ms (time to first token)Qwen 3 32B on apple-m3-max via llama-cppbenchmarks/m3max-qwen-3-32b-q8-0-llama-cpp-ctx7948-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
7240.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx1219-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
97700.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx12416-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
11130.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx1858-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
180430.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx20172-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
1740.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx264-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
18450.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx2979-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
2650.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx450-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
30510.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx4669-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
4410.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx723-ttft
verifiedlow · 0.60
M3 Max Qwen 3 32B Q8_0 TTFT (Ollama, siliconscore ref)
57180.0 ms (time to first token)Qwen 3 32B on apple-m3-max via ollamabenchmarks/m3max-qwen-3-32b-q8-0-ollama-ctx7948-ttft
verifiedlow · 0.60
M3 Max qwen-2-5-14b-instruct Q4_K - Medium decode (llamafile, siliconscore ref)
19.768525888888888 tok/s (single-request decode, per-request)qwen-2-5-14b-instruct on apple-m3-max via llamafilebenchmarks/m3max-qwen-2-5-14b-instruct-q4-k-medium-llamafile-ctxnp-tg
verifiedlow · 0.60
M3 Max qwen-2-5-14b-instruct Q4_K - Medium prefill (llamafile, siliconscore ref)
226.180183 tok/s (prompt eval)qwen-2-5-14b-instruct on apple-m3-max via llamafilebenchmarks/m3max-qwen-2-5-14b-instruct-q4-k-medium-llamafile-ctxnp-pp
verifiedlow · 0.60
M3 Max qwen-2-5-14b-instruct Q4_K - Medium TTFT (llamafile, siliconscore ref)
5407.046111222224 ms (time to first token)qwen-2-5-14b-instruct on apple-m3-max via llamafilebenchmarks/m3max-qwen-2-5-14b-instruct-q4-k-medium-llamafile-ctxnp-ttft
verifiedlow · 0.60
M3 Max Qwen3.5-35B-A3B Q4_K - Medium decode (Ollama, siliconscore ref)
22.0 tok/s (single-request decode, per-request)Qwen3.5-35B-A3B on apple-m3-max via ollamabenchmarks/m3max-qwen3-5-35b-a3b-q4-k-medium-ollama-ctxnp-tg
verifiedlow · 0.60
M3 Max Qwen3.5-35B-A3B Q4_K - Medium TTFT (Ollama, siliconscore ref)
1500.0 ms (time to first token)Qwen3.5-35B-A3B on apple-m3-max via ollamabenchmarks/m3max-qwen3-5-35b-a3b-q4-k-medium-ollama-ctxnp-ttft
verifiedlow · 0.60
M3 Ultra 256GB rapid-mlx: Devstral-Small-2 4bit cold TTFT
291 ms (cold time to first token, one request)Devstral-Small-2 (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-devstral-small-2-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Devstral-Small-2 4bit long decode
47.2 tok/s (long-context decode, per-request; token count not printed)Devstral-Small-2 (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-devstral-small-2-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: GLM-4.5-Air 4bit cold TTFT
708 ms (cold time to first token, one request)GLM-4.5-Air (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-glm45-air-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: GLM-4.5-Air 4bit long decode
53.6 tok/s (long-context decode, per-request; token count not printed)GLM-4.5-Air (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-glm45-air-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: GLM-4.7-Flash 8bit cold TTFT
362 ms (cold time to first token, one request)GLM-4.7-Flash (8bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-glm47-flash-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: GLM-4.7-Flash 8bit long decode
57.2 tok/s (long-context decode, per-request; token count not printed)GLM-4.7-Flash (8bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-glm47-flash-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: GPT-OSS-20B mxfp4-q8 cold TTFT
339 ms (cold time to first token, one request)GPT-OSS-20B (mxfp4-q8, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-gptoss-20b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: GPT-OSS-20B mxfp4-q8 long decode
124.2 tok/s (long-context decode, per-request; token count not printed)GPT-OSS-20B (mxfp4-q8, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-gptoss-20b-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Hermes-3-Llama-3.1-8B 4bit cold TTFT
152 ms (cold time to first token, one request)Hermes-3-Llama-3.1-8B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-hermes3-llama31-8b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Hermes-3-Llama-3.1-8B 4bit long decode
122.9 tok/s (long-context decode, per-request; token count not printed)Hermes-3-Llama-3.1-8B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-hermes3-llama31-8b-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: MiniMax-M2.5 4bit cold TTFT
1300 ms (cold time to first token, one request)MiniMax-M2.5 (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-minimax-m25-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: MiniMax-M2.5 4bit long decode
49.9 tok/s (long-context decode, per-request; token count not printed)MiniMax-M2.5 (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-minimax-m25-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Mistral-Small-3.2 4bit cold TTFT
1100 ms (cold time to first token, one request)Mistral-Small-3.2 (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-mistral-small-32-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Mistral-Small-3.2 4bit long decode
47.2 tok/s (long-context decode, per-request; token count not printed)Mistral-Small-3.2 (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-mistral-small-32-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3-0.6B 4bit cold TTFT
94 ms (cold time to first token, one request)Qwen3-0.6B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen3-06b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3-0.6B 4bit long decode
364.7 tok/s (long-context decode, per-request; token count not printed)Qwen3-0.6B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen3-06b-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3-Coder-Next 4bit cold TTFT
473 ms (cold time to first token, one request)Qwen3-Coder-Next (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen3-coder-next-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3-Coder-Next 4bit long decode
73.5 tok/s (long-context decode, per-request; token count not printed)Qwen3-Coder-Next (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen3-coder-next-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3-Coder-Next 6bit cold TTFT
642 ms (cold time to first token, one request)Qwen3-Coder-Next (6bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen3-coder-next-6bit-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3-Coder-Next 6bit long decode
65.6 tok/s (long-context decode, per-request; token count not printed)Qwen3-Coder-Next (6bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen3-coder-next-6bit-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-122B-A10B 8bit cold TTFT
1300 ms (cold time to first token, one request)Qwen3.5-122B-A10B (8bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-122b-a10b-8bit-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-122B-A10B 8bit long decode
42.7 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-122B-A10B (8bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-122b-a10b-8bit-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-122B-A10B mxfp4 cold TTFT
714 ms (cold time to first token, one request)Qwen3.5-122B-A10B (mxfp4, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-122b-a10b-mxfp4-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-122B-A10B mxfp4 long decode
57.0 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-122B-A10B (mxfp4, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-122b-a10b-mxfp4-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-27B 4bit cold TTFT
453 ms (cold time to first token, one request)Qwen3.5-27B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-27b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-27B 4bit long decode
37.7 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-27B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-27b-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-35B-A3B 4bit cold TTFT
322 ms (cold time to first token, one request)Qwen3.5-35B-A3B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-35b-a3b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-35B-A3B 4bit long decode
95.2 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-35B-A3B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-35b-a3b-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-35B-A3B 8bit cold TTFT
456 ms (cold time to first token, one request)Qwen3.5-35B-A3B (8bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-35b-a3b-8bit-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-35B-A3B 8bit long decode
80.0 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-35B-A3B (8bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-35b-a3b-8bit-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-4B 4bit cold TTFT
196 ms (cold time to first token, one request)Qwen3.5-4B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-4b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-4B 4bit long decode
157.6 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-4B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-4b-long-decode
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-9B 4bit cold TTFT
228 ms (cold time to first token, one request)Qwen3.5-9B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-9b-cold-ttft
verifiedlow · 0.55
M3 Ultra 256GB rapid-mlx: Qwen3.5-9B 4bit long decode
106.4 tok/s (long-context decode, per-request; token count not printed)Qwen3.5-9B (4bit, MLX) on apple-m3-ultra via mlx-lmbenchmarks/rapidmlx-m3ultra256-qwen35-9b-long-decode
verifiedlow · 0.55
M4 Max 128GB llama.cpp b8353 Qwen3.5-27B Q4_K_M pp512 depth 0
612.4 tok/sQwen3.5-27B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-27b-q4km-d0-pp512
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-27B Q4_K_M pp512 depth 8192
498.77 tok/sQwen3.5-27B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-27b-q4km-d8192-pp512
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-27B Q4_K_M tg128 depth 0
24.18 tok/sQwen3.5-27B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-27b-q4km-d0-tg128
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-27B Q4_K_M tg128 depth 8192
22.64 tok/sQwen3.5-27B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-27b-q4km-d8192-tg128
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-35B-A3B Q4_K_M pp512 depth 0
1125.86 tok/sQwen3.5-35B-A3B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-35b-a3b-q4km-d0-pp512
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-35B-A3B Q4_K_M pp512 depth 8192
903.14 tok/sQwen3.5-35B-A3B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-35b-a3b-q4km-d8192-pp512
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-35B-A3B Q4_K_M tg128 depth 0
91.47 tok/sQwen3.5-35B-A3B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-35b-a3b-q4km-d0-tg128
verifiedlow · 0.60
M4 Max 128GB llama.cpp b8353 Qwen3.5-35B-A3B Q4_K_M tg128 depth 8192
82.31 tok/sQwen3.5-35B-A3B Q4_K_M (unsloth) on apple-m4-max via llama-cppbenchmarks/llamacpp-m4max128-qwen35-35b-a3b-q4km-d8192-tg128
verifiedlow · 0.60
M5 Max 128GB mlx_lm: gpt-oss-120b-MXFP4-Q8 decode at 16k (128 generated tokens)
75.963 tok/s (single-request decode, 128 generated tokens, batch 1)gpt-oss-120b-MXFP4-Q8 on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-gptoss-120b-mxfp4-q8-16k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: gpt-oss-120b-MXFP4-Q8 decode at 32k (128 generated tokens)
64.469 tok/s (single-request decode, 128 generated tokens, batch 1)gpt-oss-120b-MXFP4-Q8 on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-gptoss-120b-mxfp4-q8-32k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: gpt-oss-120b-MXFP4-Q8 decode at 4k (128 generated tokens)
87.873 tok/s (single-request decode, 128 generated tokens, batch 1)gpt-oss-120b-MXFP4-Q8 on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-gptoss-120b-mxfp4-q8-4k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: gpt-oss-120b-MXFP4-Q8 prefill at 16k (16452 prompt tokens)
2710.46 tok/s (single-request prefill, 16452 prompt tokens)gpt-oss-120b-MXFP4-Q8 on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-gptoss-120b-mxfp4-q8-16k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: gpt-oss-120b-MXFP4-Q8 prefill at 32k (32836 prompt tokens)
2537.42 tok/s (single-request prefill, 32836 prompt tokens)gpt-oss-120b-MXFP4-Q8 on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-gptoss-120b-mxfp4-q8-32k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: gpt-oss-120b-MXFP4-Q8 prefill at 4k (4164 prompt tokens)
1325.062 tok/s (single-request prefill, 4164 prompt tokens)gpt-oss-120b-MXFP4-Q8 on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-gptoss-120b-mxfp4-q8-4k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3-Coder-Next-8bit decode at 16k (60 generated tokens)
74.293 tok/s (single-request decode, 60 generated tokens, batch 1)Qwen3-Coder-Next-8bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen3-coder-next-8bit-16k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3-Coder-Next-8bit decode at 32k (58 generated tokens)
68.624 tok/s (single-request decode, 58 generated tokens, batch 1)Qwen3-Coder-Next-8bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen3-coder-next-8bit-32k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3-Coder-Next-8bit decode at 64k (61 generated tokens)
48.212 tok/s (single-request decode, 61 generated tokens, batch 1)Qwen3-Coder-Next-8bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen3-coder-next-8bit-64k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3-Coder-Next-8bit prefill at 16k (16393 prompt tokens)
1802.144 tok/s (single-request prefill, 16393 prompt tokens)Qwen3-Coder-Next-8bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen3-coder-next-8bit-16k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3-Coder-Next-8bit prefill at 32k (32777 prompt tokens)
1887.158 tok/s (single-request prefill, 32777 prompt tokens)Qwen3-Coder-Next-8bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen3-coder-next-8bit-32k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3-Coder-Next-8bit prefill at 64k (65545 prompt tokens)
1432.73 tok/s (single-request prefill, 65545 prompt tokens)Qwen3-Coder-Next-8bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen3-coder-next-8bit-64k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-122B-A10B-4bit decode at 16k (128 generated tokens)
60.639 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3.5-122B-A10B-4bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-122b-4bit-16k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-122B-A10B-4bit decode at 32k (128 generated tokens)
54.923 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3.5-122B-A10B-4bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-122b-4bit-32k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-122B-A10B-4bit prefill at 16k (16394 prompt tokens)
1239.734 tok/s (single-request prefill, 16394 prompt tokens)Qwen3.5-122B-A10B-4bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-122b-4bit-16k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-122B-A10B-4bit prefill at 32k (32778 prompt tokens)
1067.824 tok/s (single-request prefill, 32778 prompt tokens)Qwen3.5-122B-A10B-4bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-122b-4bit-32k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit decode at 16k (128 generated tokens)
20.311 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-16k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit decode at 32k (128 generated tokens)
14.908 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-32k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit decode at 4k (128 generated tokens)
23.648 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-4k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit decode at 64k (128 generated tokens)
14.225 tok/s (single-request decode, 128 generated tokens, batch 1)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-64k-tg
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit prefill at 16k (16395 prompt tokens)
686.682 tok/s (single-request prefill, 16395 prompt tokens)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-16k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit prefill at 32k (32779 prompt tokens)
591.383 tok/s (single-request prefill, 32779 prompt tokens)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-32k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit prefill at 4k (4107 prompt tokens)
811.134 tok/s (single-request prefill, 4107 prompt tokens)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-4k-pp
verifiedlow · 0.55
M5 Max 128GB mlx_lm: Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit prefill at 64k (65547 prompt tokens)
475.828 tok/s (single-request prefill, 65547 prompt tokens)Qwen3.5-27B-Claude-4.6-Opus-Distilled-MLX-6bit on local-apple-m5-max via mlx-lmbenchmarks/m5max128-mlx-qwen35-27b-6bit-64k-pp
verifiedlow · 0.55
Mac Studio M2 Ultra Qwen3-235B-A22B 4-bit decode, MLX
26.4 tok/sQwen3-235B-A22B on apple-m2-ultra via apple-mlxbenchmarks/comm-apple-m2-ultra-qwen3-235b-a22b-mlx-tg
verifiedmedium · 0.70
MacBook Pro M4 Max Llama-3.3-70B 8-bit decode (Silicon Score)
6.5 tok/sLlama-3.3-70B (dense) 8-bit on apple-m4-max via llama-cppbenchmarks/comm-apple-m4-max-llama33-70b-8bit-decode
verifiedmedium · 0.70
MangoBoost on 4-node MI300X with LLMboost: highest recorded MLPerf Llama 2 70B Offline result (~103K tok/s)
103000 output tokens/s (aggregate; source states 'around 103K tokens/sec')Llama 2 70B (MLPerf reference, W8A8 per AMD's Quark recipe) on amd-instinct-mi300x via vllmbenchmarks/mi325x-mlperf-v5-0-mangoboost-llama2-70b-offline
verifiedmedium · 0.70
Medusa acceptance length and speedup on Vicuna-13B (temperature=0)
2.44 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-medusa-acceptance-vicuna13b-t0
verifiedhigh · 0.90
Medusa speedup degradation beyond batch size 32
0 value 0 with metric speedup_ratio: 0 is the FLOOR of the paper's documented degradation curve (the speedup decays toward zero and can go negative past batch 32), not a measured decode rate of zero tokens/s. BASELINE: vanilla autoregressive decoding, vLLM, A100 80GB PCIe, seqlen 1024, per the Medusa paper Appendix G.3.Llama-7B on nvidia-a100-80gb-pcie via vllmbenchmarks/sd-medusa-batch32-degradation
verifiedhigh · 0.85
MI300A x1 node (4 APU): KafkaLM-15B, vLLM batch inference, 812 tok/s aggregate (CUG 2025)
812 tok/s aggregate across 100-prompt batch (whole node)Seedbox/KafkaLM-15B-Base (15B, 2:4-sparse distilled from Mistral-Small-24B) on amd-instinct-mi300a via vllmbenchmarks/cug2025-mi300a-vllm-kafka15b-tok-s
verifiedmedium · 0.72
MI300A x1 node (4 APU): KafkaLM-15B, vLLM batch inference, TTFT 2.46 s (CUG 2025)
2460 ms time-to-first-token (mean over 100 prompts, whole node)Seedbox/KafkaLM-15B-Base (15B, 2:4-sparse distilled from Mistral-Small-24B) on amd-instinct-mi300a via vllmbenchmarks/cug2025-mi300a-vllm-kafka15b-ttft-ms
verifiedmedium · 0.72
MI300A x1 node (4 APU): Mistral-Small-24B-Base-2501, vLLM batch inference, 579 tok/s aggregate (CUG 2025)
579 tok/s aggregate across 100-prompt batch (whole node)mistralai/Mistral-Small-24B-Base-2501 (24B dense teacher) on amd-instinct-mi300a via vllmbenchmarks/cug2025-mi300a-vllm-mistral-small-24b-tok-s
verifiedmedium · 0.72
MI300A x1 node (4 APU): Mistral-Small-24B-Base-2501, vLLM batch inference, TTFT 4.91 s (CUG 2025)
4910 ms time-to-first-token (mean over 100 prompts, whole node)mistralai/Mistral-Small-24B-Base-2501 (24B dense teacher) on amd-instinct-mi300a via vllmbenchmarks/cug2025-mi300a-vllm-mistral-small-24b-ttft-ms
verifiedmedium · 0.72
MI300X vs H100, vLLM, Llama 3.1 405B FP8 TP8: output token throughput at 128-in/4096-out (MI300X wins 1.62x)
3171 output tok/s (aggregate over all in-flight requests)Llama 3.1 405B on amd-instinct-mi300x, nvidia-h100-sxm via vllmbenchmarks/mi300x-vs-h100-vllm-llama31-405b-fp8-tp8-output-throughput
verifiedhigh · 0.85
MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8 batch 1: end-to-end time-to-last-token at 128-in/2048-out
19089 ms (time to last token, end-to-end, batch 1, per request)Llama 3.1 70B on amd-instinct-mi300x, nvidia-h100-sxm via vllmbenchmarks/mi300x-vllm-llama31-70b-fp8-tp8-ttlt-batch1
verifiedhigh · 0.85
MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out
15105 output tok/s (aggregate over all in-flight requests)Llama 3.1 70B on amd-instinct-mi300x, nvidia-h100-sxm via vllmbenchmarks/mi300x-vs-h100-vllm-llama31-70b-fp8-tp8-output-throughput
verifiedhigh · 0.85
MI355X, MLPerf Inference v6.0 Offline multi-node: Llama 2 70B WMXFP4 on 87xMI355X across 11 nodes
1042110 output tokens/s (aggregate, MLPerf 'Tokens per second', 1.04211e+06)Llama 2 70B (WMXFP4 weights, FP8 KV cache, UINT8 scales) on amd-instinct-mi355x via vllmbenchmarks/mi355x-mlperf-v6-0-llama2-70b-87gpu-offline-tokens
verifiedhigh · 0.85
MI355X, MLPerf Inference v6.0 Offline: gpt-oss-120b (MXFP4/FP8 MoE) on 8xMI355X, aggregate output throughput
95004 output tokens/s (aggregate, MLPerf 'Tokens per second')gpt-oss-120b (MoE, WMXFP4 weights, FP8-e4m3 activations) on amd-instinct-mi355x via vllmbenchmarks/mi355x-mlperf-v6-0-gpt-oss-120b-offline-tokens
verifiedhigh · 0.95
MI355X, MLPerf Inference v6.0 Offline: Llama 2 70B WMXFP4 on 8xMI355X, aggregate output throughput
103480 output tokens/s (aggregate, MLPerf 'Tokens per second')Llama 2 70B (WMXFP4 weights, FP8 KV cache, UINT8 scales) on amd-instinct-mi355x via vllmbenchmarks/mi355x-mlperf-v6-0-llama2-70b-wmxfp4-offline-tokens
verifiedhigh · 0.95
MiniMax-M3-AWQ-INT4 with an FP8 E5M2 KV cache on 8xA800: GPQA Diamond measured against the same harness as the official API
0.909 accuracy (GPQA Diamond fraction correct)MiniMax-M3-AWQ-INT4 (weight-only int4) with --kv-cache-dtype fp8_e5m2 on nvidia-a100-80gb-sxm4 via vllmbenchmarks/minimax-m3-fp8kv-sm80-gpqa-diamond-vs-official-api
verifiedmedium · 0.80
MLPerf v3.0: Dell r7515, 4x Qualcomm Cloud AI 100 Pro, BERT-99 Offline 3059.8 samples/s
3059.8 samples/s aggregate, 4-card node, Offline scenariobert-large-squad (bert-99) on qualcomm-ai100 via qualcomm-ai-engine-direct-apps-sdk-aic100-backendbenchmarks/qaic-ai100-4x-mlperf-v3-0-bert99-offline-samples
verifiedhigh · 0.85
MLPerf v3.0: Dell r7515, 4x Qualcomm Cloud AI 100 Pro, BERT-99 Server 2724.39 samples/s
2724.39 completed samples/s, 4-card node, Server scenario under latency SLObert-large-squad (bert-99) on qualcomm-ai100 via qualcomm-ai-engine-direct-apps-sdk-aic100-backendbenchmarks/qaic-ai100-4x-mlperf-v3-0-bert99-server-samples
verifiedhigh · 0.85
MLPerf v3.0: Dell r7515, 4x Qualcomm Cloud AI 100 Pro, BERT-99.9 Offline 1487.12 samples/s
1487.12 samples/s aggregate, 4-card node, Offline scenariobert-large-squad (bert-99.9, high-accuracy) on qualcomm-ai100 via qualcomm-ai-engine-direct-apps-sdk-aic100-backendbenchmarks/qaic-ai100-4x-mlperf-v3-0-bert999-offline-samples
verifiedhigh · 0.85
MLPerf v3.0: Dell r7515, 4x Qualcomm Cloud AI 100 Pro, BERT-99.9 Server 1372.9 samples/s
1372.9 completed samples/s, 4-card node, Server scenario under latency SLObert-large-squad (bert-99.9, high-accuracy) on qualcomm-ai100 via qualcomm-ai-engine-direct-apps-sdk-aic100-backendbenchmarks/qaic-ai100-4x-mlperf-v3-0-bert999-server-samples
verifiedhigh · 0.85
MLPerf v4.1 AMD 1x MI300X: llama2-70b-99.9 Offline aggregate tok/s
3062.72 tok/s (single MI300X 192GB)llama2-70b-99.9 on amd-instinct-mi300x via vllmbenchmarks/mlperf-v4-1-amd-1xmi300x-llama2-70b-offline-tokens
verifiedmedium · 0.80
MLPerf v5.0 Cisco X215 M8 2x L40S: gptj-99 Offline aggregate tokens/s
1754.97 tok/s AGGREGATE over 2x L40S 48GB, Offlinegptj-99 (GPT-J 6B, MLPerf closed) on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-cisco-x215m8-2xl40s-gptj99-offline-tokens
verifiedhigh · 0.85
MLPerf v5.0 Cisco X215 M8 2x L40S: gptj-99 Server aggregate tokens/s
1745.78 tok/s AGGREGATE over 2x L40S 48GB, Servergptj-99 (GPT-J 6B, MLPerf closed) on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-cisco-x215m8-2xl40s-gptj99-server-tokens
verifiedhigh · 0.85
MLPerf v5.0 Dell XE7745 8x H200 NVL 141GB: llama2-70b-99.9 Offline aggregate tok/s
31149.9 tok/s AGGREGATE over 8x H200 NVLllama2-70b-99.9 on nvidia-h200-nvl via tensorrt-llmbenchmarks/mlperf-v5-0-dell-xe7745-8xh200nvl-llama2-70b-offline-tokens
verifiedmedium · 0.80
MLPerf v5.0 Dell XE7745 8x L40S: llama2-70b-99.9 Offline aggregate tokens/s
3481.53 tok/s AGGREGATE over 8x L40S 48GBllama2-70b-99.9 on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-dell-xe7745-8xl40s-llama2-70b-offline-tokens
verifiedhigh · 0.85
MLPerf v5.0 Dell XE7745 8x L40S: llama2-70b-99.9 Server aggregate tokens/s
3201.13 tok/s AGGREGATE over 8x L40S 48GBllama2-70b-99.9 on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-dell-xe7745-8xl40s-llama2-70b-server-tokens
verifiedhigh · 0.85
MLPerf v5.0 Fujitsu 8x NVIDIA H100-NVL 94GB: llama2-70b-99.9 Offline aggregate tok/s
18084.9 output tokens/s (aggregate over 8 accelerators, Offline)llama2-70b-99.9 on nvidia-h100-nvl via tensorrt-llmbenchmarks/mlperf-v5-0-fujitsu-8xh100nvl-llama2-70b-llama2-70b-99-9-offline
verifiedhigh · 0.85
MLPerf v5.0 Fujitsu 8x NVIDIA H100-NVL 94GB: mixtral-8x7b Offline aggregate tok/s
37713.6 output tokens/s (aggregate over 8 accelerators, Offline)mixtral-8x7b on nvidia-h100-nvl via tensorrt-llmbenchmarks/mlperf-v5-0-fujitsu-8xh100nvl-mixtral-mixtral-8x7b-offline
verifiedhigh · 0.85
MLPerf v5.0 HPE DL380a Gen12 8x L40S: llama2-70b-99.9 Offline aggregate tokens/s
3655.89 tok/s AGGREGATE over 8x L40S 48GBllama2-70b-99.9 on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-offline-tokens
verifiedhigh · 0.85
MLPerf v5.0 HPE DL380a Gen12 8x L40S: llama2-70b-99.9 Server aggregate tokens/s
3181.65 tok/s AGGREGATE over 8x L40S 48GB, Server scenariollama2-70b-99.9 on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-server-tokens
verifiedhigh · 0.85
MLPerf v5.0 HPE DL380a Gen12 8x L40S: llama2-70b-99.9 Server p99 first-token latency
525.5 ms, p99 first-token latency at an 11 qps target, 8x L40Sllama2-70b-99.9 on nvidia-l40s via tensorrt-llmbenchmarks/mlperf-v5-0-hpe-dl380a-8xl40s-llama2-70b-server-ttft-p99
verifiedhigh · 0.85
MLPerf v5.0 Jetson AGX Thor: llama2-70b-99 Offline 573.765 tok/s aggregate
573.765 tok/s AGGREGATE, single Thor module, Offline scenariollama2-70b-99 on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v5-0-thor-llama2-70b-offline-tokens
verifiedhigh · 0.85
MLPerf v5.0 Quanta_Cloud_Technology 4x NVIDIA H100-PCIe-80GB: llama2-70b-99.9 Offline aggregate tok/s
7013.17 output tokens/s (aggregate over 4 accelerators, Offline)llama2-70b-99.9 on nvidia-h100-pcie via tensorrt-llmbenchmarks/mlperf-v5-0-qct-4xh100pcie-llama2-70b-llama2-70b-99-9-offline
verifiedhigh · 0.85
MLPerf v5.0 Quanta_Cloud_Technology 4x NVIDIA H100-PCIe-80GB: mixtral-8x7b Offline aggregate tok/s
15843.3 output tokens/s (aggregate over 4 accelerators, Offline)mixtral-8x7b on nvidia-h100-pcie via tensorrt-llmbenchmarks/mlperf-v5-0-qct-4xh100pcie-mixtral-mixtral-8x7b-offline
verifiedhigh · 0.85
MLPerf v5.1 IBM Cloud 1x L40S vLLM: llama3.1-8b Server completed tokens/s
1207.14 tok/s (single L40S, Server scenario)llama3.1-8b on nvidia-l40s via vllmbenchmarks/mlperf-v5-1-ibm-1xl40s-vllm-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v5.1 IBM Cloud gx3 1x L40S vLLM: llama3.1-8b Offline aggregate tokens/s
1642.22 tok/s (single L40S 48GB, Offline)llama3.1-8b on nvidia-l40s via vllmbenchmarks/mlperf-v5-1-ibm-1xl40s-vllm-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v5.1 Jetson AGX Thor: llama3.1-8b-edge SingleStream TPOT p90 21.64 ms
21.641875 ms, 90th-percentile time per output token, per query (SingleStream, concurrency 1)llama3.1-8b-edge on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v5-1-thor-llama31-8b-edge-singlestream-itl
verifiedhigh · 0.85
MLPerf v5.1 Jetson AGX Thor: llama3.1-8b-edge SingleStream TTFT p90 1397 ms
1396.97016 ms, 90th-percentile first-token latency, per query (SingleStream, concurrency 1)llama3.1-8b-edge on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v5-1-thor-llama31-8b-edge-singlestream-ttft
verifiedhigh · 0.85
MLPerf v5.1 MiTAC 8x MI325X: llama2-70b-99.9 Offline aggregate tok/s
33703 tok/s AGGREGATE over 8x MI325X 256GBllama2-70b-99.9 on amd-instinct-mi325x via vllmbenchmarks/mlperf-v5-1-mitac-8xmi325x-llama2-70b-offline-tokens
verifiedmedium · 0.80
MLPerf v5.1 MiTAC 8x MI325X: mixtral-8x7b Offline aggregate tok/s
67964.4 tok/s AGGREGATE over 8x MI325Xmixtral-8x7b on amd-instinct-mi325x via vllmbenchmarks/mlperf-v5-1-mitac-8xmi325x-mixtral-8x7b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.0 Dell 8x NVIDIA H200-NVL-141GB: llama2-70b-99.9 Offline aggregate tok/s
32004.0 output tokens/s (aggregate over 8 accelerators, Offline)llama2-70b-99.9 on nvidia-h200-nvl via tensorrt-llmbenchmarks/mlperf-v6-0-dell-8xh200nvl-llama2-70b-llama2-70b-99-9-offline
verifiedhigh · 0.85
MLPerf v6.0 Dell 8x NVIDIA H200-NVL-141GB: llama2-70b-99.9 Server aggregate tok/s
29084.9 output tokens/s (aggregate over 8 accelerators, Server)llama2-70b-99.9 on nvidia-h200-nvl via tensorrt-llmbenchmarks/mlperf-v6-0-dell-8xh200nvl-llama2-70b-server-llama2-70b-99-9-ser
verifiedhigh · 0.85
MLPerf v6.0 Dell 8x NVIDIA H200-NVL-141GB: llama2-70b-99.9 Server aggregate tok/s
29084.9 output tokens/s (aggregate over 8 accelerators, Server)llama2-70b-99.9 on nvidia-h200-nvl via tensorrt-llmbenchmarks/mlperf-v6-0-dell-8xh200nvl-llama2-70b-server-llama2-70b-99-9-server
verifiedhigh · 0.85
MLPerf v6.0 Dell 8x NVIDIA H200-NVL-141GB: llama3.1-8b Offline aggregate tok/s
53193.6 output tokens/s (aggregate over 8 accelerators, Offline)llama3.1-8b on nvidia-h200-nvl via tensorrt-llmbenchmarks/mlperf-v6-0-dell-8xh200nvl-llama31-8b-llama3-1-8b-offline
verifiedhigh · 0.85
MLPerf v6.0 HPE 1x NVIDIA RTX PRO 4500 Blackwell 32GB: llama3.1-8b Offline aggregate tok/s
2340.75 output tokens/s (aggregate over 1 accelerators, Offline)llama3.1-8b on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-0-hpe-1xrtxpro4500-llama31-8b-llama3-1-8b-offline
verifiedhigh · 0.85
MLPerf v6.0 HPE 5x NVIDIA RTX PRO 4500 Blackwell 32GB: llama3.1-8b Offline aggregate tok/s
11665.0 output tokens/s (aggregate over 5 accelerators, Offline)llama3.1-8b on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-0-hpe-5xrtxpro4500-llama31-8b-llama3-1-8b-offline
verifiedhigh · 0.85
MLPerf v6.0 Intel 4x Intel Arc Pro B60: gpt-oss-120b Offline aggregate tok/s
841.038 output tokens/s (aggregate over 4 accelerators, Offline)gpt-oss-120b on intel-arc-pro-b60 via vllmbenchmarks/mlperf-v6-0-intel-4xarcb60-gptoss-120b-gpt-oss-120b-offline
verifiedhigh · 0.85
MLPerf v6.0 Intel 4x Intel Arc Pro B60: llama2-70b-99.9 Offline aggregate tok/s
1697.66 output tokens/s (aggregate over 4 accelerators, Offline)llama2-70b-99.9 on intel-arc-pro-b60 via vllmbenchmarks/mlperf-v6-0-intel-4xarcb60-llama2-70b-llama2-70b-99-9-offline
verifiedhigh · 0.85
MLPerf v6.0 Intel 4x Intel Arc Pro B60: llama3.1-8b Offline aggregate tok/s
3347.43 output tokens/s (aggregate over 4 accelerators, Offline)llama3.1-8b on intel-arc-pro-b60 via vllmbenchmarks/mlperf-v6-0-intel-4xarcb60-llama31-8b-llama3-1-8b-offline
verifiedhigh · 0.85
MLPerf v6.0 Intel 4x Intel Arc Pro B70: gpt-oss-120b Offline aggregate tok/s
1536.9 output tokens/s (aggregate over 4 accelerators, Offline)gpt-oss-120b on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-0-intel-4xarcb70-gptoss-120b-gpt-oss-120b-offline
verifiedhigh · 0.85
MLPerf v6.0 Intel 4x Intel Arc Pro B70: llama2-70b-99.9 Offline aggregate tok/s
2459.18 output tokens/s (aggregate over 4 accelerators, Offline)llama2-70b-99.9 on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-0-intel-4xarcb70-llama2-70b-llama2-70b-99-9-offline
verifiedhigh · 0.85
MLPerf v6.0 Intel 4x Intel Arc Pro B70: llama3.1-8b Offline aggregate tok/s
4617.4 output tokens/s (aggregate over 4 accelerators, Offline)llama3.1-8b on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-0-intel-4xarcb70-llama31-8b-llama3-1-8b-offline
verifiedhigh · 0.85
MLPerf v6.0 Jetson AGX Thor: llama3.1-8b-edge Offline 350.605 tok/s aggregate
350.605 tok/s AGGREGATE, single Thor module, Offline scenariollama3.1-8b-edge on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v6-0-thor-llama31-8b-edge-offline-tokens
verifiedhigh · 0.85
MLPerf v6.0 Jetson AGX Thor: llama3.1-8b-edge SingleStream TPOT p90 9.81 ms
9.810269 ms, 90th-percentile time per output token, per query (SingleStream, concurrency 1)llama3.1-8b-edge on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v6-0-thor-llama31-8b-edge-singlestream-itl
verifiedhigh · 0.85
MLPerf v6.0 Jetson AGX Thor: llama3.1-8b-edge SingleStream TTFT p90 126.5 ms
126.477827 ms, 90th-percentile first-token latency, per query (SingleStream, concurrency 1)llama3.1-8b-edge on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v6-0-thor-llama31-8b-edge-singlestream-ttft
verifiedhigh · 0.85
MLPerf v6.0 MangoBoost 8x AMD Instinct MI325X 256GB HBM3e: llama2-70b-99.9 Offline aggregate tok/s
33880.4 output tokens/s (aggregate over 8 accelerators, Offline)llama2-70b-99.9 on amd-instinct-mi325x via vllmbenchmarks/mlperf-v6-0-mangoboost-8xmi325x-llama2-70b-llama2-70b-99-9-offli
verifiedhigh · 0.85
MLPerf v6.0 MangoBoost 8x AMD Instinct MI325X 256GB HBM3e: llama2-70b-99.9 Offline aggregate tok/s
33880.4 output tokens/s (aggregate over 8 accelerators, Offline)llama2-70b-99.9 on amd-instinct-mi325x via vllmbenchmarks/mlperf-v6-0-mangoboost-8xmi325x-llama2-70b-llama2-70b-99-9-offline
verifiedhigh · 0.85
MLPerf v6.0 MangoBoost 8x AMD Instinct MI325X 256GB HBM3e: llama2-70b-99.9 Server aggregate tok/s
31888.9 output tokens/s (aggregate over 8 accelerators, Server)llama2-70b-99.9 on amd-instinct-mi325x via vllmbenchmarks/mlperf-v6-0-mangoboost-8xmi325x-llama2-70b-server-llama2-70b-99
verifiedhigh · 0.85
MLPerf v6.0 MangoBoost 8x AMD Instinct MI325X 256GB HBM3e: llama2-70b-99.9 Server aggregate tok/s
31888.9 output tokens/s (aggregate over 8 accelerators, Server)llama2-70b-99.9 on amd-instinct-mi325x via vllmbenchmarks/mlperf-v6-0-mangoboost-8xmi325x-llama2-70b-server-llama2-70b-99-9-server
verifiedhigh · 0.85
MLPerf v6.1 Atlas_Inference Strix Halo (8060S): Qwen3.6-27B NVFP4 SingleStream median TPOT
47.28 ms median TPOT (agentic SingleStream)Qwen3.6-27B-NVFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/mlperf-v6-1-atlas-strix-halo-qwen36-27b-tpot
verifiedmedium · 0.75
MLPerf v6.1 Atlas_Inference Strix Halo: Qwen3.6-27B NVFP4 SingleStream median TTFT
2713.5 ms median TTFTQwen3.6-27B-NVFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/mlperf-v6-1-atlas-strix-halo-qwen36-27b-ttft
verifiedmedium · 0.75
MLPerf v6.1 Cisco 4x Xeon 6788P: llama3.1-8b INT8 Offline aggregate tok/s
2068.63 tok/s AGGREGATE over 4x Xeon 6788Pllama3.1-8b on intel-xeon-6788p-granite-rapids via vllmbenchmarks/mlperf-v6-1-cisco-4x6788p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Cisco 4x Xeon 6788P: llama3.1-8b INT8 Server aggregate tok/s
1030.45 tok/s AGGREGATE over 4x Xeon 6788P (Server scenario)llama3.1-8b on intel-xeon-6788p-granite-rapids via vllmbenchmarks/mlperf-v6-1-cisco-4x6788p-llama31-8b-int8-server
verifiedmedium · 0.80
MLPerf v6.1 Cisco C240 M8 5x RTX PRO 4500 32GB: llama3.1-8b Interactive 8494.68 tok/s
8494.68 tok/s aggregate InteractiveLlama-3.1-8B on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-5xrtxpro4500-llama31-8b-interactive-tokens
verifiedhigh · 0.85
MLPerf v6.1 Cisco C240 M8 5x RTX PRO 4500 32GB: llama3.1-8b Server 11426.1 tok/s
11426.1 tok/s aggregate ServerLlama-3.1-8B on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-5xrtxpro4500-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Cisco C240M8 5x RTX PRO 4500 32GB: llama3.1-8b Offline aggregate tok/s
12017.7 tok/s AGGREGATE over 5x RTX PRO 4500llama3.1-8b on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-5xrtxpro4500-llama31-8b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 Cisco C845A M8 8x RTX PRO 4500 32GB: llama3.1-8b Offline 18800.9 tok/s
18800.9 tok/s aggregate OfflineLlama-3.1-8B on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-8xrtxpro4500-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Cisco C845A M8 8x RTX PRO 4500 32GB: llama3.1-8b Server 18186.2 tok/s
18186.2 tok/s aggregate ServerLlama-3.1-8B on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-8xrtxpro4500-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Cisco X210 M8 X580P 4x RTX PRO 4500 32GB: llama3.1-8b Offline 9518.66 tok/s
9518.66 tok/s aggregate OfflineLlama-3.1-8B on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-4xrtxpro4500-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Cisco X210 M8 X580P 4x RTX PRO 4500 32GB: llama3.1-8b Server 9277.73 tok/s
9277.73 tok/s aggregate ServerLlama-3.1-8B on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-cisco-4xrtxpro4500-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell 2x Xeon 6980P: whisper INT8 Offline aggregate samples/s
2275.93 samples/s AGGREGATE (whisper Offline; NOT tok/s)whisper on intel-xeon-6980p-granite-rapids via vllmbenchmarks/mlperf-v6-1-dell-2x6980p-whisper-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Dell R770 2x Xeon 6787P: llama3.1-8b INT8 Offline aggregate tok/s
1227.99 tok/s AGGREGATE over 2x Xeon 6787P (86c, 8-channel)llama3.1-8b on intel-xeon-6787p-granite-rapids via vllmbenchmarks/mlperf-v6-1-dell-2x6787p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Dell R770 2x Xeon 6787P: llama3.1-8b INT8 Server aggregate tok/s
516.012 tok/s AGGREGATE over 2x Xeon 6787P (Server scenario)llama3.1-8b on intel-xeon-6787p-granite-rapids via vllmbenchmarks/mlperf-v6-1-dell-2x6787p-llama31-8b-int8-server
verifiedmedium · 0.80
MLPerf v6.1 Dell XE7745 8x MI350P: gpt-oss-120b Offline aggregate tok/s
48971.1 tok/s AGGREGATE over 8x MI350Pgpt-oss-120b on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-gpt-oss-120b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 Dell XE7745 8x MI350P: gpt-oss-120b Server 40867.9 tok/s
40867.9 tok/s aggregate Servergpt-oss-120b on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-gptoss-120b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell XE7745 8x MI350P: llama2-70b-99.9 Interactive 22860.5 tok/s
22860.5 tok/s aggregate InteractiveLlama-2-70B (99.9 acc) on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-llama2-70b-interactive-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell XE7745 8x MI350P: llama2-70b-99.9 Offline aggregate tok/s
42286.2 tok/s AGGREGATE over 8x MI350Pllama2-70b-99.9 on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-llama2-70b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 Dell XE7745 8x MI350P: llama2-70b-99.9 Server 41830 tok/s
41830.0 tok/s aggregate ServerLlama-2-70B (99.9 acc) on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-llama2-70b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell XE7745 8x MI350P: llama3.1-8b Interactive 62723.7 tok/s
62723.7 tok/s aggregate InteractiveLlama-3.1-8B on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-llama31-8b-interactive-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell XE7745 8x MI350P: llama3.1-8b Offline aggregate tok/s
73643 tok/s AGGREGATE over 8x MI350Pllama3.1-8b on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-llama31-8b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 Dell XE7745 8x MI350P: llama3.1-8b Server 72305.1 tok/s
72305.1 tok/s aggregate ServerLlama-3.1-8B on amd-instinct-mi350p via vllmbenchmarks/mlperf-v6-1-dell-8xmi350p-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell XE7745 8x RTX PRO 4500 32GB: llama2-70b-99.9 Offline aggregate tok/s
4374.06 tok/s AGGREGATE over 8x RTX PRO 4500llama2-70b-99.9 on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-dell-8xrtxpro4500-llama2-70b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 Dell XE7745 8x RTX PRO 4500: llama2-70b-99.9 Offline 4374.06 tok/s (8B row n/a; 70B only)
4374.06 tok/s aggregate OfflineLlama-2-70B (99.9 acc) on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-dell-8xrtxpro4500-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Dell XE7745 8x RTX PRO 4500: llama2-70b-99.9 Server aggregate tok/s
3012.33 tok/s AGGREGATE over 8x RTX PRO 4500llama2-70b-99.9 on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-dell-8xrtxpro4500-llama2-70b-server-tokens
verifiedmedium · 0.80
MLPerf v6.1 edge OPEN: HPE DL345 4x RTX PRO 4500 PCIe 32GB llama3.1-8b-edge SingleStream 1203.79 ms
1203.785699 ms SingleStream latency for the agentic-edge benchmark (lower is better; scenario-completion latency)MLPerf 'agentic edge' workload on nvidia-rtx-pro-4500-blackwell via tensorrt-llmbenchmarks/mlperf-v6-1-edgeopen-rtxpro4500-4x-llama31-8b-singlestream-lat
verifiedmedium · 0.70
MLPerf v6.1 edge: AMD Strix Halo (Ryzen AI Max+ 395 / 8060S) agentic-edge SingleStream 7058.99 ms
7058.989199685204 ms SingleStream latency for the agentic-edge benchmark (lower is better; scenario-completion latency)MLPerf 'agentic edge' workload on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via tensorrt-llmbenchmarks/mlperf-v6-1-edge-strix-halo-8060s-agentic-singlestream-lat
verifiedmedium · 0.70
MLPerf v6.1 edge: HPE DL145 1x RTX PRO 4500 SE 32GB agentic-edge SingleStream 2757.95 ms
2757.954404037736 ms SingleStream latency for the agentic-edge benchmark (lower is better; scenario-completion latency)MLPerf 'agentic edge' workload on nvidia-rtx-pro-4500-blackwell via llama-cppbenchmarks/mlperf-v6-1-edge-rtxpro4500-1x-agentic-singlestream-lat
verifiedmedium · 0.70
MLPerf v6.1 edge: HPE DL345 4x RTX PRO 4500 SE 32GB agentic-edge SingleStream 2616.16 ms
2616.15900548858 ms SingleStream latency for the agentic-edge benchmark (lower is better; scenario-completion latency)MLPerf 'agentic edge' workload on nvidia-rtx-pro-4500-blackwell via llama-cppbenchmarks/mlperf-v6-1-edge-rtxpro4500-4x-agentic-singlestream-lat
verifiedmedium · 0.70
MLPerf v6.1 edge: NVIDIA DGX Spark (GB10) agentic-edge SingleStream 3807.66 ms
3807.66351136147 ms SingleStream latency for the agentic-edge benchmark (lower is better; scenario-completion latency)MLPerf 'agentic edge' workload on nvidia-gb10 via tensorrt-llmbenchmarks/mlperf-v6-1-edge-dgx-spark-gb10-agentic-singlestream-lat
verifiedmedium · 0.70
MLPerf v6.1 edge: NVIDIA Jetson AGX Thor 128GB agentic-edge SingleStream 1466.01 ms
1466.007675716981 ms SingleStream latency for the agentic-edge benchmark (lower is better; scenario-completion latency)MLPerf 'agentic edge' workload on nvidia-jetson-agx-thor via tensorrt-llmbenchmarks/mlperf-v6-1-edge-jetson-thor-agentic-singlestream-lat
verifiedmedium · 0.70
MLPerf v6.1 Intel 1-node-4x-BMG-B70: gpt-oss-120b Offline 1956.4 tok/s
1956.4 tok/s aggregate Offlinegpt-oss-120b on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-intel-4xarcb70-gptoss-120b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Intel 1-node-4x-BMG-B70: gpt-oss-120b Server 1296.87 tok/s
1296.87 tok/s aggregate Servergpt-oss-120b on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-intel-4xarcb70-gptoss-120b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Intel 1-node-4x-BMG-B70: llama2-70b-99.9 Server 1745.48 tok/s
1745.48 tok/s aggregate ServerLlama-2-70B (99.9 acc) on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-intel-4xarcb70-llama2-70b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Intel 1-node-4x-BMG-B70: llama3.1-8b Offline 4613.28 tok/s
4613.28 tok/s aggregate OfflineLlama-3.1-8B on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-intel-4xarcb70-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Intel 1-node-4x-BMG-B70: llama3.1-8b Server 4173.25 tok/s
4173.25 tok/s aggregate ServerLlama-3.1-8B on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-intel-4xarcb70-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Intel 2x Xeon 6980P: llama3.1-8b INT8 Offline aggregate tok/s
1916.74 tok/s AGGREGATE over 2x Xeon 6980P (128c, 12-channel)llama3.1-8b on intel-xeon-6980p-granite-rapids via vllmbenchmarks/mlperf-v6-1-intel-2x6980p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Intel 2x Xeon 6980P: llama3.1-8b INT8 Server aggregate tok/s
1139.51 tok/s AGGREGATE over 2x Xeon 6980P (Server scenario)llama3.1-8b on intel-xeon-6980p-granite-rapids via vllmbenchmarks/mlperf-v6-1-intel-2x6980p-llama31-8b-int8-server
verifiedmedium · 0.80
MLPerf v6.1 MangoBoost 8x MI300X (stock vLLM 0.24.0 ROCm): gpt-oss-120b Offline 30198.4 tok/s
30198.4 tok/s aggregate Offlinegpt-oss-120b on amd-instinct-mi300x via vllmbenchmarks/mlperf-v6-1-mangoboost-8xmi300x-gptoss-120b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 MangoBoost 8x MI300X (stock vLLM 0.24.0 ROCm): gpt-oss-120b Server 24863 tok/s
24863.0 tok/s aggregate Servergpt-oss-120b on amd-instinct-mi300x via vllmbenchmarks/mlperf-v6-1-mangoboost-8xmi300x-gptoss-120b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 MangoBoost heterogeneous 2x8 MI300X + 2x8 MI355X (LLMBoost P/D): gpt-oss-120b Offline 285454 tok/s
285454.0 tok/s aggregate Offlinegpt-oss-120b on amd-instinct-mi300x, amd-instinct-mi355x via vllmbenchmarks/mlperf-v6-1-mangoboost-het-mi300x-mi355x-gptoss-120b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 MangoBoost heterogeneous 2x8 MI300X + 2x8 MI355X (LLMBoost P/D): gpt-oss-120b Server 253501 tok/s
253501.0 tok/s aggregate Servergpt-oss-120b on amd-instinct-mi300x, amd-instinct-mi355x via vllmbenchmarks/mlperf-v6-1-mangoboost-het-mi300x-mi355x-gptoss-120b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 MiTAC 8x MI350X: gpt-oss-120b Offline aggregate tok/s
97019.7 tok/s AGGREGATE over 8x MI350Xgpt-oss-120b on amd-instinct-mi350x via vllmbenchmarks/mlperf-v6-1-mitac-8xmi350x-gpt-oss-120b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 MiTAC 8x MI350X: llama2-70b-99.9 Offline aggregate tok/s
84016.4 tok/s AGGREGATE over 8x MI350X 288GBllama2-70b-99.9 on amd-instinct-mi350x via vllmbenchmarks/mlperf-v6-1-mitac-8xmi350x-llama2-70b-offline-tokens
verifiedmedium · 0.80
MLPerf v6.1 Naeem Khoshnevis 1x H200-SXM-141GB (RDI): llama3.1-8b Offline 7796.59 tok/s
7796.59 tok/s aggregate OfflineLlama-3.1-8B FP8 on nvidia-h200-sxm via tensorrt-llmbenchmarks/mlperf-v6-1-1xh200-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Naeem Khoshnevis 1x H200-SXM-141GB (RDI): llama3.1-8b Server 7376.6 tok/s
7376.6 tok/s aggregate ServerLlama-3.1-8B FP8 on nvidia-h200-sxm via tensorrt-llmbenchmarks/mlperf-v6-1-1xh200-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Oracle 8x GB300 (Blackwell Ultra): llama2-70b-99.9 Offline aggregate tok/s
132253 tok/s AGGREGATE over 8x GB300 288GB (NVL72 partition)llama2-70b-99.9 on nvidia-gb300 via tensorrt-llmbenchmarks/mlperf-v6-1-oracle-8xgb300-llama2-70b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Oracle 8x GB300: llama2-70b-99.9 Server aggregate tok/s
127983.87 tok/s AGGREGATE over 8x GB300llama2-70b-99.9 on nvidia-gb300 via tensorrt-llmbenchmarks/mlperf-v6-1-oracle-8xgb300-llama2-70b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Quanta 1-node-4x-BMG-B70: llama3.1-8b Offline 4977.67 tok/s
4977.67 tok/s aggregate OfflineLlama-3.1-8B on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-quanta-4xarcb70-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Quanta 1-node-4x-BMG-B70: llama3.1-8b Server 4122.73 tok/s
4122.73 tok/s aggregate ServerLlama-3.1-8B on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-quanta-4xarcb70-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Quanta 2x Xeon 6788P: llama3.1-8b INT8 Offline aggregate tok/s
1170.51 tok/s AGGREGATE over 2x Xeon 6788P (86c, 8-channel)llama3.1-8b on intel-xeon-6788p-granite-rapids via vllmbenchmarks/mlperf-v6-1-qct-2x6788p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Quanta 2x Xeon 6788P: llama3.1-8b INT8 Server aggregate tok/s
516.012 tok/s AGGREGATE over 2x Xeon 6788P (Server scenario)llama3.1-8b on intel-xeon-6788p-granite-rapids via vllmbenchmarks/mlperf-v6-1-qct-2x6788p-llama31-8b-int8-server
verifiedmedium · 0.80
MLPerf v6.1 Red Hat 2x Xeon 6972P: llama3.1-8b INT8 Offline aggregate tok/s
1507.21 tok/s AGGREGATE over 2x Xeon 6972P (96c, 12-channel)llama3.1-8b on intel-xeon-6972p-granite-rapids via vllmbenchmarks/mlperf-v6-1-redhat-2x6972p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Red Hat 2x Xeon 6972P: llama3.1-8b INT8 Server aggregate tok/s
644.788 tok/s AGGREGATE over 2x Xeon 6972P (Server scenario)llama3.1-8b on intel-xeon-6972p-granite-rapids via vllmbenchmarks/mlperf-v6-1-redhat-2x6972p-llama31-8b-int8-server
verifiedmedium · 0.80
MLPerf v6.1 Supermicro 1-node-4x-BMG-B70: gpt-oss-120b Offline 2063.93 tok/s
2063.93 tok/s aggregate Offlinegpt-oss-120b on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-supermicro-4xarcb70-gptoss-120b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Supermicro 1-node-4x-BMG-B70: gpt-oss-120b Server 1351.13 tok/s
1351.13 tok/s aggregate Servergpt-oss-120b on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-supermicro-4xarcb70-gptoss-120b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Supermicro 1-node-4x-BMG-B70: llama2-70b-99.9 Offline 2484.14 tok/s
2484.14 tok/s aggregate OfflineLlama-2-70B (99.9 acc) on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-supermicro-4xarcb70-llama2-70b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Supermicro 1-node-4x-BMG-B70: llama2-70b-99.9 Server 1759.95 tok/s
1759.95 tok/s aggregate ServerLlama-2-70B (99.9 acc) on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-supermicro-4xarcb70-llama2-70b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Supermicro 1-node-4x-BMG-B70: llama3.1-8b Offline 4596.62 tok/s
4596.62 tok/s aggregate OfflineLlama-3.1-8B on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-supermicro-4xarcb70-llama31-8b-offline-tokens
verifiedhigh · 0.85
MLPerf v6.1 Supermicro 1-node-4x-BMG-B70: llama3.1-8b Server 4208.73 tok/s
4208.73 tok/s aggregate ServerLlama-3.1-8B on intel-arc-pro-b70 via vllmbenchmarks/mlperf-v6-1-supermicro-4xarcb70-llama31-8b-server-tokens
verifiedhigh · 0.85
MLPerf v6.1 Supermicro 2x Xeon 6787P: llama3.1-8b INT8 Offline aggregate tok/s
1189.89 tok/s AGGREGATE over 2x Xeon 6787Pllama3.1-8b on intel-xeon-6787p-granite-rapids via vllmbenchmarks/mlperf-v6-1-smci-2x6787p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Supermicro 2x Xeon 6980P: llama3.1-8b INT8 Offline aggregate tok/s
1833.97 tok/s AGGREGATE over 2x Xeon 6980Pllama3.1-8b on intel-xeon-6980p-granite-rapids via vllmbenchmarks/mlperf-v6-1-smci-2x6980p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Supermicro 2x Xeon 6980P: llama3.1-8b INT8 Server aggregate tok/s
1101.19 tok/s AGGREGATE over 2x Xeon 6980P (Server scenario)llama3.1-8b on intel-xeon-6980p-granite-rapids via vllmbenchmarks/mlperf-v6-1-smci-2x6980p-llama31-8b-int8-server
verifiedmedium · 0.80
MLPerf v6.1 Supermicro 4x Xeon 6788P: llama3.1-8b INT8 Offline aggregate tok/s
2063.24 tok/s AGGREGATE over 4x Xeon 6788Pllama3.1-8b on intel-xeon-6788p-granite-rapids via vllmbenchmarks/mlperf-v6-1-smci-4x6788p-llama31-8b-int8-offline
verifiedmedium · 0.80
MLPerf v6.1 Supermicro 4x Xeon 6788P: llama3.1-8b INT8 Server aggregate tok/s
966.241 tok/s AGGREGATE over 4x Xeon 6788P (Server scenario)llama3.1-8b on intel-xeon-6788p-granite-rapids via vllmbenchmarks/mlperf-v6-1-smci-4x6788p-llama31-8b-int8-server
verifiedmedium · 0.80
MLX-LM on 64GB Apple M4 Max: Qwen3-30B-A3B-Instruct-2507 4-bit generation throughput (128 tokens)
113.33 tok/s (generation of 128 tokens, per-request, batch 1)Qwen3-30B-A3B-Instruct-2507 (MLX 4-bit, mlx-community repo) on apple-m4-max via mlx-lmbenchmarks/mlx-m4-max-qwen3-30b-a3b-4bit-decode-tok-s
verifiedlow · 0.60
MLX-LM on 64GB Apple M4 Max: Qwen3-30B-A3B-Instruct-2507 4-bit prompt processing at 2048 tokens
1753.9 tok/s (prompt processing of a 2048-token prompt, batch 1)Qwen3-30B-A3B-Instruct-2507 (MLX 4-bit, mlx-community repo) on apple-m4-max via mlx-lmbenchmarks/mlx-m4-max-qwen3-30b-a3b-q4-prefill-tok-s
verifiedmedium · 0.80
MLX-LM on 64GB Apple M4 Max: Qwen3-30B-A3B-Instruct-2507 8-bit generation throughput (128 tokens)
83.16 tok/s (generation of 128 tokens, batch 1)Qwen3-30B-A3B-Instruct-2507 (MLX 8-bit, mlx-community repo) on apple-m4-max via mlx-lmbenchmarks/mlx-m4-max-qwen3-30b-a3b-q8-decode-tok-s
verifiedmedium · 0.80
MLX-LM on 64GB Apple M4 Max: Qwen3-4B-Instruct-2507 4-bit generation throughput (128 tokens)
134.52 tok/s (generation of 128 tokens, batch 1)Qwen3-4B-Instruct-2507 (MLX 4-bit, mlx-community repo) on apple-m4-max via mlx-lmbenchmarks/mlx-m4-max-qwen3-4b-q4-decode-tok-s
verifiedmedium · 0.80
MLX-LM on 64GB Apple M4 Max: Qwen3-4B-Instruct-2507 bf16 generation throughput (128 tokens)
52.47 tok/s (generation of 128 tokens, batch 1)Qwen3-4B-Instruct-2507 (bf16, original Qwen repo) on apple-m4-max via mlx-lmbenchmarks/mlx-m4-max-qwen3-4b-bf16-decode-tok-s
verifiedmedium · 0.80
MLX-LM on 64GB Apple M4 Max: Qwen3-4B-Instruct-2507 bf16 prompt processing at 2048 tokens
1780.63 tok/s (prompt processing of a 2048-token prompt, batch 1)Qwen3-4B-Instruct-2507 (bf16, original Qwen repo) on apple-m4-max via mlx-lmbenchmarks/mlx-m4-max-qwen3-4b-fp16-prefill-tok-s
verifiedlow · 0.60
Modeled US volume-server idle/maximum power dynamic range: 0.44 minimum by 2020 from measured SERT data, 0.28 in the Best Practices scenario, with an explicit ~20-25% idle-power latency floor
0.44 dimensionless ratio: idle power / maximum power. 0.44 is the MODELLED lower bound for volume servers by 2020, not a measurement of a specific part. Not an accuracy score and not a speedup.? on ? via ollamabenchmarks/us-volume-server-idle-dynamic-range-model-0-44-floor
verifiedmedium · 0.72
MXINT4-128 WikiText-2 perplexity delta vs fp16, Llama3.1-8B (AWQ+GPTQ)
0.6 perplexity_delta (WikiText-2-test ppl points, quantized minus FP16)Llama3.1-8B on ? via mxptq-research-harnessbenchmarks/mxint4-128-wikitext2-ppl-delta-llama3-1-8b-awq-gptq
verifiedhigh · 0.90
NVIDIA A100 80GB SXM, llama.cpp (RunPod): Llama-3-8B + 70B Q4_K_M decode tok/s (measured)
133.0 decode tok/s (8B Q4_K_M; 70B Q4_K_M 22-24 t/s)Llama-3-8B + Llama-3-70B Q4_K_M (GGUF) on nvidia-a100-80gb-sxm4 via llama-cppbenchmarks/comm-a100-80gb-sxm-llama3-8b-q4-decode
verifiedmedium · 0.82
NVIDIA A100 PCIe 80GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M pp1024 prefill tok/s
726.65 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 70B on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xd-nvidia-a100-80gb-pcie-70b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA A100 PCIe 80GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
22.11 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 70B on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xd-nvidia-a100-80gb-pcie-70b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA A100 PCIe 80GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
7504.24 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xd-nvidia-a100-80gb-pcie-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA A100 PCIe 80GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
54.56 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xd-nvidia-a100-80gb-pcie-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA A100 PCIe 80GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
5800.48 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xd-nvidia-a100-80gb-pcie-8b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA A100 PCIe 80GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
138.31 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-a100-80gb-pcie via llama-cppbenchmarks/xd-nvidia-a100-80gb-pcie-8b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA A40 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M pp1024 prefill tok/s
239.92 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 70B Q4_K_M on nvidia-a40 via llama-cppbenchmarks/xd-nvidia-a40-70b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA A40 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
12.08 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 70B Q4_K_M on nvidia-a40 via llama-cppbenchmarks/xd-nvidia-a40-70b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA A40 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
33.95 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-a40 via llama-cppbenchmarks/xd-nvidia-a40-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA A40 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
3240.95 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-a40 via llama-cppbenchmarks/xd-nvidia-a40-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA A40 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
88.95 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-a40 via llama-cppbenchmarks/xd-nvidia-a40-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA DGX B300, MLPerf Inference v6.1 Offline: Llama 2 70B on 8x B300-SXM-270GB, aggregate output throughput
109277 output tokens/s (aggregate, harness 'Tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-b300 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-nvidia-b300-llama2-70b-offline
verifiedhigh · 0.95
NVIDIA DGX B300, MLPerf Inference v6.1 Server: Llama 2 70B on 8x B300-SXM-270GB, aggregate output throughput under the Server latency constraint
106949.36 output tokens/s (aggregate, harness 'Completed tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-b300 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-nvidia-b300-llama2-70b-server
verifiedhigh · 0.95
NVIDIA GB300 NVL72 8-GPU slice (2 nodes), MLPerf Inference v6.1 Offline: Llama 2 70B - the per-GPU-efficient end of the same rack platform
131464 output tokens/s (aggregate across 8 GPUs on 2 nodes, harness 'Tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-b300 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-nvidia-gb300-8gpu-llama2-70b-offline
verifiedhigh · 0.95
NVIDIA GB300 NVL72 8-GPU slice (2 nodes), MLPerf Inference v6.1 Server: Llama 2 70B under the Server latency constraint
127911.34 output tokens/s (aggregate across 8 GPUs on 2 nodes, harness 'Completed tokens per second')Llama 2 70B (MLPerf v6.1 submission, FP4 weights) on nvidia-b300 via tensorrt-llmbenchmarks/bench-mlperf-v6-1-nvidia-gb300-8gpu-llama2-70b-server
verifiedhigh · 0.95
NVIDIA GH200, TensorRT-LLM: Llama-3.1-8B FP8 TP1 1024in/2048out aggregate output tok/s
16102.52 tok/s total output, aggregate (max-load)Llama-3.1-8B-Instruct-FP8 on nvidia-gh200 via tensorrt-llmbenchmarks/trtllm-gh200-llama31-8b-fp8-1024-2048-tps
verifiedmedium · 0.70
NVIDIA GH200, TensorRT-LLM: Llama-3.1-8B FP8 TP1 128in/128out aggregate output tok/s
27304.25 tok/s total output, aggregate (max-load)Llama-3.1-8B-Instruct-FP8 on nvidia-gh200 via tensorrt-llmbenchmarks/trtllm-gh200-llama31-8b-fp8-128-128-tps
verifiedmedium · 0.70
NVIDIA GH200, TensorRT-LLM: Llama-3.1-8B FP8 TP1 128in/2048out aggregate output tok/s
24045.6 tok/s total output, aggregate (max-load)Llama-3.1-8B-Instruct-FP8 on nvidia-gh200 via tensorrt-llmbenchmarks/trtllm-gh200-llama31-8b-fp8-128-2048-tps
verifiedmedium · 0.70
NVIDIA GH200, TensorRT-LLM: Llama-3.1-8B FP8 TP1 2048in/2048out aggregate output tok/s
10767.05 tok/s total output, aggregate (max-load)Llama-3.1-8B-Instruct-FP8 on nvidia-gh200 via tensorrt-llmbenchmarks/trtllm-gh200-llama31-8b-fp8-2048-2048-tps
verifiedmedium · 0.70
NVIDIA H100 NVL 94GB x2 (Cisco C245M8), MLPerf Offline (v5.0): Llama-2-70B aggregate tok/s
3741.32 output tok/s AGGREGATE across 2x H100-NVL-94GB (Offline (v5.0); ~1871 tok/s per card)Llama-2-70B (FP8) on nvidia-h100-nvl via tensorrt-llmbenchmarks/h100nvl-2x-mlperf-v5-0-llama2-70b-offline-tokens
verifiedmedium · 0.80
NVIDIA H100 NVL 94GB x2 (Cisco C245M8), MLPerf Server (v5.0): Llama-2-70B aggregate tok/s
3337.55 output tok/s AGGREGATE across 2x H100-NVL-94GB (Server (v5.0); ~1669 tok/s per card)Llama-2-70B (FP8) on nvidia-h100-nvl via tensorrt-llmbenchmarks/h100nvl-2x-mlperf-v5-0-llama2-70b-server-tokens
verifiedmedium · 0.80
NVIDIA H100 NVL 94GB x2 (Cisco X215M8), MLPerf Offline (v5.0): Llama-2-70B aggregate tok/s
3879.95 output tok/s AGGREGATE across 2x H100-NVL-94GB (Offline (v5.0); ~1940 tok/s per card)Llama-2-70B (FP8) on nvidia-h100-nvl via tensorrt-llmbenchmarks/h100nvl-2x-mlperf-v5-0-llama2-70b-offline-tokens-x215
verifiedmedium · 0.80
NVIDIA H100 NVL 94GB x4 (HPE DL380a), MLPerf Offline (v4.1, open): Mixtral-8x7B aggregate tok/s
18839 output tok/s AGGREGATE across 4x H100-NVL-94GB (Offline (v4.1, open); ~4710 tok/s per card)Mixtral-8x7B (FP8) on nvidia-h100-nvl via tensorrt-llmbenchmarks/h100nvl-4x-mlperf-v4-1-mixtral-offline-tokens
verifiedmedium · 0.80
NVIDIA H100 NVL 94GB x8 (Lenovo SR675 V3), MLPerf Offline (v4.1): GPT-J 6B aggregate tok/s
14073.6 output tok/s AGGREGATE across 8x H100-NVL-94GB (Offline (v4.1); ~1759 tok/s per card)GPT-J 6B (FP8) on nvidia-h100-nvl via tensorrt-llmbenchmarks/h100nvl-8x-mlperf-v4-1-gptj-offline-tokens
verifiedmedium · 0.80
NVIDIA H100 NVL 94GB x8 (Lenovo SR675 V3), MLPerf Offline (v4.1): Llama-2-70B aggregate tok/s
15875.8 output tok/s AGGREGATE across 8x H100-NVL-94GB (Offline (v4.1); ~1984 tok/s per card)Llama-2-70B (FP8) on nvidia-h100-nvl via tensorrt-llmbenchmarks/h100nvl-8x-mlperf-v4-1-llama2-70b-offline-tokens
verifiedmedium · 0.80
NVIDIA H100 PCIe 80GB x2 (Cisco C245M8), MLPerf v4.1 Offline: GPT-J 6B aggregate tok/s
3268.94 output tok/s AGGREGATE across 2x H100-PCIe-80GB (Offline; ~1634 tok/s per card)GPT-J 6B (FP8) on nvidia-h100-pcie via tensorrt-llmbenchmarks/h100pcie-2x-mlperf-v4-1-gptj-offline-tokens-cisco
verifiedmedium · 0.80
NVIDIA H100 PCIe 80GB x2 (Dell R760), MLPerf v4.1 Offline: GPT-J 6B aggregate tok/s
3317.08 output tok/s AGGREGATE across 2x H100-PCIe-80GB (Offline; ~1658 tok/s per card)GPT-J 6B (FP8) on nvidia-h100-pcie via tensorrt-llmbenchmarks/h100pcie-2x-mlperf-v4-1-gptj-offline-tokens
verifiedmedium · 0.80
NVIDIA H100 PCIe 80GB x2 (Dell R760), MLPerf v4.1 Server: GPT-J 6B aggregate tok/s
2759.33 output tok/s AGGREGATE across 2x H100-PCIe-80GB (Server; ~1380 tok/s per card)GPT-J 6B (FP8) on nvidia-h100-pcie via tensorrt-llmbenchmarks/h100pcie-2x-mlperf-v4-1-gptj-server-tokens
verifiedmedium · 0.80
NVIDIA Jetson page: Thor vLLM DeepSeek-R1 32B, concurrency 1, 13.31 tok/s
13.31 tok/s per request (max concurrency 1)DeepSeek-R1 32B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-deepseek-r1-32b-c1-decode
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM DeepSeek-R1 32B, concurrency 8, 82.6 tok/s aggregate
82.6 tok/s AGGREGATE across max concurrency 8DeepSeek-R1 32B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-deepseek-r1-32b-c8-aggregate
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM DeepSeek-R1 7B, concurrency 1, 41.32 tok/s
41.32 tok/s per request (max concurrency 1)DeepSeek-R1 7B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-deepseek-r1-7b-c1-decode
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM DeepSeek-R1 7B, concurrency 8, 304.8 tok/s aggregate
304.8 tok/s AGGREGATE across max concurrency 8DeepSeek-R1 7B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-deepseek-r1-7b-c8-aggregate
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Llama 3.1 8B, concurrency 1, 41.3 tok/s
41.3 tok/s per request (max concurrency 1)Llama 3.1 8B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-llama31-8b-c1-decode
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Llama 3.1 8B, concurrency 8, 150.8 tok/s aggregate
150.8 tok/s AGGREGATE across max concurrency 8Llama 3.1 8B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-llama31-8b-c8-aggregate
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Llama 3.3 70B, concurrency 1, 4.7 tok/s
4.7 tok/s per request (max concurrency 1)Llama 3.3 70B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-llama33-70b-c1-decode
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Llama 3.3 70B, concurrency 8, 12.6 tok/s aggregate
12.6 tok/s AGGREGATE across max concurrency 8Llama 3.3 70B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-llama33-70b-c8-aggregate
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Qwen3 30B-A3B, concurrency 1, 61 tok/s
61.0 tok/s per request (max concurrency 1)Qwen3 30B-A3B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-qwen3-30b-a3b-c1-decode
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Qwen3 30B-A3B, concurrency 8, 226.4 tok/s aggregate
226.4 tok/s AGGREGATE across max concurrency 8Qwen3 30B-A3B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-qwen3-30b-a3b-c8-aggregate
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Qwen3 32B, concurrency 1, 13.19 tok/s
13.19 tok/s per request (max concurrency 1)Qwen3 32B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-qwen3-32b-c1-decode
verifiedlow · 0.60
NVIDIA Jetson page: Thor vLLM Qwen3 32B, concurrency 8, 79.1 tok/s aggregate
79.1 tok/s AGGREGATE across max concurrency 8Qwen3 32B on nvidia-jetson-agx-thor via vllmbenchmarks/nv-thor-vllm-qwen3-32b-c8-aggregate
verifiedlow · 0.60
NVIDIA L4 x8 (ASUSTeK ESC8000_E11), MLPerf v4.0 Offline: bert-99 aggregate samples/s
7597.71 samples/s AGGREGATE across 8x L4-PCIe-24GB (Offline; ~950 samples/s per card)bert-99 (TensorRT, INT8+FP16) on nvidia-l4 via tensorrt-llmbenchmarks/l4-8x-mlperf-v4-0-bert99-offline-samples
verifiedmedium · 0.75
NVIDIA L4 x8 (ASUSTeK ESC8000_E11), MLPerf v4.0 Offline: bert-99.9 aggregate samples/s
5189.92 samples/s AGGREGATE across 8x L4-PCIe-24GB (Offline; ~649 samples/s per card)bert-99.9 (TensorRT, INT8+FP16) on nvidia-l4 via tensorrt-llmbenchmarks/l4-8x-mlperf-v4-0-bert999-offline-samples
verifiedmedium · 0.75
NVIDIA L4 x8 (ASUSTeK ESC8000_E11), MLPerf v4.0 Server: bert-99 aggregate samples/s
7304.78 samples/s AGGREGATE across 8x L4-PCIe-24GB (Server; ~913 samples/s per card)bert-99 (TensorRT, INT8+FP16) on nvidia-l4 via tensorrt-llmbenchmarks/l4-8x-mlperf-v4-0-bert99-server-samples
verifiedmedium · 0.75
NVIDIA L4, llama.cpp b5540 CUDA: Qwen2.5-7B Q4_K_M short(256/128) c=1 output tok/s
47.37 output tok/s at concurrency 1 (= per-request ~47.4 t/s)Qwen2.5-7B-Instruct-GGUF Q4_K_M (~4.4 GB) on nvidia-l4 via llama-cppbenchmarks/l4-llamacpp-qwen25-7b-q4km-short-c1-tps
verifiedmedium · 0.70
NVIDIA L4, llama.cpp b5540 CUDA: Qwen2.5-7B Q4_K_M short(256/128) c=64 aggregate tok/s
189.19 output tok/s AGGREGATE across 64 concurrent requests (~2.96 t/s per request)Qwen2.5-7B-Instruct-GGUF Q4_K_M on nvidia-l4 via llama-cppbenchmarks/l4-llamacpp-qwen25-7b-q4km-short-c64-tps
verifiedmedium · 0.70
NVIDIA L4, SGLang 0.4.6: Qwen2.5-7B-Instruct FP16 short(256/128) c=1 output tok/s
17.12 output tok/s at concurrency 1 (= per-request ~17.1 t/s)Qwen2.5-7B-Instruct (FP16) on nvidia-l4 via sglangbenchmarks/l4-sglang-qwen25-7b-fp16-short-c1-tps
verifiedmedium · 0.70
NVIDIA L4, SGLang 0.4.6: Qwen2.5-7B-Instruct FP16 short(256/128) c=64 aggregate tok/s
913.9 output tok/s AGGREGATE across 64 concurrent requests (~14.3 t/s per request)Qwen2.5-7B-Instruct (FP16) on nvidia-l4 via sglangbenchmarks/l4-sglang-qwen25-7b-fp16-short-c64-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-AWQ short(256/128) c=64 aggregate tok/s
975.64 output tok/s AGGREGATE across 64 concurrent requests (~15.2 t/s per request)Qwen2.5-7B-Instruct-AWQ (INT4) on nvidia-l4 via vllmbenchmarks/l4-vllm-awq-qwen25-7b-short-c64-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct FP16 long(2048/512) c=64 aggregate tok/s
776.53 output tok/s AGGREGATE across 64 concurrent requests (~12.1 t/s per request)Qwen2.5-7B-Instruct (FP16) on nvidia-l4 via vllmbenchmarks/l4-vllm-qwen25-7b-fp16-long-c64-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct FP16 short(256/128) c=1 output tok/s
17.11 output tok/s at concurrency 1 (= per-request decode ~17.1 t/s)Qwen2.5-7B-Instruct (FP16) on nvidia-l4 via vllmbenchmarks/l4-vllm-qwen25-7b-fp16-short-c1-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct FP16 short(256/128) c=64 aggregate tok/s
831.76 output tok/s AGGREGATE across 64 concurrent requests (~13.0 t/s per request)Qwen2.5-7B-Instruct (FP16) on nvidia-l4 via vllmbenchmarks/l4-vllm-qwen25-7b-fp16-short-c64-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct-AWQ short(256/128) c=1 output tok/s
43.19 output tok/s at concurrency 1 (= per-request ~43.2 t/s)Qwen2.5-7B-Instruct-AWQ (INT4) on nvidia-l4 via vllmbenchmarks/l4-vllm-awq-qwen25-7b-short-c1-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen3-8B FP16 reasoning(256/~8192) c=16 aggregate tok/s
174.21 output tok/s AGGREGATE across 16 concurrent reasoning requests (~10.9 t/s per request)Qwen3-8B (FP16, thinking tokens) on nvidia-l4 via vllmbenchmarks/l4-vllm-qwen3-8b-fp16-reasoning-c16-tps
verifiedmedium · 0.70
NVIDIA L4, vLLM 0.8.5: Qwen3-8B-AWQ reasoning(256/~8192) c=16 aggregate tok/s
345.49 output tok/s AGGREGATE across 16 concurrent reasoning requests (~21.6 t/s per request)Qwen3-8B-AWQ (INT4, thinking tokens) on nvidia-l4 via vllmbenchmarks/l4-vllm-awq-qwen3-8b-reasoning-c16-tps
verifiedmedium · 0.70
NVIDIA L40S 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M pp1024 prefill tok/s
649.08 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 70B on nvidia-l40s via llama-cppbenchmarks/xd-nvidia-l40s-70b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA L40S 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
15.31 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 70B on nvidia-l40s via llama-cppbenchmarks/xd-nvidia-l40s-70b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA L40S 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
2491.65 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-l40s via llama-cppbenchmarks/xd-nvidia-l40s-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA L40S 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
43.42 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-l40s via llama-cppbenchmarks/xd-nvidia-l40s-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA L40S 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
5908.52 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-l40s via llama-cppbenchmarks/xd-nvidia-l40s-8b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA L40S 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
113.6 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-l40s via llama-cppbenchmarks/xd-nvidia-l40s-8b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
678.2 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
972.6 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
22.7 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
31.2 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 8B (Q4_K) prompt processing at 32k context
764.7 tok/s (pp32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-8b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
1696.8 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 8B (Q4_K) token generation at 32k context
31.9 tok/s (tg32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-8b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 3060 12GB llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
55.2 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3060 via llama-cppbenchmarks/hc26-3060-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3070 8GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
2283.62 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-3070 via llama-cppbenchmarks/xd-rtx-3070-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX 3070 8GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
70.94 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-3070 via llama-cppbenchmarks/xd-rtx-3070-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX 3080 10GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
3557.02 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-3080-10gb via llama-cppbenchmarks/xd-rtx-3080-10gb-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX 3080 10GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
106.4 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-3080-10gb via llama-cppbenchmarks/xd-rtx-3080-10gb-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX 3080 Ti 12GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
3556.67 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-3080-ti via llama-cppbenchmarks/xd-rtx-3080-ti-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX 3080 Ti 12GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
106.71 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-3080-ti via llama-cppbenchmarks/xd-rtx-3080-ti-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
1773.9 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
2600.1 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
52.3 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
69.9 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 8B (Q4_K) prompt processing at 32k context
1761.2 tok/s (pp32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-8b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
4211.7 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 8B (Q4_K) token generation at 32k context
68.1 tok/s (tg32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-8b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 3080 Ti llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
115.2 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc26-3080ti-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 128k context
1147.1 tok/s (pp128k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 16k context
3068.9 tok/s (pp16k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 256k context
671.4 tok/s (pp256k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-256k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 32k context
2453.4 tok/s (pp32k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 4k context
3625.6 tok/s (pp4k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 64k context
1765.1 tok/s (pp64k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) token generation at 128k context
83.0 tok/s (tg128k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) token generation at 16k context
115.0 tok/s (tg16k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) token generation at 256k context
64.4 tok/s (tg256k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-256k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) token generation at 32k context
107.5 tok/s (tg32k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) token generation at 4k context
119.4 tok/s (tg4k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 26B (Q4_K) token generation at 64k context
98.9 tok/s (tg64k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-26b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 16k context
913.2 tok/s (pp16k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-31b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 32k context
723.7 tok/s (pp32k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-31b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 4k context
1155.8 tok/s (pp4k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-31b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 31B (Q4_K) token generation at 16k context
33.5 tok/s (tg16k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-31b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 31B (Q4_K) token generation at 32k context
31.4 tok/s (tg32k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-31b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Gemma4 31B (Q4_K) token generation at 4k context
34.7 tok/s (tg4k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-gemma4-31b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 4k context
2988.6 tok/s (pp4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 64k context
800.9 tok/s (pp64k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 4k context
153.6 tok/s (tg4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 64k context
58.3 tok/s (tg64k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen3-30b-a3b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 16k context
977.4 tok/s (pp16k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 32k context
848.2 tok/s (pp32k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 4k context
1104.2 tok/s (pp4k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 64k context
678.9 tok/s (pp64k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 16k context
32.3 tok/s (tg16k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 32k context
31.0 tok/s (tg32k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 4k context
33.5 tok/s (tg4k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 64k context
28.8 tok/s (tg64k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-27b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 128k context
1288.9 tok/s (pp128k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-128k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 16k context
2381.3 tok/s (pp16k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-16k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 32k context
2121.6 tok/s (pp32k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-32k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 4k context
2622.1 tok/s (pp4k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-4k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 64k context
1749.8 tok/s (pp64k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-64k-pp
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 128k context
79.4 tok/s (tg128k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-128k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 16k context
107.1 tok/s (tg16k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-16k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 32k context
101.2 tok/s (tg32k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-32k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 4k context
111.2 tok/s (tg4k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-4k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 64k context
93.1 tok/s (tg64k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-3090 via llama-cppbenchmarks/hc26-3090-qwen35-35b-mxfp4-64k-tg
verifiedlow · 0.60
NVIDIA RTX 3090 Ti 24GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
3556.67 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/xd-nvidia-rtx-3090-ti-8b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA RTX 3090 Ti 24GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
106.71 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/xd-nvidia-rtx-3090-ti-8b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA RTX 4000 Ada 20GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
2951.87 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-rtx-4000-ada via llama-cppbenchmarks/xd-rtx-4000-ada-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA RTX 4000 Ada 20GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
20.85 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-rtx-4000-ada via llama-cppbenchmarks/xd-rtx-4000-ada-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA RTX 4000 Ada 20GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
2310.53 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-4000-ada via llama-cppbenchmarks/xd-rtx-4000-ada-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX 4000 Ada 20GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
58.59 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-4000-ada via llama-cppbenchmarks/xd-rtx-4000-ada-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX 4070 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
1186.9 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
2075.0 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
32.9 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
42.5 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 32k context
812.1 tok/s (pp32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-8b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
3445.5 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 8B (Q4_K) token generation at 32k context
36.1 tok/s (tg32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-8b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
71.2 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070 via llama-cppbenchmarks/hc26-4070-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
1578.0 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
2455.6 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
37.2 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
45.5 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 8B (Q4_K) prompt processing at 32k context
980.2 tok/s (pp32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-8b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
4012.3 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 8B (Q4_K) token generation at 32k context
37.2 tok/s (tg32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-8b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 SUPER llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
75.4 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-super via llama-cppbenchmarks/hc26-4070s-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti 12GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
3653.07 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-4070-ti via llama-cppbenchmarks/xd-rtx-4070-ti-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX 4070 Ti 12GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
82.21 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-4070-ti via llama-cppbenchmarks/xd-rtx-4070-ti-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
1951.4 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
2866.6 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
37.8 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
45.8 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 8B (Q4_K) prompt processing at 32k context
1099.7 tok/s (pp32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-8b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
4877.7 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 8B (Q4_K) token generation at 32k context
42.1 tok/s (tg32k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-8b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
75.8 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc26-4070ti-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 128k context
993.5 tok/s (pp128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 4k context
5628.1 tok/s (pp4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-gptoss-20b-mxfp4-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B (MXFP4) token generation at 128k context
57.5 tok/s (tg128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B (MXFP4) token generation at 4k context
128.9 tok/s (tg4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-gptoss-20b-mxfp4-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 14B (Q4_K) prompt processing at 32k context
1236.9 tok/s (pp32k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-14b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
3048.0 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 14B (Q4_K) token generation at 32k context
37.7 tok/s (tg32k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-14b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
58.1 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
5220.1 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 8B (Q4_K) prompt processing at 64k context
829.4 tok/s (pp64k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-8b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
96.3 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 4070 Ti SUPER llama.cpp: Qwen3 8B (Q4_K) token generation at 64k context
36.6 tok/s (tg64k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc26-4070tis-qwen3-8b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 4080 16GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
6758.9 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-4080 via llama-cppbenchmarks/xd-nvidia-rtx-4080-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA RTX 4080 16GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
40.29 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-4080 via llama-cppbenchmarks/xd-nvidia-rtx-4080-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA RTX 4080 16GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
5064.99 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-4080 via llama-cppbenchmarks/xd-nvidia-rtx-4080-8b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA RTX 4080 16GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
106.22 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-4080 via llama-cppbenchmarks/xd-nvidia-rtx-4080-8b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA RTX 5000 Ada 32GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
5835.41 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-rtx-5000-ada via llama-cppbenchmarks/xd-rtx-5000-ada-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA RTX 5000 Ada 32GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
32.67 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-rtx-5000-ada via llama-cppbenchmarks/xd-rtx-5000-ada-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA RTX 5000 Ada 32GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
4467.46 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-5000-ada via llama-cppbenchmarks/xd-rtx-5000-ada-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX 5000 Ada 32GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
89.87 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-5000-ada via llama-cppbenchmarks/xd-rtx-5000-ada-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX 5070 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
1315.0 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
2381.6 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
40.6 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
54.2 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 32k context
1018.4 tok/s (pp32k, per-request batch-1)Qwen3 8B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-8b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
3862.2 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 8B (Q4_K) token generation at 32k context
42.5 tok/s (tg32k, per-request batch-1)Qwen3 8B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-8b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5070 llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
85.8 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc26-5070-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 128k context
1726.6 tok/s (pp128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 4k context
9146.1 tok/s (pp4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-mxfp4-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) token generation at 128k context
105.6 tok/s (tg128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) token generation at 4k context
172.4 tok/s (tg4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-mxfp4-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 32k context
1326.1 tok/s (pp32k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-14b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
3820.5 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 14B (Q4_K) token generation at 32k context
51.9 tok/s (tg32k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-14b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
80.6 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
6410.1 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 64k context
1124.6 tok/s (pp64k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-8b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
129.1 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5080 llama.cpp: Qwen3 8B (Q4_K) token generation at 64k context
49.0 tok/s (tg64k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-8b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 128k context
2839.1 tok/s (pp128k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 16k context
7733.2 tok/s (pp16k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 256k context
1707.2 tok/s (pp256k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-256k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 32k context
6292.6 tok/s (pp32k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 4k context
8799.2 tok/s (pp4k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) prompt processing at 64k context
4360.6 tok/s (pp64k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) token generation at 128k context
130.2 tok/s (tg128k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) token generation at 16k context
167.2 tok/s (tg16k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) token generation at 256k context
106.0 tok/s (tg256k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-256k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) token generation at 32k context
159.4 tok/s (tg32k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) token generation at 4k context
180.3 tok/s (tg4k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 26B (Q4_K) token generation at 64k context
149.4 tok/s (tg64k, per-request batch-1)Gemma4 26B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-26b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 128k context
900.2 tok/s (pp128k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 16k context
2794.3 tok/s (pp16k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 32k context
2229.1 tok/s (pp32k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 4k context
3395.0 tok/s (pp4k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) prompt processing at 64k context
1459.3 tok/s (pp64k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) token generation at 128k context
43.4 tok/s (tg128k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) token generation at 16k context
59.2 tok/s (tg16k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) token generation at 32k context
55.4 tok/s (tg32k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) token generation at 4k context
61.1 tok/s (tg4k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Gemma4 31B (Q4_K) token generation at 64k context
51.6 tok/s (tg64k, per-request batch-1)Gemma4 31B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gemma4-31b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) prompt processing at 128k context
1636.4 tok/s (pp128k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) prompt processing at 16k context
7168.1 tok/s (pp16k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) prompt processing at 32k context
5183.1 tok/s (pp32k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) prompt processing at 4k context
9443.8 tok/s (pp4k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) prompt processing at 64k context
3019.7 tok/s (pp64k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) token generation at 128k context
112.0 tok/s (tg128k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) token generation at 16k context
249.2 tok/s (tg16k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) token generation at 32k context
215.1 tok/s (tg32k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) token generation at 4k context
298.2 tok/s (tg4k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: gpt-oss 20B (Q4_K) token generation at 64k context
169.0 tok/s (tg64k, per-request batch-1)gpt-oss 20B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-gptoss-20b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 128k context
908.4 tok/s (pp128k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 16k context
4473.1 tok/s (pp16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 32k context
2908.4 tok/s (pp32k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
6497.6 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) prompt processing at 64k context
1707.2 tok/s (pp64k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) token generation at 128k context
37.2 tok/s (tg128k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) token generation at 16k context
102.7 tok/s (tg16k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) token generation at 32k context
82.4 tok/s (tg32k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
123.8 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 14B (Q4_K) token generation at 64k context
57.6 tok/s (tg64k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-14b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 128k context
985.0 tok/s (pp128k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-30b-a3b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 4k context
7093.0 tok/s (pp4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-30b-a3b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 128k context
76.8 tok/s (tg128k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-30b-a3b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 4k context
226.1 tok/s (tg4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-30b-a3b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 32B (Q4_K) prompt processing at 16k context
2077.2 tok/s (pp16k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-32b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 32B (Q4_K) prompt processing at 32k context
1451.1 tok/s (pp32k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-32b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 32B (Q4_K) prompt processing at 4k context
2931.3 tok/s (pp4k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-32b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 32B (Q4_K) token generation at 16k context
50.9 tok/s (tg16k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-32b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 32B (Q4_K) token generation at 32k context
43.8 tok/s (tg32k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-32b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 32B (Q4_K) token generation at 4k context
61.4 tok/s (tg4k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-32b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 128k context
1209.1 tok/s (pp128k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-8b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 16k context
8538.4 tok/s (pp16k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-8b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
11933.4 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 8B (Q4_K) token generation at 128k context
58.8 tok/s (tg128k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-8b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 8B (Q4_K) token generation at 16k context
162.3 tok/s (tg16k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-8b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
200.4 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 128k context
1019.8 tok/s (pp128k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 16k context
2721.5 tok/s (pp16k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 32k context
2341.8 tok/s (pp32k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 4k context
3004.1 tok/s (pp4k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) prompt processing at 64k context
1606.4 tok/s (pp64k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 128k context
44.1 tok/s (tg128k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 16k context
55.8 tok/s (tg16k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 32k context
53.8 tok/s (tg32k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 4k context
58.8 tok/s (tg4k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 27B (Q4_K) token generation at 64k context
50.1 tok/s (tg64k, per-request batch-1)Qwen3.5 27B (Q4_K) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-27b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 128k context
3242.6 tok/s (pp128k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-128k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 16k context
6142.3 tok/s (pp16k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-16k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 256k context
2003.7 tok/s (pp256k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-256k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 32k context
5611.4 tok/s (pp32k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-32k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 4k context
6605.2 tok/s (pp4k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-4k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) prompt processing at 64k context
4624.5 tok/s (pp64k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-64k-pp
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 128k context
118.2 tok/s (tg128k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-128k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 16k context
148.3 tok/s (tg16k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-16k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 256k context
97.3 tok/s (tg256k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-256k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 32k context
143.2 tok/s (tg32k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-32k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 4k context
165.2 tok/s (tg4k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-4k-tg
verifiedlow · 0.60
NVIDIA RTX 5090 llama.cpp: Qwen3.5 35B (MXFP4) token generation at 64k context
133.5 tok/s (tg64k, per-request batch-1)Qwen3.5 35B (MXFP4) on nvidia-rtx-5090 via llama-cppbenchmarks/hc26-5090-qwen35-35b-mxfp4-64k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M pp1024 prefill tok/s
547.03 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 70B on nvidia-rtx-6000-ada via llama-cppbenchmarks/xd-nvidia-rtx-6000-ada-70b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA RTX 6000 Ada 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
18.36 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 70B on nvidia-rtx-6000-ada via llama-cppbenchmarks/xd-nvidia-rtx-6000-ada-70b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA RTX 6000 Ada 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
6205.44 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-6000-ada via llama-cppbenchmarks/xd-nvidia-rtx-6000-ada-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA RTX 6000 Ada 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
51.97 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-6000-ada via llama-cppbenchmarks/xd-nvidia-rtx-6000-ada-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA RTX 6000 Ada 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
5560.94 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-6000-ada via llama-cppbenchmarks/xd-nvidia-rtx-6000-ada-8b-q4-k-m-pp1024
verifiedmedium · 0.70
NVIDIA RTX 6000 Ada 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
130.99 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B on nvidia-rtx-6000-ada via llama-cppbenchmarks/xd-nvidia-rtx-6000-ada-8b-q4-k-m-tg1024
verifiedmedium · 0.70
NVIDIA RTX 6000 Ada llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 128k context
1177.7 tok/s (pp128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 4k context
6495.8 tok/s (pp4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-mxfp4-4k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: gpt-oss 20B (MXFP4) token generation at 128k context
57.5 tok/s (tg128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: gpt-oss 20B (MXFP4) token generation at 4k context
160.8 tok/s (tg4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-gptoss-20b-mxfp4-4k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Llama 3.3 70B (Q4_K) prompt processing at 16k context
526.0 tok/s (pp16k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Llama 3.3 70B (Q4_K) prompt processing at 4k context
678.1 tok/s (pp4k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Llama 3.3 70B (Q4_K) token generation at 16k context
13.7 tok/s (tg16k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Llama 3.3 70B (Q4_K) token generation at 4k context
18.1 tok/s (tg4k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-llama33-70b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 14B (Q4_K) prompt processing at 128k context
358.6 tok/s (pp128k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 14B (Q4_K) prompt processing at 4k context
2912.1 tok/s (pp4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 14B (Q4_K) token generation at 128k context
15.2 tok/s (tg128k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 14B (Q4_K) token generation at 4k context
81.2 tok/s (tg4k, per-request batch-1)Qwen3 14B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 128k context
448.6 tok/s (pp128k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 4k context
3888.0 tok/s (pp4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 128k context
20.7 tok/s (tg128k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 4k context
199.6 tok/s (tg4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-30b-a3b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 32B (Q4_K) prompt processing at 4k context
1203.9 tok/s (pp4k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 32B (Q4_K) prompt processing at 64k context
348.6 tok/s (pp64k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 32B (Q4_K) token generation at 4k context
37.4 tok/s (tg4k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 32B (Q4_K) token generation at 64k context
11.8 tok/s (tg64k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-32b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 8B (Q4_K) prompt processing at 128k context
541.3 tok/s (pp128k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
4932.5 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 8B (Q4_K) token generation at 128k context
20.8 tok/s (tg128k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX 6000 Ada llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
133.9 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-6000-ada via llama-cppbenchmarks/hc26-6000ada-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M pp1024 prefill tok/s
466.82 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 70B Q4_K_M on nvidia-rtx-a6000 via llama-cppbenchmarks/xd-rtx-a6000-70b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 70B Q4_K_M tg1024 decode tok/s
14.58 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 70B Q4_K_M on nvidia-rtx-a6000 via llama-cppbenchmarks/xd-rtx-a6000-70b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 pp1024 prefill tok/s
4315.18 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-rtx-a6000 via llama-cppbenchmarks/xd-rtx-a6000-8b-f16-pp1024
verifiedmedium · 0.70
NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 8B F16 tg1024 decode tok/s
40.25 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B F16 on nvidia-rtx-a6000 via llama-cppbenchmarks/xd-rtx-a6000-8b-f16-tg1024
verifiedmedium · 0.70
NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M pp1024 prefill tok/s
3621.81 tok/s (pp1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-a6000 via llama-cppbenchmarks/xd-rtx-a6000-8b-q4km-pp1024
verifiedmedium · 0.70
NVIDIA RTX A6000 48GB llama.cpp cuBLAS: LLaMA-3 8B Q4_K_M tg1024 decode tok/s
102.22 tok/s (tg1024 avg, per-request batch-1)LLaMA-3 8B Q4_K_M on nvidia-rtx-a6000 via llama-cppbenchmarks/xd-rtx-a6000-8b-q4km-tg1024
verifiedmedium · 0.70
NVIDIA RTX A6000 llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 128k context
702.8 tok/s (pp128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: gpt-oss 20B (MXFP4) prompt processing at 4k context
3490.6 tok/s (pp4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-mxfp4-4k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: gpt-oss 20B (MXFP4) token generation at 128k context
33.4 tok/s (tg128k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: gpt-oss 20B (MXFP4) token generation at 4k context
108.4 tok/s (tg4k, per-request batch-1)gpt-oss 20B (MXFP4) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-gptoss-20b-mxfp4-4k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Llama 3.3 70B (Q4_K) prompt processing at 16k context
280.3 tok/s (pp16k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-16k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Llama 3.3 70B (Q4_K) prompt processing at 4k context
423.0 tok/s (pp4k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Llama 3.3 70B (Q4_K) token generation at 16k context
9.1 tok/s (tg16k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-16k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Llama 3.3 70B (Q4_K) token generation at 4k context
12.6 tok/s (tg4k, per-request batch-1)Llama 3.3 70B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-llama33-70b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 128k context
429.2 tok/s (pp128k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 30B A3B (Q4_K) prompt processing at 4k context
2142.4 tok/s (pp4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 128k context
14.8 tok/s (tg128k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 30B A3B (Q4_K) token generation at 4k context
131.2 tok/s (tg4k, per-request batch-1)Qwen3 30B A3B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-30b-a3b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 32B (Q4_K) prompt processing at 4k context
736.2 tok/s (pp4k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 32B (Q4_K) prompt processing at 64k context
319.0 tok/s (pp64k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-q4k-64k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 32B (Q4_K) token generation at 4k context
24.6 tok/s (tg4k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-q4k-4k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 32B (Q4_K) token generation at 64k context
8.7 tok/s (tg64k, per-request batch-1)Qwen3 32B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-32b-q4k-64k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 128k context
527.9 tok/s (pp128k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-q4k-128k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 8B (Q4_K) prompt processing at 4k context
2756.0 tok/s (pp4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 8B (Q4_K) token generation at 128k context
16.0 tok/s (tg128k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-q4k-128k-tg
verifiedlow · 0.60
NVIDIA RTX A6000 llama.cpp: Qwen3 8B (Q4_K) token generation at 4k context
90.5 tok/s (tg4k, per-request batch-1)Qwen3 8B (Q4_K) on nvidia-rtx-a6000 via llama-cppbenchmarks/hc26-a6000-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
Ollama (llama.cpp backend) on MacBook Pro M3 Max 128GB: Qwen3.8-27B Q4_K_M decode tokens/s
12.0 tok/s (batch 1, short prompt decode)Qwen3.8 27B on apple-m3-max via llama-cppbenchmarks/ollama-m3-max-qwen3-8-27b-llamacpp-decode-tok-s
verifiedmedium · 0.80
Ollama (MLX backend) on MacBook Pro M3 Max 128GB: Qwen3.8-27B 4-bit decode tokens/s
42.5 tok/s (batch 1, short prompt decode)Qwen3.8 27B on apple-m3-max via ollamabenchmarks/ollama-m3-max-qwen3-8-27b-mlx-decode-tok-s
verifiedmedium · 0.80
Oracle 8xMI355X, MLPerf Inference v6.1 Server: Llama 3.1 8B - a case where Server throughput EXCEEDS Offline on one node
148594 output tokens/s (aggregate; summary.csv value, raw log not independently re-read for this record)Llama 3.1 8B (MLPerf v6.1 submission, FP8 weights) on amd-instinct-mi355x via vllmbenchmarks/bench-mlperf-v6-1-oracle-mi355x-llama3-1-8b-server
verifiedmedium · 0.80
Per-user decode tok/s, MPT-7B static batch 1, 1x A100
57.6 decode tokens/s experienced by one user (batch size 1, so per-user == aggregate)MPT-7B (FP16) on ? via fastertransformerbenchmarks/mpt7b-a100-bs1-per-user-decode-tps
verifiedmedium · 0.80
Per-user decode tok/s, MPT-7B static batch 64, 1x A100
12.5 decode tokens/s experienced by one user (batch size 64, aggregate is 800)MPT-7B (FP16) on ? via fastertransformerbenchmarks/mpt7b-a100-bs64-per-user-decode-tps
verifiedmedium · 0.80
Perplexity: Llama 2 70B FP8 on 8x H100 80GB, TP=2 / batch 128, 767 output tokens/s per GPU
767 output tokens/s per GPU (aggregate across concurrent requests)Llama-2-70B on nvidia-h100-sxm via tensorrt-llmbenchmarks/perplexity-llama2-70b-h100-fp8-tp2-bs128-tps-per-gpu
verifiedmedium · 0.75
PLD (prompt lookup) acceptance length and speedup on Vicuna-13B (temperature=0)
1.69 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-pld-acceptance-vicuna13b-t0
verifiedhigh · 0.90
Qualcomm Cloud AI 100 Ultra, vLLM: DeepSeek-R1-70B peak aggregate tok/s (arXiv:2507.00418)
3285.2 tok/s aggregate (peak over 32-256 concurrent requests; 1 card)DeepSeek-R1-70B on qualcomm-ai200 via vllmbenchmarks/arxiv2507-qaic100ultra-deepseek-r1-70b-tps-agg
verifiedmedium · 0.75
Qualcomm Cloud AI 100 Ultra, vLLM: Llama-3.1-8B peak aggregate tok/s (arXiv:2507.00418)
5062.6 tok/s aggregate (peak over 32-256 concurrent requests; 1 card)Llama-3.1-8B on qualcomm-ai200 via vllmbenchmarks/arxiv2507-qaic100ultra-llama3-1-8b-tps-agg
verifiedmedium · 0.75
Qualcomm Cloud AI 100 Ultra, vLLM: Llama-3.3-70B peak aggregate tok/s (arXiv:2507.00418)
4528.3 tok/s aggregate (peak over 32-256 concurrent requests; 1 card)Llama-3.3-70B on qualcomm-ai200 via vllmbenchmarks/arxiv2507-qaic100ultra-llama3-3-70b-tps-agg
verifiedmedium · 0.75
Qualcomm Cloud AI 100 Ultra, vLLM: Qwen2.5-32B peak aggregate tok/s (arXiv:2507.00418)
4926.6 tok/s aggregate (peak over 32-256 concurrent requests; 1 card)Qwen2.5-32B on qualcomm-ai200 via vllmbenchmarks/arxiv2507-qaic100ultra-qwen2-5-32b-tps-agg
verifiedmedium · 0.75
Qwen docs: Qwen3-0.6B AWQ-INT4 on 1x H20 HF-Transformers, input 30720, 345.71 tok/s e2e
345.71 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-0.6B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-0p6b-awq-int4-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-0.6B BF16 on 1x H20 HF-Transformers, input 1, 58.57 tok/s e2e
58.57 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-0.6B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-0p6b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-0.6B BF16 on 1x H20 SGLang, input 1, 414.17 tok/s e2e
414.17 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-0.6B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-0p6b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-0.6B BF16 on 1x H20 SGLang, input 30720, 3577.42 tok/s e2e
3577.42 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-0.6B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-0p6b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-0.6B FP8 on 1x H20 SGLang, input 1, 458.03 tok/s e2e
458.03 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-0.6B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-0p6b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-0.6B FP8 on 1x H20 SGLang, input 30720, 3819.86 tok/s e2e
3819.86 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-0.6B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-0p6b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-1.7B AWQ-INT4 on 1x H20 HF-Transformers, input 30720, 386.85 tok/s e2e
386.85 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-1.7B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-1p7b-awq-int4-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-1.7B BF16 on 1x H20 HF-Transformers, input 1, 59.83 tok/s e2e
59.83 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-1.7B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-1p7b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-1.7B BF16 on 1x H20 SGLang, input 1, 227.8 tok/s e2e
227.8 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-1.7B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-1p7b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-1.7B BF16 on 1x H20 SGLang, input 30720, 2439.03 tok/s e2e
2439.03 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-1.7B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-1p7b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-1.7B FP8 on 1x H20 SGLang, input 1, 333.9 tok/s e2e
333.9 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-1.7B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-1p7b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-1.7B FP8 on 1x H20 SGLang, input 30720, 3165.32 tok/s e2e
3165.32 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-1.7B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-1p7b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-14B AWQ-INT4 on 1x H20 HF-Transformers, input 30720, 363.25 tok/s e2e
363.25 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-14B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-14b-awq-int4-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-14B BF16 on 1x H20 HF-Transformers, input 1, 40.66 tok/s e2e
40.66 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-14B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-14b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-14B BF16 on 1x H20 SGLang, input 1, 47.1 tok/s e2e
47.1 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-14B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-14b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-14B BF16 on 1x H20 SGLang, input 30720, 525.8 tok/s e2e
525.8 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-14B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-14b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-14B BF16 on 1x H20 SGLang, input 63488, 742.36 tok/s e2e
742.36 tok/s (batch-1 e2e, in=63488+2048gen)Qwen3-14B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-14b-bf16-in63488-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-14B FP8 on 1x H20 SGLang, input 1, 97.11 tok/s e2e
97.11 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-14B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-14b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-14B FP8 on 1x H20 SGLang, input 30720, 880.72 tok/s e2e
880.72 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-14B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-14b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-235B-A22B BF16 on 8x H20 SGLang, input 1, 74.5 tok/s e2e
74.5 tok/s (batch-1 e2e, in=1+2048gen, TP=8)Qwen3-235B-A22B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-235B-A22B BF16 on 8x H20 SGLang, input 129024, 1639.54 tok/s e2e
1639.54 tok/s (batch-1 e2e, in=129024+2048gen, TP=8)Qwen3-235B-A22B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-bf16-in129024-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-235B-A22B BF16 on 8x H20 SGLang, input 30720, 979.41 tok/s e2e
979.41 tok/s (batch-1 e2e, in=30720+2048gen, TP=8)Qwen3-235B-A22B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-235B-A22B BF16 on 8x H20 SGLang, input 63488, 1493.91 tok/s e2e
1493.91 tok/s (batch-1 e2e, in=63488+2048gen, TP=8)Qwen3-235B-A22B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-bf16-in63488-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-235B-A22B FP8 on 4x H20 SGLang, input 1, 71.65 tok/s e2e
71.65 tok/s (batch-1 e2e, in=1+2048gen, TP=4)Qwen3-235B-A22B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-235B-A22B FP8 on 4x H20 SGLang, input 30720, 887.9 tok/s e2e
887.9 tok/s (batch-1 e2e, in=30720+2048gen, TP=4)Qwen3-235B-A22B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-30B-A3B BF16 on 1x H20 HF-Transformers, input 1, 1.89 tok/s e2e
1.89 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-30B-A3B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-30b-a3b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-30B-A3B BF16 on 1x H20 HF-Transformers, input 30720, 27.03 tok/s e2e
27.03 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-30B-A3B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-30b-a3b-bf16-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-30B-A3B BF16 on 1x H20 SGLang, input 1, 137.18 tok/s e2e
137.18 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-30B-A3B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-30b-a3b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-30B-A3B BF16 on 1x H20 SGLang, input 30720, 1283.94 tok/s e2e
1283.94 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-30B-A3B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-30b-a3b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-30B-A3B BF16 on 1x H20 SGLang, input 63488, 1538.79 tok/s e2e
1538.79 tok/s (batch-1 e2e, in=63488+2048gen)Qwen3-30B-A3B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-30b-a3b-bf16-in63488-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-30B-A3B FP8 on 1x H20 SGLang, input 1, 155.55 tok/s e2e
155.55 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-30B-A3B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-30b-a3b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-30B-A3B FP8 on 1x H20 SGLang, input 30720, 1405.91 tok/s e2e
1405.91 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-30B-A3B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-30b-a3b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-32B AWQ-INT4 on 1x H20 HF-Transformers, input 30720, 188.11 tok/s e2e
188.11 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-32B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-32b-awq-int4-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-32B AWQ-INT4 on 1x H20 SGLang, input 1, 47.67 tok/s e2e
47.67 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-32B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-32b-awq-int4-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-32B BF16 on 1x H20 HF-Transformers, input 1, 26.24 tok/s e2e
26.24 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-32B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-32b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-32B BF16 on 1x H20 SGLang, input 1, 20.72 tok/s e2e
20.72 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-32B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-32b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-32B BF16 on 1x H20 SGLang, input 30720, 240.75 tok/s e2e
240.75 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-32B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-32b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-32B BF16 on 1x H20 SGLang, input 63488, 342.96 tok/s e2e
342.96 tok/s (batch-1 e2e, in=63488+2048gen)Qwen3-32B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-32b-bf16-in63488-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-32B BF16 on 2x H20 SGLang, input 129024, 711.4 tok/s e2e
711.4 tok/s (batch-1 e2e, in=129024+2048gen, TP=2)Qwen3-32B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-32b-bf16-in129024-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-32B FP8 on 1x H20 SGLang, input 1, 46.17 tok/s e2e
46.17 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-32B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-32b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-4B AWQ-INT4 on 1x H20 HF-Transformers, input 30720, 481.69 tok/s e2e
481.69 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-4B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-4b-awq-int4-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-4B AWQ-INT4 on 1x H20 SGLang, input 1, 199.71 tok/s e2e
199.71 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-4B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-4b-awq-int4-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-4B BF16 on 1x H20 HF-Transformers, input 1, 45.94 tok/s e2e
45.94 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-4B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-4b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-4B BF16 on 1x H20 SGLang, input 1, 133.13 tok/s e2e
133.13 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-4B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-4b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-4B BF16 on 1x H20 SGLang, input 30720, 1165.75 tok/s e2e
1165.75 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-4B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-4b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-4B FP8 on 1x H20 SGLang, input 1, 200.61 tok/s e2e
200.61 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-4B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-4b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-4B FP8 on 1x H20 SGLang, input 30720, 1467.71 tok/s e2e
1467.71 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-4B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-4b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B AWQ-INT4 on 1x H20 HF-Transformers, input 30720, 438.62 tok/s e2e
438.62 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-8B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-8b-awq-int4-in30720-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-8B AWQ-INT4 on 1x H20 SGLang, input 1, 144.11 tok/s e2e
144.11 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-awq-int4-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B BF16 on 1x H20 HF-Transformers, input 1, 45.32 tok/s e2e
45.32 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-8B on nvidia-h20 via hf-optimumbenchmarks/qwen-bench-h20-transformers-qwen3-8b-bf16-in1-tps
verifiedmedium · 0.70
Qwen docs: Qwen3-8B BF16 on 1x H20 SGLang, input 1, 81.73 tok/s e2e
81.73 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-bf16-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B BF16 on 1x H20 SGLang, input 129024, 1173.32 tok/s e2e
1173.32 tok/s (batch-1 e2e, in=129024+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-bf16-in129024-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B BF16 on 1x H20 SGLang, input 30720, 832.67 tok/s e2e
832.67 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-bf16-in30720-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B BF16 on 1x H20 SGLang, input 63488, 1112.78 tok/s e2e
1112.78 tok/s (batch-1 e2e, in=63488+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-bf16-in63488-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B FP8 on 1x H20 SGLang, input 1, 150.25 tok/s e2e
150.25 tok/s (batch-1 e2e, in=1+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-fp8-in1-tps
verifiedmedium · 0.75
Qwen docs: Qwen3-8B FP8 on 1x H20 SGLang, input 30720, 1242.24 tok/s e2e
1242.24 tok/s (batch-1 e2e, in=30720+2048gen)Qwen3-8B on nvidia-h20 via sglangbenchmarks/qwen-bench-h20-sglang-qwen3-8b-fp8-in30720-tps
verifiedmedium · 0.75
Qwen3-235B-A22B-Instruct-2507, temperature 0, 1000 completions of 1000 tokens: 80 unique completions, first divergence at token 103; all 1000 identical with batch-invariant kernels
80.0 distinct completions observed across 1000 runs at temperature 0 (a perfect implementation gives 1.0)Qwen/Qwen3-235B-A22B-Instruct-2507 (non-thinking mode) on nvidia-h100-sxm via vllmbenchmarks/qwen3-235b-a22b-greedy-decode-80-unique-completions-of-1000
verifiedhigh · 0.85
Qwen3-8B BF16 reference: AIME-24/25 + GSM8K + MATH-500 average accuracy (baseline for the MXFP4 W4A16 dispatch comparison)
82.36 accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)Qwen3-8B BF16 (unquantized reference) on nvidia-b200 via vllmbenchmarks/qwen3-8b-bf16-aime-math-gsm8k-avg-reference
verifiedhigh · 0.85
Qwen3-8B MXFP4 W4A16 served on MarlinMxfp4Kernel (correct W4A16 path): AIME-24/25 + GSM8K + MATH-500 average accuracy
78.16 accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)Qwen3-8B MXFP4 W4A16 (compressed-tensors, MXFP4 weights group_size=32, input_activations: null) on nvidia-b200 via vllmbenchmarks/mxfp4-w4a16-on-w4a16-marlin-aime-math-gsm8k-avg
verifiedhigh · 0.85
Qwen3-8B MXFP4 W4A16 served on the FlashInfer W4A4 kernel: AIME-24/25 + GSM8K + MATH-500 average accuracy
70.58 accuracy_percent (unweighted average of AIME-24, AIME-25, GSM8K, MATH-500)Qwen3-8B MXFP4 W4A16 (compressed-tensors, MXFP4 weights group_size=32, input_activations: null) on nvidia-b200 via vllmbenchmarks/mxfp4-w4a16-on-w4a4-kernel-aime-math-gsm8k-avg
verifiedhigh · 0.85
Qwen3.6-35B-A3B NVFP4 W4A16 MoE on the FlashInfer b12x backend: GSM8K accuracy (W4A16 checkpoint executed as W4A4)
0.9037 accuracy (GSM8K fraction correct, 5-shot as configured by the reporter's shared eval settings)Qwen3.6-35B-A3B-NVFP4 (quant_algo W4A16_NVFP4 for mlp.experts) on nvidia-rtx-pro-6000-blackwell via vllmbenchmarks/nvfp4-w4a16-flashinfer-b12x-moe-gsm8k
verifiedmedium · 0.80
Qwen3.6-35B-A3B NVFP4 W4A16 MoE on the Marlin backend: GSM8K accuracy (reference W4A16 path)
0.9136 accuracy (GSM8K fraction correct, same eval settings as the flashinfer_b12x comparison)Qwen3.6-35B-A3B-NVFP4 (quant_algo W4A16_NVFP4 for mlp.experts) on nvidia-rtx-pro-6000-blackwell via vllmbenchmarks/nvfp4-w4a16-marlin-moe-gsm8k
verifiedmedium · 0.80
R9700 1-GPU ROCm Qwen3.8-27B UD-Q4_K_XL depth-0 decode
30.6553398763441 tok/sQwen3.8-27B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-1gpu-rocm-qwen38-27b-q4kxl-d0-decode
verifiedlow · 0.60
R9700 1-GPU ROCm Qwen3.8-27B UD-Q4_K_XL depth-0 prefill
3048.7813629132934 tok/sQwen3.8-27B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-1gpu-rocm-qwen38-27b-q4kxl-d0-prefill
verifiedlow · 0.60
R9700 1-GPU Vulkan RADV Qwen3.8-27B UD-Q4_K_XL depth-0 decode
28.393187540146396 tok/sQwen3.8-27B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-1gpu-vulkan-qwen38-27b-q4kxl-d0-decode
verifiedlow · 0.60
R9700 1-GPU Vulkan RADV Qwen3.8-27B UD-Q4_K_XL depth-0 prefill
1016.0627860162153 tok/sQwen3.8-27B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-1gpu-vulkan-qwen38-27b-q4kxl-d0-prefill
verifiedlow · 0.60
R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-27B decode at 2.9K prompt (u/neuromacmd)
30.6 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-27b-amdvlk-q4km-2k9-tg
verifiedlow · 0.50
R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-27B prefill at 2.9K prompt (u/neuromacmd)
244 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-27b-amdvlk-q4km-2k9-pp
verifiedlow · 0.50
R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 2.9K prompt (u/neuromacmd)
133.0 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-35b-amdvlk-q4km-2k9-tg
verifiedlow · 0.50
R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 2.9K prompt (u/neuromacmd)
1030 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-35b-amdvlk-q4km-2k9-pp
verifiedlow · 0.50
R9700 llama.cpp RADV: Qwen3.5-27B decode at 2.9K prompt (u/neuromacmd, commit 48cda24)
30.4 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-27b-radv-q4km-2k9-tg
verifiedlow · 0.50
R9700 llama.cpp RADV: Qwen3.5-27B prefill at 2.9K prompt (u/neuromacmd, commit 48cda24)
971 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-27b-radv-q4km-2k9-pp
verifiedlow · 0.50
R9700 llama.cpp RADV: Qwen3.5-35B-A3B decode at 2.9K prompt (u/neuromacmd, commit 48cda24)
112.0 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-35b-radv-q4km-2k9-tg
verifiedlow · 0.50
R9700 llama.cpp RADV: Qwen3.5-35B-A3B prefill at 2.9K prompt (u/neuromacmd, commit 48cda24)
2987 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-35b-radv-q4km-2k9-pp
verifiedlow · 0.50
R9700 llama.cpp ROCm: Qwen3.5-27B decode at 2.9K prompt (u/neuromacmd)
25.2 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-27b-rocm-q4km-2k9-tg
verifiedlow · 0.50
R9700 llama.cpp ROCm: Qwen3.5-27B prefill at 2.9K prompt (u/neuromacmd)
547 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-27b-rocm-q4km-2k9-pp
verifiedlow · 0.50
R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 2.9K prompt (u/neuromacmd)
68.8 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-35b-rocm-q4km-2k9-tg
verifiedlow · 0.50
R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 2.9K prompt (u/neuromacmd)
1190 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-r9700-35b-rocm-q4km-2k9-pp
verifiedlow · 0.50
R9700 ROCm 10.0 LFM2.5-1.2B-Instruct BF16 64k-depth decode
155.8044 tok/s (per-request, concurrency 1)LFM2.5-1.2B-Instruct BF16 on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-rocm10-lfm25-1-2b-bf16-d64k-decode
verifiedlow · 0.60
R9700 ROCm 10.0 LFM2.5-1.2B-Instruct BF16 depth-0 decode
184.234 tok/s (per-request, concurrency 1)LFM2.5-1.2B-Instruct BF16 on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-rocm10-lfm25-1-2b-bf16-d0-decode
verifiedlow · 0.60
R9700 ROCm 10.0 LFM2.5-1.2B-Instruct BF16 depth-0 prefill
20717.405 tok/s (per-request, concurrency 1)LFM2.5-1.2B-Instruct BF16 on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-rocm10-lfm25-1-2b-bf16-d0-prefill
verifiedlow · 0.60
R9700 ROCm 10.0 Qwen3.6-35B-A3B UD-Q4_K_XL depth-0 decode
76.4278 tok/sQwen3.6-35B-A3B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-rocm10-qwen36-35b-a3b-q4kxl-d0-decode
verifiedlow · 0.60
R9700 Vulkan RADV Qwen3.6-35B-A3B UD-Q4_K_XL depth-0 decode
114.1852 tok/sQwen3.6-35B-A3B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-vulkan-qwen36-35b-a3b-q4kxl-d0-decode
verifiedlow · 0.60
R9700 Vulkan RADV Qwen3.6-35B-A3B UD-Q4_K_XL depth-0 prefill
3792.5094 tok/sQwen3.6-35B-A3B UD-Q4_K_XL on amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/llmb-r9700-vulkan-qwen36-35b-a3b-q4kxl-d0-prefill
verifiedlow · 0.60
Radeon Pro W7900 48GB llama.cpp b9600: shisa-v2.1-unphi4-14b Q4_K_M decode
50.8 tok/s decode (tg128, batch-1, ctx 0.5k, FA on); prefill 1721 t/sshisa-v2.1-unphi4-14b Q4_K_M on amd-radeon-pro-w7900 via llama-cppbenchmarks/llamabench-w7900-shisa14b-q4km-tg128
verifiedlow · 0.55
Radeon Pro W7900 48GB llama.cpp b9600: shisa-v2.1-unphi4-14b Q4_K_M prefill
1721.0 tok/s prefill (ctx 0.5k, FA on); decode 50.8 t/s tg128shisa-v2.1-unphi4-14b Q4_K_M on amd-radeon-pro-w7900 via llama-cppbenchmarks/llamabench-w7900-shisa14b-q4km-pp
verifiedlow · 0.55
Raspberry Pi AI HAT+ 2 (Hailo-10H) Qwen2.5-1.5B TPS
7.35 TPS (Hailo compiled-model table, single-stream)Qwen2.5-1.5B on hailo-10h via ollamabenchmarks/comm-hailo10h-pi-ai-hat2-qwen25-1p5b-tps
verifiedmedium · 0.75
Raspberry Pi AI HAT+ 2 (Hailo-10H): Llama-3.2-1B compiled-model TPS (vendor table)
9.89 tokens/s (Hailo compiled-model TPS table, single-stream)Llama-3.2-1B-Instruct on hailo-10h via ollamabenchmarks/comm-hailo10h-pi-ai-hat2-llama32-1b-tps
verifiedhigh · 0.85
RK3588 ChatGLM3-6B token-gen (RKLLM w8a8)
4.98 tok/s w8a8 128-in/64-outChatGLM3-6B on rockchip-rk3588-npu via local-onnxruntime-mobilebenchmarks/comm-rk3588-chatglm3-6b-tg
verifiedmedium · 0.70
RK3588 RKLLM Phi3-3.8B decode (w8a8)
7.45 tok/s decode (w8a8, 128-in/64-out)Phi3-3.8B on rockchip-rk3588-npu via local-onnxruntime-mobilebenchmarks/comm-rk3588-phi3-3p8b-decode
verifiedmedium · 0.70
RK3588 RKLLM Qwen2-0.5B decode (w8a8)
41.6 tok/s decode (w8a8, 128-in/64-out)Qwen2-0.5B on rockchip-rk3588-npu via local-onnxruntime-mobilebenchmarks/comm-rk3588-qwen2-0p5b-decode
verifiedmedium · 0.70
Rockchip RK3588 NPU Qwen2.5-1.5B decode (w8a8)
16.7 tok/s (w8a8, 128-in/64-out)Qwen2.5-1.5B on rockchip-rk3588-npu via local-onnxruntime-mobilebenchmarks/comm-rk3588-qwen25-1p5b-decode
verifiedmedium · 0.75
Rockchip RK3588 NPU, RKLLM w8a8: ChatGLM3-6B decode tok/s (vendor benchmark table)
4.98 tok/s (w8a8, 128-in/64-out; largest model Rockchip benchmarks)ChatGLM3-6B (w8a8) on rockchip-rk3588-npu via local-onnxruntime-mobilebenchmarks/comm-rk3588-chatglm3-6b-decode
verifiedmedium · 0.80
RTX 3060 12GB llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
31.2 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-3060 via llama-cppbenchmarks/hc-3060-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 3060 12GB llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
678.2 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-3060 via llama-cppbenchmarks/hc-3060-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 3060 12GB llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
972.6 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-3060 via llama-cppbenchmarks/hc-3060-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 3060 12GB llama.cpp: Qwen3 8B Q4_K decode at 4k ctx (Hardware Corner)
55.2 tok/s token generation, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-3060 via llama-cppbenchmarks/hc-3060-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
RTX 3060 12GB llama.cpp: Qwen3 8B Q4_K prefill at 16k ctx (Hardware Corner)
1119.2 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 8B on nvidia-rtx-3060 via llama-cppbenchmarks/hc-3060-qwen3-8b-q4k-16k-pp
verifiedlow · 0.60
RTX 3060 12GB llama.cpp: Qwen3 8B Q4_K prefill at 4k ctx (Hardware Corner)
1696.8 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-3060 via llama-cppbenchmarks/hc-3060-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
RTX 3080 Ti 12GB llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
69.9 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-3080ti-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 3080 Ti 12GB llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
1773.9 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-3080ti-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 3080 Ti 12GB llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
2600.1 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-3080ti-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 3080 Ti 12GB llama.cpp: Qwen3 8B Q4_K decode at 4k ctx (Hardware Corner)
115.2 tok/s token generation, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-3080ti-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
RTX 3080 Ti 12GB llama.cpp: Qwen3 8B Q4_K prefill at 16k ctx (Hardware Corner)
2658.5 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 8B on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-3080ti-qwen3-8b-q4k-16k-pp
verifiedlow · 0.60
RTX 3080 Ti 12GB llama.cpp: Qwen3 8B Q4_K prefill at 4k ctx (Hardware Corner)
4211.7 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-3080-ti via llama-cppbenchmarks/hc-3080ti-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
RTX 3090 Qwen3-32B Q4 token-gen
35.0 tok/s TG batch-1 @4K ctxQwen3-32B on nvidia-rtx-3090 via llama-cppbenchmarks/comm-rtx-3090-qwen3-32b-q4-tg
verifiedmedium · 0.75
RTX 3090 Qwen3-8B Q4_K token-gen @4K ctx
115.3 tok/s TG batch-1 @4K ctxQwen3-8B on nvidia-rtx-3090 via llama-cppbenchmarks/comm-rtx-3090-qwen3-8b-q4-tg-4k
verifiedmedium · 0.75
RTX 3090 Ti Qwen3-8B Q4 token-gen @4K (Hardware Corner)
124.0 tok/sQwen3-8B on nvidia-rtx-3090-ti via llama-cppbenchmarks/comm-rtx-3090-ti-qwen3-8b-q4-tg-4k
verifiedmedium · 0.75
RTX 4060 Ti 16GB llama.cpp: gpt-oss 20B MXFP4 decode at 128k ctx (Hardware Corner)
31.1 tok/s token generation, batch 1, per-request, ctx 128kgpt-oss 20B on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc-4060ti-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
RTX 4060 Ti 16GB llama.cpp: gpt-oss 20B MXFP4 prefill at 128k ctx (Hardware Corner)
780.2 tok/s prompt processing, batch 1, per-request, ctx 128kgpt-oss 20B on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc-4060ti-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
RTX 4060 Ti 16GB llama.cpp: gpt-oss 20B MXFP4 prefill at 16k ctx (Hardware Corner)
2552.9 tok/s prompt processing, batch 1, per-request, ctx 16kgpt-oss 20B on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc-4060ti-gptoss-20b-mxfp4-16k-pp
verifiedlow · 0.60
RTX 4060 Ti 16GB llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
27.4 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc-4060ti-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 4060 Ti 16GB llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
917.6 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc-4060ti-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 4060 Ti 16GB llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
1645.7 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4060-ti via llama-cppbenchmarks/hc-4060ti-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 4070 llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
42.5 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4070 via llama-cppbenchmarks/hc-4070-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 4070 llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
1355.8 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-4070 via llama-cppbenchmarks/hc-4070-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 4070 llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
2099.9 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4070 via llama-cppbenchmarks/hc-4070-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 4070 llama.cpp: Qwen3 8B Q4_K decode at 4k ctx (Hardware Corner)
71.2 tok/s token generation, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-4070 via llama-cppbenchmarks/hc-4070-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
RTX 4070 llama.cpp: Qwen3 8B Q4_K prefill at 16k ctx (Hardware Corner)
2064.3 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 8B on nvidia-rtx-4070 via llama-cppbenchmarks/hc-4070-qwen3-8b-q4k-16k-pp
verifiedlow · 0.60
RTX 4070 llama.cpp: Qwen3 8B Q4_K prefill at 4k ctx (Hardware Corner)
3564.1 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-4070 via llama-cppbenchmarks/hc-4070-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
RTX 4070 SUPER llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
45.5 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4070-super via llama-cppbenchmarks/hc-4070s-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 4070 SUPER llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
1578.1 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-4070-super via llama-cppbenchmarks/hc-4070s-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 4070 SUPER llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
2522.2 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4070-super via llama-cppbenchmarks/hc-4070s-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 4070 SUPER llama.cpp: Qwen3 8B Q4_K decode at 4k ctx (Hardware Corner)
75.4 tok/s token generation, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-4070-super via llama-cppbenchmarks/hc-4070s-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
RTX 4070 SUPER llama.cpp: Qwen3 8B Q4_K prefill at 16k ctx (Hardware Corner)
2525.8 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 8B on nvidia-rtx-4070-super via llama-cppbenchmarks/hc-4070s-qwen3-8b-q4k-16k-pp
verifiedlow · 0.60
RTX 4070 SUPER llama.cpp: Qwen3 8B Q4_K prefill at 4k ctx (Hardware Corner)
4321.7 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 8B on nvidia-rtx-4070-super via llama-cppbenchmarks/hc-4070s-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
RTX 4070 Ti llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
1951.0 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-4070-ti via llama-cppbenchmarks/hc-4070ti-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B MXFP4 decode at 128k ctx (Hardware Corner)
57.5 tok/s token generation, batch 1, per-request, ctx 128kgpt-oss 20B on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc-4070tis-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B MXFP4 prefill at 128k ctx (Hardware Corner)
993.5 tok/s prompt processing, batch 1, per-request, ctx 128kgpt-oss 20B on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc-4070tis-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
RTX 4070 Ti SUPER llama.cpp: gpt-oss 20B MXFP4 prefill at 16k ctx (Hardware Corner)
4182.7 tok/s prompt processing, batch 1, per-request, ctx 16kgpt-oss 20B on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc-4070tis-gptoss-20b-mxfp4-16k-pp
verifiedlow · 0.60
RTX 4070 Ti SUPER llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
58.1 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc-4070tis-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 4070 Ti SUPER llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
2003.4 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc-4070tis-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 4070 Ti SUPER llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
3048.0 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4070-ti-super via llama-cppbenchmarks/hc-4070tis-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 4080 llama.cpp: gpt-oss 20B (MXFP4) pp at 16k (Hardware Corner)
4329.2 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-gptoss-20b-pp-16k
verifiedmedium · 0.65
RTX 4080 llama.cpp: gpt-oss 20B (MXFP4) pp at 32k (Hardware Corner)
2831.1 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-gptoss-20b-pp-32k
verifiedmedium · 0.65
RTX 4080 llama.cpp: gpt-oss 20B (MXFP4) pp at 64k (Hardware Corner)
1559.0 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-gptoss-20b-pp-64k
verifiedmedium · 0.65
RTX 4080 llama.cpp: gpt-oss 20B (MXFP4) tg at 16k (Hardware Corner)
120.2 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-gptoss-20b-tg-16k
verifiedmedium · 0.65
RTX 4080 llama.cpp: gpt-oss 20B (MXFP4) tg at 32k (Hardware Corner)
106.0 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-gptoss-20b-tg-32k
verifiedmedium · 0.65
RTX 4080 llama.cpp: gpt-oss 20B (MXFP4) tg at 64k (Hardware Corner)
81.3 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-gptoss-20b-tg-64k
verifiedmedium · 0.65
RTX 4080 llama.cpp: Qwen3 14B (Q4_K) pp at 16k (Hardware Corner)
2295.1 tok/s (prompt processing, per-request, batch 1)Qwen3 14B (Q4_K) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-qwen3-14b-pp-16k
verifiedmedium · 0.65
RTX 4080 llama.cpp: Qwen3 14B (Q4_K) tg at 16k (Hardware Corner)
51.2 tok/s (token generation, per-request, batch 1)Qwen3 14B (Q4_K) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-qwen3-14b-tg-16k
verifiedmedium · 0.65
RTX 4080 llama.cpp: Qwen3 8B (Q4_K) pp at 16k (Hardware Corner)
3809.8 tok/s (prompt processing, per-request, batch 1)Qwen3 8B (Q4_K) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-qwen3-8b-pp-16k
verifiedmedium · 0.65
RTX 4080 llama.cpp: Qwen3 8B (Q4_K) pp at 32k (Hardware Corner)
1968.4 tok/s (prompt processing, per-request, batch 1)Qwen3 8B (Q4_K) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-qwen3-8b-pp-32k
verifiedmedium · 0.65
RTX 4080 llama.cpp: Qwen3 8B (Q4_K) tg at 16k (Hardware Corner)
77.9 tok/s (token generation, per-request, batch 1)Qwen3 8B (Q4_K) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-qwen3-8b-tg-16k
verifiedmedium · 0.65
RTX 4080 llama.cpp: Qwen3 8B (Q4_K) tg at 32k (Hardware Corner)
59.0 tok/s (token generation, per-request, batch 1)Qwen3 8B (Q4_K) on nvidia-rtx-4080 via llama-cppbenchmarks/hc26-4080-qwen3-8b-tg-32k
verifiedmedium · 0.65
RTX 4080 SUPER llama.cpp: gpt-oss 20B MXFP4 decode at 128k ctx (Hardware Corner)
60.0 tok/s token generation, batch 1, per-request, ctx 128kgpt-oss 20B on nvidia-rtx-4080-super via llama-cppbenchmarks/hc-4080s-gptoss-20b-mxfp4-128k-tg
verifiedlow · 0.60
RTX 4080 SUPER llama.cpp: gpt-oss 20B MXFP4 prefill at 128k ctx (Hardware Corner)
1145.0 tok/s prompt processing, batch 1, per-request, ctx 128kgpt-oss 20B on nvidia-rtx-4080-super via llama-cppbenchmarks/hc-4080s-gptoss-20b-mxfp4-128k-pp
verifiedlow · 0.60
RTX 4080 SUPER llama.cpp: gpt-oss 20B MXFP4 prefill at 16k ctx (Hardware Corner)
4708.7 tok/s prompt processing, batch 1, per-request, ctx 16kgpt-oss 20B on nvidia-rtx-4080-super via llama-cppbenchmarks/hc-4080s-gptoss-20b-mxfp4-16k-pp
verifiedlow · 0.60
RTX 4080 SUPER llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
64.2 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4080-super via llama-cppbenchmarks/hc-4080s-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 4080 SUPER llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
2526.7 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on nvidia-rtx-4080-super via llama-cppbenchmarks/hc-4080s-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 4080 SUPER llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
3745.0 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on nvidia-rtx-4080-super via llama-cppbenchmarks/hc-4080s-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 4090 gpt-oss-20B token-gen @128K context (Hardware Corner)
70.0 tok/sgpt-oss-20B on nvidia-rtx-4090 via llama-cppbenchmarks/comm-rtx-4090-gpt-oss-20b-tg-128k
verifiedmedium · 0.75
RTX 4090 Qwen3-8B Q4 token-gen @4K
141.0 tok/s TG batch-1 @4K ctxQwen3-8B on nvidia-rtx-4090 via llama-cppbenchmarks/comm-rtx-4090-qwen3-8b-q4-tg-4k
verifiedmedium · 0.75
RTX 5070 llama.cpp: Qwen3 14B Q4_K decode at 16k ctx (Hardware Corner)
40.6 tok/s token generation, batch 1, per-request, ctx 16kQwen3 14B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-14b-q4k-16k-tg
verifiedlow · 0.60
RTX 5070 llama.cpp: Qwen3 14B Q4_K decode at 4k ctx (Hardware Corner)
54.2 tok/s token generation, batch 1, per-request, ctx 4kQwen3 14B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-14b-q4k-4k-tg
verifiedlow · 0.60
RTX 5070 llama.cpp: Qwen3 14B Q4_K prefill at 16k ctx (Hardware Corner)
1315.2 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 14B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-14b-q4k-16k-pp
verifiedlow · 0.60
RTX 5070 llama.cpp: Qwen3 14B Q4_K prefill at 4k ctx (Hardware Corner)
2144.2 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 14B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-14b-q4k-4k-pp
verifiedlow · 0.60
RTX 5070 llama.cpp: Qwen3 8B Q4_K decode at 4k ctx (Hardware Corner)
85.8 tok/s token generation, batch 1, per-request, ctx 4kQwen3 8B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-8b-q4k-4k-tg
verifiedlow · 0.60
RTX 5070 llama.cpp: Qwen3 8B Q4_K prefill at 16k ctx (Hardware Corner)
1600.8 tok/s prompt processing, batch 1, per-request, ctx 16kQwen3 8B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-8b-q4k-16k-pp
verifiedlow · 0.60
RTX 5070 llama.cpp: Qwen3 8B Q4_K prefill at 4k ctx (Hardware Corner)
3487.7 tok/s prompt processing, batch 1, per-request, ctx 4kQwen3 8B on local-nvidia-rtx-5070 via llama-cppbenchmarks/hc-5070-qwen3-8b-q4k-4k-pp
verifiedlow · 0.60
RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) pp at 16k (Hardware Corner)
7560.5 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-pp-16k
verifiedmedium · 0.65
RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) pp at 32k (Hardware Corner)
5906.9 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-pp-32k
verifiedmedium · 0.65
RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) pp at 64k (Hardware Corner)
3041.1 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-pp-64k
verifiedmedium · 0.65
RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) tg at 16k (Hardware Corner)
161.3 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-tg-16k
verifiedmedium · 0.65
RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) tg at 32k (Hardware Corner)
149.3 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-tg-32k
verifiedmedium · 0.65
RTX 5080 llama.cpp: gpt-oss 20B (MXFP4) tg at 64k (Hardware Corner)
130.9 tok/s (token generation, per-request, batch 1)gpt-oss 20B (MXFP4) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-gptoss-20b-tg-64k
verifiedmedium · 0.65
RTX 5080 llama.cpp: Qwen3 14B (Q4_K) pp at 16k (Hardware Corner)
2542.0 tok/s (prompt processing, per-request, batch 1)Qwen3 14B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-14b-pp-16k
verifiedmedium · 0.65
RTX 5080 llama.cpp: Qwen3 14B (Q4_K) tg at 16k (Hardware Corner)
64.0 tok/s (token generation, per-request, batch 1)Qwen3 14B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-14b-tg-16k
verifiedmedium · 0.65
RTX 5080 llama.cpp: Qwen3 8B (Q4_K) pp at 16k (Hardware Corner)
4024.4 tok/s (prompt processing, per-request, batch 1)Qwen3 8B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-8b-pp-16k
verifiedmedium · 0.65
RTX 5080 llama.cpp: Qwen3 8B (Q4_K) pp at 32k (Hardware Corner)
1943.3 tok/s (prompt processing, per-request, batch 1)Qwen3 8B (Q4_K) on nvidia-rtx-5080 via llama-cppbenchmarks/hc26-5080-qwen3-8b-pp-32k
verifiedmedium · 0.65
RTX 5090 32GB, llama.cpp CUDA: Qwen3-32B Q4 decode tok/s @4K ctx (Hardware Corner measured)
61.4 decode tok/s (TG, single-stream, batch-1, 4K context)Qwen3-32B Q4 (GGUF) on nvidia-rtx-5090 via llama-cppbenchmarks/comm-rtx5090-qwen3-32b-q4-decode
verifiedhigh · 0.85
RTX 5090 gpt-oss-120B MXFP4 token-gen @128K
112.0 tok/s TG batch-1 @131K ctxgpt-oss-120B on nvidia-rtx-5090 via llama-cppbenchmarks/comm-rtx-5090-gpt-oss-120b-mxfp4-tg-128k
verifiedmedium · 0.75
RTX 5090 gpt-oss-20B token-gen @128K context (Hardware Corner)
112.0 tok/sgpt-oss-20B on nvidia-rtx-5090 via llama-cppbenchmarks/comm-rtx-5090-gpt-oss-20b-tg-128k
verifiedmedium · 0.75
RTX 5090 llama.cpp: gpt-oss 20B Q4_K_XL decode at 4k ctx (Hardware Corner)
298.2 tok/s (tg128, batch 1, per-request)gpt-oss 20B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-gpt-oss-20b-q4kxl-4k-tg128
verifiedmedium · 0.70
RTX 5090 llama.cpp: gpt-oss 20B Q4_K_XL prefill at 4k ctx (Hardware Corner)
9443.81 tok/s (pp~1024-2048 at ctx depth)gpt-oss 20B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-gpt-oss-20b-q4kxl-4k-pp1024
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 14B Q4_K_XL decode at 4k ctx (Hardware Corner)
123.79 tok/s (tg128, batch 1, per-request)Qwen3 14B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-14b-q4kxl-4k-tg128
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 14B Q4_K_XL prefill at 4k ctx (Hardware Corner)
6497.55 tok/s (pp~1024-2048 at ctx depth)Qwen3 14B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-14b-q4kxl-4k-pp1024
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 30B-A3B (MoE) Q4_K_XL decode at 4k ctx (Hardware Corner)
234.3 tok/s (tg128, batch 1, per-request)Qwen3 30B-A3B (MoE) on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-30b-a3b-q4kxl-4k-tg128
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 30B-A3B (MoE) Q4_K_XL prefill at 4k ctx (Hardware Corner)
6630.42 tok/s (pp~1024-2048 at ctx depth)Qwen3 30B-A3B (MoE) on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-30b-a3b-q4kxl-4k-pp1024
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 32B dense Q4_K_XL decode at 4k ctx (Hardware Corner)
61.38 tok/s (tg128, batch 1, per-request)Qwen3 32B dense on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-32b-q4kxl-4k-tg128
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 32B dense Q4_K_XL prefill at 4k ctx (Hardware Corner)
2931.32 tok/s (pp~1024-2048 at ctx depth)Qwen3 32B dense on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-32b-q4kxl-4k-pp1024
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 8B Q4_K_XL decode at 32k ctx (Hardware Corner)
111.91 tok/s (tg128, batch 1, per-request)Qwen3 8B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-8b-q4kxl-32k-tg128
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 8B Q4_K_XL decode at 4k ctx (Hardware Corner)
185.91 tok/s (tg128, batch 1, per-request)Qwen3 8B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-8b-q4kxl-4k-tg128
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 8B Q4_K_XL prefill at 32k ctx (Hardware Corner)
3687.76 tok/s (pp~1024-2048 at ctx depth)Qwen3 8B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-8b-q4kxl-32k-pp1024
verifiedmedium · 0.70
RTX 5090 llama.cpp: Qwen3 8B Q4_K_XL prefill at 4k ctx (Hardware Corner)
10406.47 tok/s (pp~1024-2048 at ctx depth)Qwen3 8B on nvidia-rtx-5090 via llama-cppbenchmarks/hc-rtx5090-qwen3-8b-q4kxl-4k-pp1024
verifiedmedium · 0.70
RTX 5090 Qwen3-8B Q4_K_XL token-gen @4K
185.9 tok/s TG batch-1 @4K ctxQwen3-8B on nvidia-rtx-5090 via llama-cppbenchmarks/comm-rtx-5090-qwen3-8b-q4-tg-4k
verifiedmedium · 0.75
RTX A4000 Ollama 0.5.11: deepseek-coder-v2:16b per-request eval rate
22.89 tok/s (per-request decode, batch size 1)deepseek-coder-v2-16b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-deepseek-coder-v2-16b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: deepseek-r1:14b per-request eval rate
30.2 tok/s (per-request decode, batch size 1)deepseek-r1-14b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-deepseek-r1-14b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: deepseek-r1:7b per-request eval rate
52.61 tok/s (per-request decode, batch size 1)deepseek-r1-7b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-deepseek-r1-7b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: deepseek-r1:8b per-request eval rate
51.6 tok/s (per-request decode, batch size 1)deepseek-r1-distill-llama-8b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-deepseek-r1-distill-llama-8b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: gemma2:27b per-request eval rate
2.38 tok/s (per-request decode, batch size 1)gemma-2-27b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-gemma-2-27b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: gemma2:9b per-request eval rate
39.04 tok/s (per-request decode, batch size 1)gemma-2-9b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-gemma-2-9b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: llama2:13b per-request eval rate
38.46 tok/s (per-request decode, batch size 1)llama-2-13b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-llama-2-13b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: llama2:7b per-request eval rate
65.06 tok/s (per-request decode, batch size 1)llama-2-7b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-llama-2-7b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: llama3.1:8b per-request eval rate
51.35 tok/s (per-request decode, batch size 1)llama-3.1-8b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-llama-31-8b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: mistral:7b per-request eval rate
64.16 tok/s (per-request decode, batch size 1)mistral-7b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-mistral-7b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: qwen2.5:14b per-request eval rate
30.05 tok/s (per-request decode, batch size 1)qwen2.5-14b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-qwen25-14b-eval
verifiedlow · 0.60
RTX A4000 Ollama 0.5.11: qwen2.5:7b per-request eval rate
52.68 tok/s (per-request decode, batch size 1)qwen2.5-7b on nvidia-rtx-a4000 via ollamabenchmarks/dm-a4000-ollama-qwen25-7b-eval
verifiedlow · 0.60
RTX A6000 vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 50 concurrent
1218.63 tok/s (aggregate output tokens, 50 concurrent)llama-3-1-8b on nvidia-rtx-a6000 via vllmbenchmarks/dm-a6000-vllm-llama-3-1-8b-c50-out
verifiedlow · 0.60
RTX A6000 vLLM: meta-llama/Llama-3.1-8B-Instruct median TTFT, 50 concurrent
593.26 ms (median time to first token, 50 concurrent)llama-3-1-8b on nvidia-rtx-a6000 via vllmbenchmarks/dm-a6000-vllm-llama-3-1-8b-c50-ttft
verifiedlow · 0.60
RTX A6000 vLLM: Qwen/Qwen2.5-14B-Instruct aggregate output throughput, 50 concurrent
695.51 tok/s (aggregate output tokens, 50 concurrent)qwen2.5-14b-instruct on nvidia-rtx-a6000 via vllmbenchmarks/dm-a6000-vllm-qwen25-14b-instruct-c50-out
verifiedlow · 0.60
RTX A6000 vLLM: Qwen/Qwen2.5-14B-Instruct median TTFT, 50 concurrent
1095.08 ms (median time to first token, 50 concurrent)qwen2.5-14b-instruct on nvidia-rtx-a6000 via vllmbenchmarks/dm-a6000-vllm-qwen25-14b-instruct-c50-ttft
verifiedlow · 0.60
RTX A6000 vLLM: Qwen/Qwen2.5-7B-Instruct aggregate output throughput, 50 concurrent
1337.24 tok/s (aggregate output tokens, 50 concurrent)qwen2.5-7b-instruct on nvidia-rtx-a6000 via vllmbenchmarks/dm-a6000-vllm-qwen25-7b-instruct-c50-out
verifiedlow · 0.60
RTX A6000 vLLM: Qwen/Qwen2.5-7B-Instruct median TTFT, 50 concurrent
583.54 ms (median time to first token, 50 concurrent)qwen2.5-7b-instruct on nvidia-rtx-a6000 via vllmbenchmarks/dm-a6000-vllm-qwen25-7b-instruct-c50-ttft
verifiedlow · 0.60
RTX PRO 2000 Blackwell Ollama: gpt-oss:20b eval rate
61.51 tok/s (eval rate, per-request, batch 1)gpt-oss-20b on nvidia-rtx-pro-2000-blackwell via ollamabenchmarks/dm-pro2000-ollama-gptoss-20b-eval
verifiedlow · 0.60
RTX PRO 2000 Blackwell Ollama: llama3.1:8b eval rate
59.05 tok/s (eval rate, per-request, batch 1)llama-3-1-8b on nvidia-rtx-pro-2000-blackwell via ollamabenchmarks/dm-pro2000-ollama-llama31-8b-eval
verifiedlow · 0.60
RTX PRO 2000 Blackwell Ollama: qwen3:8b eval rate
48.75 tok/s (eval rate, per-request, batch 1)qwen3-8b on nvidia-rtx-pro-2000-blackwell via ollamabenchmarks/dm-pro2000-ollama-qwen3-8b-eval
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: google/gemma-3-12b-it aggregate output throughput, 1000 requests
512.92 tok/s (aggregate output tokens, 1000 concurrent)gemma-3-12b-it on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-gemma-3-12b-it-c1000-out
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: google/gemma-3-12b-it median TTFT, 1000 requests
16642.44 ms (median time to first token, 1000 concurrent)gemma-3-12b-it on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-gemma-3-12b-it-c1000-ttft
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 1000 requests
943.99 tok/s (aggregate output tokens, 1000 concurrent)llama-3-1-8b on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-llama-3-1-8b-c1000-out
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: meta-llama/Llama-3.1-8B-Instruct median TTFT, 1000 requests
9976.3 ms (median time to first token, 1000 concurrent)llama-3-1-8b on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-llama-3-1-8b-c1000-ttft
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: openai/gpt-oss-20b aggregate output throughput, 1000 requests
1037.61 tok/s (aggregate output tokens, 1000 concurrent)gpt-oss-20b on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-gpt-oss-20b-c1000-out
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: openai/gpt-oss-20b median TTFT, 1000 requests
10275.87 ms (median time to first token, 1000 concurrent)gpt-oss-20b on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-gpt-oss-20b-c1000-ttft
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: Qwen/Qwen3-8B aggregate output throughput, 1000 requests
941.32 tok/s (aggregate output tokens, 1000 concurrent)qwen3-8b on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-qwen3-8b-c1000-out
verifiedlow · 0.60
RTX PRO 2000 Blackwell vLLM: Qwen/Qwen3-8B median TTFT, 1000 requests
9595.67 ms (median time to first token, 1000 concurrent)qwen3-8b on nvidia-rtx-pro-2000-blackwell via vllmbenchmarks/dm-pro2000-vllm-qwen3-8b-c1000-ttft
verifiedlow · 0.60
RTX Pro 4000 Blackwell Ollama: Llama 3.1 8B eval rate (Computingforgeeks)
96.8 tok/s (eval rate, per-request)llama3.1:8b on nvidia-rtx-pro-4000-blackwell via ollamabenchmarks/cfgk-pro4000-ollama-llama31-8b-eval
verifiedlow · 0.55
RTX Pro 4000 Blackwell Ollama: Qwen2.5 32B eval rate (Computingforgeeks)
24.6 tok/s (eval rate, per-request)qwen2.5:32b on nvidia-rtx-pro-4000-blackwell via ollamabenchmarks/cfgk-pro4000-ollama-qwen25-32b-eval
verifiedlow · 0.55
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B decode at 4k (Hardware Corner)
213.4 tok/s (token generation, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-gptoss-20b-4k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B pp at 128k (Hardware Corner)
3163.3 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-pp-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B pp at 16k (Hardware Corner)
8747.7 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-pp-16k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B pp at 32k (Hardware Corner)
7257.7 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-pp-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B pp at 64k (Hardware Corner)
5273.9 tok/s (prompt processing, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-pp-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B prefill at 4k (Hardware Corner)
10339.6 tok/s (prompt processing, per-request)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-gptoss-20b-4k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B tg at 128k (Hardware Corner)
130.9 tok/s (token generation, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-tg-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B tg at 16k (Hardware Corner)
196.2 tok/s (token generation, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-tg-16k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B tg at 32k (Hardware Corner)
182.8 tok/s (token generation, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-tg-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: gpt-oss 20B tg at 64k (Hardware Corner)
161.4 tok/s (token generation, per-request, batch 1)gpt-oss 20B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-gptoss-20b-tg-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Llama 3.3 70B decode at 16k (Hardware Corner)
22.4 tok/s (token generation, per-request, batch 1)Llama 3.3 70B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-llama33-70b-16k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Llama 3.3 70B decode at 4k (Hardware Corner)
24.0 tok/s (token generation, per-request, batch 1)Llama 3.3 70B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-llama33-70b-4k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Llama 3.3 70B prefill at 16k (Hardware Corner)
826.3 tok/s (prompt processing, per-request)Llama 3.3 70B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-llama33-70b-16k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Llama 3.3 70B prefill at 4k (Hardware Corner)
1022.9 tok/s (prompt processing, per-request)Llama 3.3 70B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-llama33-70b-4k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B decode at 16k (Hardware Corner)
84.3 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-14b-16k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B decode at 4k (Hardware Corner)
98.9 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-14b-4k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B pp at 128k (Hardware Corner)
434.2 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-14b-pp-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B pp at 32k (Hardware Corner)
2069.8 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-14b-pp-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B pp at 64k (Hardware Corner)
1125.1 tok/s (prompt processing, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-14b-pp-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B prefill at 16k (Hardware Corner)
3064.2 tok/s (prompt processing, per-request)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-14b-16k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B prefill at 4k (Hardware Corner)
4515.7 tok/s (prompt processing, per-request)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-14b-4k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B tg at 128k (Hardware Corner)
36.5 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-14b-tg-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B tg at 32k (Hardware Corner)
70.8 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-14b-tg-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 14B tg at 64k (Hardware Corner)
53.8 tok/s (token generation, per-request, batch 1)Qwen3 14B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-14b-tg-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) decode at 16k (Hardware Corner)
148.5 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-30b-a3b-16k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) decode at 4k (Hardware Corner)
188.8 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-30b-a3b-4k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) pp at 128k (Hardware Corner)
702.7 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-pp-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) pp at 256k (Hardware Corner)
364.1 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-pp-256k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) pp at 32k (Hardware Corner)
3032.7 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-pp-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) pp at 64k (Hardware Corner)
1708.9 tok/s (prompt processing, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-pp-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) prefill at 16k (Hardware Corner)
4043.8 tok/s (prompt processing, per-request)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-30b-a3b-16k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) prefill at 4k (Hardware Corner)
5266.1 tok/s (prompt processing, per-request)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-30b-a3b-4k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) tg at 128k (Hardware Corner)
61.4 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-tg-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) tg at 256k (Hardware Corner)
37.0 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-tg-256k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) tg at 32k (Hardware Corner)
123.2 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-tg-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 30B-A3B (MoE) tg at 64k (Hardware Corner)
91.9 tok/s (token generation, per-request, batch 1)Qwen3 30B-A3B (MoE) on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-30ba3b-tg-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B decode at 4k (Hardware Corner)
46.6 tok/s (token generation, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-32b-4k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B pp at 16k (Hardware Corner)
1497.7 tok/s (prompt processing, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-32b-pp-16k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B pp at 32k (Hardware Corner)
1079.8 tok/s (prompt processing, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-32b-pp-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B pp at 64k (Hardware Corner)
629.6 tok/s (prompt processing, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-32b-pp-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B prefill at 4k (Hardware Corner)
2030.3 tok/s (prompt processing, per-request)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-32b-4k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B tg at 16k (Hardware Corner)
42.0 tok/s (token generation, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-32b-tg-16k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B tg at 32k (Hardware Corner)
36.4 tok/s (token generation, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-32b-tg-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 32B tg at 64k (Hardware Corner)
28.9 tok/s (token generation, per-request, batch 1)Qwen3 32B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-32b-tg-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B decode at 16k (Hardware Corner)
126.3 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-8b-16k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B decode at 4k (Hardware Corner)
159.8 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-8b-4k-tg
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B pp at 128k (Hardware Corner)
899.1 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-8b-pp-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B pp at 32k (Hardware Corner)
4006.2 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-8b-pp-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B pp at 64k (Hardware Corner)
2120.7 tok/s (prompt processing, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-8b-pp-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B prefill at 16k (Hardware Corner)
5846.4 tok/s (prompt processing, per-request)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-8b-16k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B prefill at 4k (Hardware Corner)
8175.3 tok/s (prompt processing, per-request)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc-rtxpro5000-qwen3-8b-4k-pp
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B tg at 128k (Hardware Corner)
38.1 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-8b-tg-128k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B tg at 32k (Hardware Corner)
98.3 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-8b-tg-32k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell llama.cpp: Qwen3 8B tg at 64k (Hardware Corner)
64.1 tok/s (token generation, per-request, batch 1)Qwen3 8B on nvidia-rtx-pro-5000-blackwell via llama-cppbenchmarks/hc26-pro5000-qwen3-8b-tg-64k
verifiedmedium · 0.65
RTX PRO 5000 Blackwell Ollama: gpt-oss 20B eval rate (Database Mart)
169.41 tok/s (eval rate, per-request)gpt-oss:20b on nvidia-rtx-pro-5000-blackwell via ollamabenchmarks/dm-pro5000-ollama-gptoss20b-eval
verifiedlow · 0.60
RTX PRO 5000 Blackwell Ollama: Llama 3.3 70B eval rate (Database Mart)
21.42 tok/s (eval rate, per-request)llama3.3:70b on nvidia-rtx-pro-5000-blackwell via ollamabenchmarks/dm-pro5000-ollama-llama33-70b-eval
verifiedlow · 0.60
RTX PRO 5000 Blackwell Ollama: Qwen2.5 14B eval rate (Database Mart)
86.49 tok/s (eval rate, per-request)qwen2.5:14b on nvidia-rtx-pro-5000-blackwell via ollamabenchmarks/dm-pro5000-ollama-qwen25-14b-eval
verifiedlow · 0.60
RTX Pro 5000 Blackwell Ollama: Qwen2.5 32B eval rate (Computingforgeeks)
42.9 tok/s (eval rate, per-request)qwen2.5:32b on nvidia-rtx-pro-5000-blackwell via ollamabenchmarks/cfgk-pro5000-ollama-qwen25-32b-eval
verifiedlow · 0.55
RTX PRO 5000 Blackwell Ollama: Qwen2.5 32B eval rate (Database Mart)
42.92 tok/s (eval rate, per-request)qwen2.5:32b on nvidia-rtx-pro-5000-blackwell via ollamabenchmarks/dm-pro5000-ollama-qwen25-32b-eval
verifiedlow · 0.60
RTX Pro 5000 Blackwell Ollama: Qwen2.5 72B eval rate (Computingforgeeks)
18.4 tok/s (eval rate, per-request)qwen2.5:72b on nvidia-rtx-pro-5000-blackwell via ollamabenchmarks/cfgk-pro5000-ollama-qwen25-72b-eval
verifiedlow · 0.55
RTX PRO 5000 Blackwell vLLM: google/gemma-3-12b-it aggregate output throughput, 1000 requests
1706.59 tok/s (aggregate output tokens, 1000 concurrent)gemma-3-12b-it on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-gemma-3-12b-it-c1000-out
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: google/gemma-3-12b-it median TTFT, 1000 requests
5338.65 ms (median time to first token, 1000 concurrent)gemma-3-12b-it on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-gemma-3-12b-it-c1000-ttft
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 1000 requests
2878.77 tok/s (aggregate output tokens, 1000 concurrent)llama-3-1-8b on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-llama-3-1-8b-c1000-out
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: meta-llama/Llama-3.1-8B-Instruct median TTFT, 1000 requests
3605.52 ms (median time to first token, 1000 concurrent)llama-3-1-8b on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-llama-3-1-8b-c1000-ttft
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: openai/gpt-oss-20b aggregate output throughput, 1000 requests
2113.77 tok/s (aggregate output tokens, 1000 concurrent)gpt-oss-20b on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-gpt-oss-20b-c1000-out
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: openai/gpt-oss-20b median TTFT, 1000 requests
2747.92 ms (median time to first token, 1000 concurrent)gpt-oss-20b on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-gpt-oss-20b-c1000-ttft
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: Qwen/Qwen3-8B aggregate output throughput, 1000 requests
2743.63 tok/s (aggregate output tokens, 1000 concurrent)qwen3-8b on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-qwen3-8b-c1000-out
verifiedlow · 0.60
RTX PRO 5000 Blackwell vLLM: Qwen/Qwen3-8B median TTFT, 1000 requests
3839.02 ms (median time to first token, 1000 concurrent)qwen3-8b on nvidia-rtx-pro-5000-blackwell via vllmbenchmarks/dm-pro5000-vllm-qwen3-8b-c1000-ttft
verifiedlow · 0.60
RTX PRO 6000 Blackwell gpt-oss-120B Q8 token-gen @4K (Hardware Corner)
221.0 tok/sgpt-oss-120B on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/comm-rtx-pro-6000-gpt-oss-120b-q8-tg-4k
verifiedmedium · 0.75
RTX PRO 6000 Blackwell Llama-3.3-70B Q4 token-gen @4K
32.0 tok/s TG batch-1 @4K ctxLlama-3.3-70B on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/comm-rtx-pro-6000-llama33-70b-q4-tg-4k
verifiedmedium · 0.75
RTX PRO 6000 Blackwell Ollama 0.13.5: gpt-oss:120b eval rate
134.28 tok/s (eval rate, per-request, batch 1)gpt-oss-120b on nvidia-rtx-pro-6000-blackwell via ollamabenchmarks/dm-pro6000-ollama-gptoss-120b-eval
verifiedlow · 0.60
RTX PRO 6000 Blackwell Ollama 0.13.5: gpt-oss:20b eval rate
185.09 tok/s (eval rate, per-request, batch 1)gpt-oss-20b on nvidia-rtx-pro-6000-blackwell via ollamabenchmarks/dm-pro6000-ollama-gptoss-20b-eval
verifiedlow · 0.60
RTX PRO 6000 Blackwell Ollama 0.13.5: llama3.3:70b eval rate
31.96 tok/s (eval rate, per-request, batch 1)llama-3.3-70b on nvidia-rtx-pro-6000-blackwell via ollamabenchmarks/dm-pro6000-ollama-llama33-70b-eval
verifiedlow · 0.60
RTX PRO 6000 Blackwell Qwen3-8B Q4 token-gen @4K
174.0 tok/s TG batch-1 @4K ctxQwen3-8B on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/comm-rtx-pro-6000-qwen3-8b-q4-tg-4k
verifiedmedium · 0.75
RTX PRO 6000 Blackwell Qwen3.5-122B Q4 token-gen (Hardware Corner)
98.0 tok/sQwen3.5-122B on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/comm-rtx-pro-6000-qwen35-122b-q4-tg-4k
verifiedmedium · 0.70
RX 7900 XTX (GPU0) llama.cpp ROCm 7.1.1: Llama-2-7B Q4_0 FA-off pp512
3487.15 tok/s (pp512, per-request, batch 1, one GPU)llama-2-7b Q4_0 (3.56 GiB, 6.74B) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/rocm711-rx7900xtx-gpu0-llama27b-q40-nofa-pp512
verifiedlow · 0.60
RX 7900 XTX (GPU0) llama.cpp ROCm 7.1.1: Llama-2-7B Q4_0 FA-off tg128
118.82 tok/s (tg128, per-request, batch 1, one GPU)llama-2-7b Q4_0 (3.56 GiB, 6.74B) on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/rocm711-rx7900xtx-gpu0-llama27b-q40-nofa-tg128
verifiedlow · 0.60
RX 7900 XTX llama2-7B Q4_0 token-gen (ROCm)
118.9 tok/s TG batch-1Llama-2-7B on amd-radeon-rx-7900-xtx via llama-cppbenchmarks/comm-rx-7900-xtx-llama2-7b-q4-tg
verifiedmedium · 0.75
RX 9070 XT: Qwen3 8B Q4_K_M decode tok/s, batch 1 (Ollama, ROCm)
96.7 tok/sQwen3 8B on amd-radeon-rx-9070-xt via ollamabenchmarks/local-rx9070xt-qwen3-8b-q4-decode-b1
verifiedmedium · 0.70
Ryzen AI 9 HX 370 (dual-channel) Qwen2.5-32B Q4 decode
3.5 tok/s decode, single-streamQwen2.5-32B Q4 dense on amd-ryzen-ai-9-hx-370 via llama-cppbenchmarks/comm-ryzen-ai-9-hx370-qwen25-32b-q4-decode
verifiedmedium · 0.75
Ryzen AI Max+ 395 gpt-oss-120b MXFP4 token-gen
33.7 tok/s decode (Vulkan AMDVLK)gpt-oss-120b on amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/comm-strix-halo-gpt-oss-120b-mxfp4-tg
verifiedmedium · 0.75
Ryzen AI Max+ 395 Llama-3.3-70B dense Q4_K_M token-gen
5.0 tok/s decode (Vulkan AMDVLK)Llama-3.3-70B on amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/comm-strix-halo-llama33-70b-dense-q4-tg
verifiedmedium · 0.75
Ryzen AI Max+ 395 Strix Halo GLM-4.5-Air UD-Q4_K_XL decode
23.4 tok/s (single-stream)GLM-4.5-Air UD-Q4_K_XL (106B, ~12B active) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/comm-strix-halo-glm45-air-udq4-decode
verifiedmedium · 0.75
Ryzen AI Max+ 395 Strix Halo Qwen3-235B-A22B UD-Q3_K_XL decode
15.1 tok/s (single-stream)Qwen3-235B-A22B UD-Q3_K_XL (235B, ~22B active) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/comm-strix-halo-qwen3-235b-a22b-udq3-decode
verifiedmedium · 0.75
SGLang Chunked PP: DeepSeek-V3.1 prefill throughput 3.31x with PP4 TP8 vs TP8 on H20
3.31 x speedup vs BASELINE: prefill throughput ratio of PP4 TP8 against TP8 alone (no pipeline parallelism) on the same H20 cluster, batch size 1, chunked prefill 12288, dynamic chunking smooth factor 0.65. NOT 3.31 tok/s.DeepSeek-V3.1 on nvidia-h20 via sglangbenchmarks/sglang-pp-deepseek-v31-prefill-throughput-128k
verifiedhigh · 0.85
sglang DeepSeek-V3 FP8 on 16x H800 (TP16, 2 nodes): 622 tok/s aggregate @96 concurrent
622.05 tok/s generated (aggregate over 96 concurrent requests, avg 253 output tok each)deepseek-ai/DeepSeek-V3 on nvidia-h800-80gb via sglangbenchmarks/sglang-h800x16-deepseekv3-fp8-c96-tps
verifiedlow · 0.55
sglang DeepSeek-V3 FP8 on 16x H800 (TP16): 266 tok/s aggregate @32 concurrent
266.11 tok/s generated (aggregate over 32 concurrent requests)deepseek-ai/DeepSeek-V3 on nvidia-h800-80gb via sglangbenchmarks/sglang-h800x16-deepseekv3-fp8-c32-tps
verifiedlow · 0.55
sglang DeepSeek-V3 FP8 on 16x H800 (TP16): 459 tok/s aggregate @64 concurrent
459.36 tok/s generated (aggregate over 64 concurrent requests)deepseek-ai/DeepSeek-V3 on nvidia-h800-80gb via sglangbenchmarks/sglang-h800x16-deepseekv3-fp8-c64-tps
verifiedlow · 0.55
SGLang on GB300 NVL72: DeepSeek-R1-NVFP4 TTFT 8.6s at 128K/8K ISL/OSL with dynamic chunking
8600 ms (TTFT, 128K input, 8K output, PD disaggregation, chunked PP, dynamic chunking)DeepSeek-R1 (NVFP4) on nvidia-b300 via sglangbenchmarks/sglang-gb300-deepseek-r1-ttft-128k
verifiedhigh · 0.85
SGLang v0.5.18 (VibeSys-ported+tuned) on 4x MI300A: Qwen3.5-397B-A17B MXFP4 goodput 961 tok/s @48sess
961.0 tok/s goodput (aggregate over 48 concurrent sessions, SLO-bound)Qwen3.5-397B-A17B on amd-instinct-mi300a via sglangbenchmarks/syfi-mi300a-sglang-tuned-qwen35-397b-mxfp4-goodput
verifiedmedium · 0.80
SGLang: DeepSeek 671B PD-disaggregated + large-scale EP on GB200 NVL72, 7583 decode tokens/s per GPU
7583 decode tokens/s per GPU (aggregate decode throughput per GPU across the rack)DeepSeek-V3 (671B MoE, FP8) on nvidia-gb200-nvl72 via sglangbenchmarks/sglang-deepseek-v3-gb200-nvl72-decode-tps-per-gpu
verifiedmedium · 0.80
SGLang: DeepSeek-V3 671B FP8 on 96x H100 with PD disaggregation + large-scale EP, 22300 output tokens/s per node
22300 output (decode) tokens/s per 8-GPU node (aggregate)DeepSeek-V3 (671B MoE, FP8) on nvidia-h100-sxm via sglangbenchmarks/sglang-deepseek-v3-96xh100-output-tps-per-node
verifiedmedium · 0.80
Silex (third-party genai-perf): Llama 3.3 70B Instruct bf16 on 4x H100 80GB SXM, ~7000 total tokens/s at 200/200
7000 total tokens/s (aggregate; source states '~7,000 TPS' peak at 500 concurrent users)llama-3.3-70b-instruct on nvidia-h100-sxm via tensorrt-llmbenchmarks/silex-llama33-70b-4xh100-tp4-tps-200x200
verifiedlow · 0.60
SlimSpec low-rank draft LM-head for EAGLE-3: Qwen3-30B-A3B at batch 1
1.46 speedup_ratio: 1.46x end-to-end generation speedup over vanilla autoregressive baseline across MT-Bench, HumanEval, GSM8K at temperature=1, batch size 1. BASELINE: standard autoregressive decoding without speculation.Qwen/Qwen3-30B-A3B on nvidia-h100-sxm via niche-eaglebenchmarks/sd-slimspec-eagle3-qwen3-30b-b1
verifiedhigh · 0.85
SlimSpec low-rank draft LM-head for EAGLE-3: Qwen3-30B-A3B at batch 64
1.21 speedup_ratio: 1.21x end-to-end generation speedup over vanilla autoregressive baseline across MT-Bench, HumanEval, GSM8K at temperature=1, batch size 64. BASELINE: standard autoregressive decoding without speculation at batch 64.Qwen/Qwen3-30B-A3B on nvidia-h100-sxm via niche-eaglebenchmarks/sd-slimspec-eagle3-qwen3-30b-b64
verifiedhigh · 0.85
SmoothQuant W8A8 WikiText perplexity delta vs fp16, OPT-175B
0.18 perplexity_delta (WikiText ppl points, quantized minus FP16)OPT-175B on ? via smoothquant-referencebenchmarks/smoothquant-w8a8-wikitext-ppl-delta-opt-175b
verifiedhigh · 0.90
SpecExec massively parallel speculative execution on RTX 4090 with RAM offload: Llama-2-70B batch 1
8.3 speedup_ratio: 8.3x speedup over sequential offloaded autoregressive decoding (throughput increased to 5.66 tok/s with generation rate 13.46 accepted tokens per iteration). BASELINE: standard autoregressive decode with RAM offloading on RTX 4090.meta-llama/Llama-2-70b-chat-hf on nvidia-rtx-4090 via niche-specexecbenchmarks/sd-specexec-offload-llama2-70b-b1
verifiedhigh · 0.85
SpinQuant random-vs-learned rotation accuracy spread, Mistral-7B W4A4KV4
16.2 accuracy_delta_pp (learned rotations minus random Hadamard rotations, identical W4A4KV4 setting)Mistral-7B on ? via spinquant-research-harnessbenchmarks/spinquant-random-vs-learned-rotation-spread-mistral-7b
verifiedhigh · 0.85
SpinQuant W4A4KV4 8-task zero-shot accuracy delta vs fp16, LLaMA-2 7B
2.9 accuracy_delta_pp (magnitude of the drop: average zero-shot 8-task accuracy percentage points lost vs FP16)LLaMA-2 7B on ? via spinquant-research-harnessbenchmarks/spinquant-w4a4kv4-zeroshot-accuracy-delta-llama2-7b
verifiedhigh · 0.85
SpinQuant W4A8KV8 8-task zero-shot accuracy delta vs fp16, Mistral-7B
1.6 accuracy_delta_pp (magnitude of the drop: average zero-shot 8-task accuracy percentage points lost vs FP16)Mistral-7B on ? via spinquant-research-harnessbenchmarks/spinquant-w4a8kv8-zeroshot-accuracy-delta-mistral-7b
verifiedhigh · 0.90
ss-m3ultra-256-glm5-mlx-4bit decode
16.7 tok/s (avg)GLM-5 (MLX 4-bit) on apple-m3-ultra via mlx-lmbenchmarks/ss-m3ultra-256-glm5-mlx-4bit-tg
verifiedlow · 0.55
ss-m3ultra-256-qwen35-122b-mlx-mxfp4 decode
57.0 tok/s (avg)Qwen3.5-122B-A10B (MLX MXFP4) on apple-m3-ultra via mlx-lmbenchmarks/ss-m3ultra-256-qwen35-122b-mlx-mxfp4-tg
verifiedlow · 0.55
ss-m3ultra-512-qwen3-235b-lmstudio-4bit decode
27.36 tok/s (avg)Qwen 3 235B-A22B (4bit LM Studio) on apple-m3-ultra via llama-cppbenchmarks/ss-m3ultra-512-qwen3-235b-lmstudio-4bit-tg
verifiedlow · 0.55
ss-m3ultra-512-qwen35-397b-mlx-q41 decode
40.2 tok/s (avg)Qwen3.5-397B-A17B (MLX q4.1bit) on apple-m3-ultra via mlx-lmbenchmarks/ss-m3ultra-512-qwen35-397b-mlx-q41-tg
verifiedlow · 0.55
ss-m4max-128-qwen3-235b-mlx-3bit decode
30.0 tok/s (avg)Qwen 3 235B-A22B (MLX 3-bit) on apple-m4-max via mlx-lmbenchmarks/ss-m4max-128-qwen3-235b-mlx-3bit-tg
verifiedlow · 0.55
ss-m4max-40core-qwen3-30b-mlx-q4 decode
92.09 tok/s (avg)Qwen 3 30B-A3B (MLX Q4) on apple-m4-max via mlx-lmbenchmarks/ss-m4max-40core-qwen3-30b-mlx-q4-tg
verifiedlow · 0.55
ss-m4max-40core-qwen3-30b-mlx-q4 prefill
822.59 tok/s (prompt eval)Qwen 3 30B-A3B (MLX Q4) on apple-m4-max via mlx-lmbenchmarks/ss-m4max-40core-qwen3-30b-mlx-q4-pp
verifiedlow · 0.50
ss-m4max-40core-qwen3-4b-mlx-q4 decode
148.08 tok/s (avg)Qwen 3 4B (MLX Q4) on apple-m4-max via mlx-lmbenchmarks/ss-m4max-40core-qwen3-4b-mlx-q4-tg
verifiedlow · 0.55
ss-m4max-40core-qwen3-4b-mlx-q4 prefill
2976.68 tok/s (prompt eval)Qwen 3 4B (MLX Q4) on apple-m4-max via mlx-lmbenchmarks/ss-m4max-40core-qwen3-4b-mlx-q4-pp
verifiedlow · 0.50
ss-m4max-64-gemma3-27b-lmstudio-q8 decode
14.49 tok/s (avg)Gemma 3 27B (Q8_0) on apple-m4-max via llama-cppbenchmarks/ss-m4max-64-gemma3-27b-lmstudio-q8-tg
verifiedlow · 0.55
ss-m4ultra-192-gptoss120b-mlx-q4km decode
10.0 tok/s (avg)gpt-oss 120B (Q4_K_M) on apple-m4-ultra via mlx-lmbenchmarks/ss-m4ultra-192-gptoss120b-mlx-q4km-tg
verifiedlow · 0.55
ss-m4ultra-192-llama33-70b-mlx-q4km decode
18.0 tok/s (avg)Llama 3.3 70B (Q4_K_M) on apple-m4-ultra via mlx-lmbenchmarks/ss-m4ultra-192-llama33-70b-mlx-q4km-tg
verifiedlow · 0.55
ss-m4ultra-192-llama4-scout-mlx-q4km decode
30.0 tok/s (avg)Llama 4 Scout 17B-16E (Q4_K_M) on apple-m4-ultra via mlx-lmbenchmarks/ss-m4ultra-192-llama4-scout-mlx-q4km-tg
verifiedlow · 0.55
ss-m4ultra-192-mistral-small4-119b-mlx-q4km decode
45.0 tok/s (avg)Mistral Small 4 119B (Q4_K_M) on apple-m4-ultra via mlx-lmbenchmarks/ss-m4ultra-192-mistral-small4-119b-mlx-q4km-tg
verifiedlow · 0.55
ss-m4ultra-192-qwen3-235b-mlx-q4km decode
22.0 tok/s (avg)Qwen 3 235B-A22B (Q4_K_M) on apple-m4-ultra via mlx-lmbenchmarks/ss-m4ultra-192-qwen3-235b-mlx-q4km-tg
verifiedlow · 0.55
ss-m4ultra-192-qwen3-32b-ollama-q4km decode
32.0 tok/s (avg)Qwen 3 32B (Q4_K_M) on apple-m4-ultra via ollamabenchmarks/ss-m4ultra-192-qwen3-32b-ollama-q4km-tg
verifiedlow · 0.55
ss-m5max-128-gemma4-e2b-mlx-q4km decode
158.0 tok/s (avg)Gemma 4 E2B (Q4_K_M) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-gemma4-e2b-mlx-q4km-tg
verifiedlow · 0.55
ss-m5max-128-gptoss120b-ollama-q4km decode
7.0 tok/s (avg)gpt-oss 120B (Q4_K_M) on local-apple-m5-max via ollamabenchmarks/ss-m5max-128-gptoss120b-ollama-q4km-tg
verifiedlow · 0.55
ss-m5max-128-llama4-scout-mlx-q4km decode
26.0 tok/s (avg)Llama 4 Scout 17B-16E (Q4_K_M) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-llama4-scout-mlx-q4km-tg
verifiedlow · 0.55
ss-m5max-128-mistral-small4-119b-ollama-q4km decode
38.0 tok/s (avg)Mistral Small 4 119B (Q4_K_M) on local-apple-m5-max via ollamabenchmarks/ss-m5max-128-mistral-small4-119b-ollama-q4km-tg
verifiedlow · 0.55
ss-m5max-128-qwen3-235b-mlx-q4km decode
18.0 tok/s (avg)Qwen 3 235B-A22B (Q4_K_M) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-qwen3-235b-mlx-q4km-tg
verifiedlow · 0.55
ss-m5max-128-qwen3-coder-next-mlx-8bit decode
79.296 tok/s (avg)Qwen3-Coder-Next (MLX 8-bit) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-qwen3-coder-next-mlx-8bit-tg
verifiedlow · 0.55
ss-m5max-128-qwen3-coder-next-mlx-8bit prefill
754.927 tok/s (prompt eval)Qwen3-Coder-Next (MLX 8-bit) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-qwen3-coder-next-mlx-8bit-pp
verifiedlow · 0.50
ss-m5max-128-qwen35-122b-mlx-4bit decode
65.853 tok/s (avg)Qwen3.5-122B-A10B (MLX 4-bit) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-qwen35-122b-mlx-4bit-tg
verifiedlow · 0.55
ss-m5max-128-qwen35-122b-mlx-4bit prefill
881.466 tok/s (prompt eval)Qwen3.5-122B-A10B (MLX 4-bit) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-128-qwen35-122b-mlx-4bit-pp
verifiedlow · 0.50
ss-m5max-48-qwen35-35b-llamacpp-q4km decode
89.4 tok/s (avg)Qwen3.5-35B-A3B (Q4_K_M) on local-apple-m5-max via llama-cppbenchmarks/ss-m5max-48-qwen35-35b-llamacpp-q4km-tg
verifiedlow · 0.55
ss-m5max-48-qwen35-35b-llamacpp-q4km prefill
783.0 tok/s (prompt eval)Qwen3.5-35B-A3B (Q4_K_M) on local-apple-m5-max via llama-cppbenchmarks/ss-m5max-48-qwen35-35b-llamacpp-q4km-pp
verifiedlow · 0.50
ss-m5max-48-qwen35-35b-mlx-4bit decode
128.0 tok/s (avg)Qwen3.5-35B-A3B (MLX 4-bit) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-48-qwen35-35b-mlx-4bit-tg
verifiedlow · 0.55
ss-m5max-48-qwen35-35b-mlx-4bit prefill
3235.0 tok/s (prompt eval)Qwen3.5-35B-A3B (MLX 4-bit) on local-apple-m5-max via mlx-lmbenchmarks/ss-m5max-48-qwen35-35b-mlx-4bit-pp
verifiedlow · 0.50
ss-m5pro-24-gemma4-26b-ollama-q4km decode
35.0 tok/s (avg)Gemma 4 26B-A4B (Q4_K_M) on local-apple-m5-pro via ollamabenchmarks/ss-m5pro-24-gemma4-26b-ollama-q4km-tg
verifiedlow · 0.55
ss-m5pro-24-gemma4-e4b-ollama-q4km decode
92.0 tok/s (avg)Gemma 4 E4B (Q4_K_M) on local-apple-m5-pro via ollamabenchmarks/ss-m5pro-24-gemma4-e4b-ollama-q4km-tg
verifiedlow · 0.55
ss-m5pro-64-qwen35-35b-llamacpp-q4kl decode
41.9 tok/s (avg)Qwen3.5-35B-A3B (Q4_K_L) on local-apple-m5-pro via llama-cppbenchmarks/ss-m5pro-64-qwen35-35b-llamacpp-q4kl-tg
verifiedlow · 0.55
ss-m5pro-64-qwen35-9b-llamacpp-q4km decode
25.0 tok/s (avg)Qwen3.5-9B (Q4_K_M) on local-apple-m5-pro via llama-cppbenchmarks/ss-m5pro-64-qwen35-9b-llamacpp-q4km-tg
verifiedlow · 0.55
Standard speculative sampling acceptance length and speedup on Vicuna-13B (temperature=0)
2.18 tokens_per_cycleVicuna-13B on ? via vllmbenchmarks/sd-sps-acceptance-vicuna13b-t0
verifiedhigh · 0.90
Strix Halo (gorgon-halo) DwarfStar ROCm 10.0 DeepSeek-V4.1-Flash Q2 64k-depth decode
14.3066 tok/s (per-request, concurrency 1)DeepSeek-V4.1-Flash Q2 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-dsv41-flash-q2-d64k-decode
verifiedlow · 0.60
Strix Halo (gorgon-halo) DwarfStar ROCm 10.0 DeepSeek-V4.1-Flash Q2 depth-0 decode
14.7622 tok/s (per-request, concurrency 1)DeepSeek-V4.1-Flash Q2 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-dsv41-flash-q2-d0-decode
verifiedlow · 0.60
Strix Halo (gorgon-halo) DwarfStar ROCm 10.0 DeepSeek-V4.1-Flash Q2 depth-0 prefill
240.8634 tok/s (per-request, concurrency 1)DeepSeek-V4.1-Flash Q2 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-dsv41-flash-q2-d0-prefill
verifiedlow · 0.60
Strix Halo (gorgon-halo) DwarfStar ROCm 10.0 GLM-5.3-Flash Q2 64k-depth decode
9.159 tok/s (per-request, concurrency 1)GLM-5.3-Flash Q2 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-glm53-flash-q2-d64k-decode
verifiedlow · 0.60
Strix Halo (gorgon-halo) DwarfStar ROCm 10.0 GLM-5.3-Flash Q2 depth-0 decode
11.9241 tok/s (per-request, concurrency 1)GLM-5.3-Flash Q2 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-glm53-flash-q2-d0-decode
verifiedlow · 0.60
Strix Halo (gorgon-halo) DwarfStar ROCm 10.0 GLM-5.3-Flash Q2 depth-0 prefill
227.1627 tok/s (per-request, concurrency 1)GLM-5.3-Flash Q2 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-glm53-flash-q2-d0-prefill
verifiedlow · 0.60
Strix Halo 8060S (HIP (ROCm)) Gemma 3 27B UD-Q4_K_XL: pp512 prefill tok/s
302.2 tok/s (pp512, per-request batch-1)Gemma 3 27B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-gemma3-27b-udq4-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Gemma 3 27B UD-Q4_K_XL: tg128 decode tok/s
10.7 tok/s (tg128, per-request batch-1)Gemma 3 27B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-gemma3-27b-udq4-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) gpt-oss 20B MXFP4: pp512 prefill tok/s
1196.2 tok/s (pp512, per-request batch-1)gpt-oss 20B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-gpt-oss-20b-mxfp4-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) gpt-oss 20B MXFP4: tg128 decode tok/s
46.9 tok/s (tg128, per-request batch-1)gpt-oss 20B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-gpt-oss-20b-mxfp4-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Llama 4 Scout 17B-16E UD-Q4_K_XL: pp512 prefill tok/s
277.4 tok/s (pp512, per-request batch-1)Llama 4 Scout 17B-16E on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-llama4-scout-udq4-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Llama 4 Scout 17B-16E UD-Q4_K_XL: tg128 decode tok/s
17.6 tok/s (tg128, per-request batch-1)Llama 4 Scout 17B-16E on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-llama4-scout-udq4-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Qwen 3 235B-A22B UD-Q3_K_XL: pp512 prefill tok/s
117.1 tok/s (pp512, per-request batch-1)Qwen 3 235B-A22B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-qwen3-235b-udq3-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Qwen 3 235B-A22B UD-Q3_K_XL: tg128 decode tok/s
12.9 tok/s (tg128, per-request batch-1)Qwen 3 235B-A22B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-qwen3-235b-udq3-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Qwen 3 32B Q8_0: pp512 prefill tok/s
226.1 tok/s (pp512, per-request batch-1)Qwen 3 32B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-qwen3-32b-q80-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Qwen 3 32B Q8_0: tg128 decode tok/s
6.4 tok/s (tg128, per-request batch-1)Qwen 3 32B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-qwen3-32b-q80-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Shisa V2 8B i1-Q4_K_M: pp512 prefill tok/s
878.2 tok/s (pp512, per-request batch-1)Shisa V2 8B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-shisa-v2-8b-q4km-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP (ROCm)) Shisa V2 8B i1-Q4_K_M: tg128 decode tok/s
37.2 tok/s (tg128, per-request batch-1)Shisa V2 8B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-shisa-v2-8b-q4km-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 0
2311.45 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d0
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 131072
221.25 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d128k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 16384
1002.74 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d16k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 32768
646.32 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d32k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 4096
1831.12 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d4k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 65536
384.67 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d64k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 8192
1394.79 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-pp512-d8k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 0
130.33 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d0
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 131072
15.99 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d128k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 16384
69.67 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d16k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 32768
45.31 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d32k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 4096
113.2 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d4k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 65536
27.42 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d64k
verifiedmedium · 0.72
Strix Halo 8060S (HIP (TheRock ROCm 7.0.0-0, hipBLASLt)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 8192
95.57 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-hip-tg128-d8k
verifiedmedium · 0.72
Strix Halo 8060S (HIP rocWMMA (ROCm)) Llama 2 7B Q4_K_M: pp512 prefill tok/s
1083.0 tok/s (pp512, per-request batch-1)Llama 2 7B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-llama2-7b-q4km-hip-rocwmma-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Llama 2 7B Q4_K_M: tg128 decode tok/s
41.3 tok/s (tg128, per-request batch-1)Llama 2 7B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-llama2-7b-q4km-hip-rocwmma-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Mistral Small 3.1 24B UD-Q4_K_XL: pp512 prefill tok/s
368.5 tok/s (pp512, per-request batch-1)Mistral Small 3.1 24B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-mistral-small-31-udq4-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Mistral Small 3.1 24B UD-Q4_K_XL: tg128 decode tok/s
13.8 tok/s (tg128, per-request batch-1)Mistral Small 3.1 24B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-mistral-small-31-udq4-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Qwen 3 30B-A3B UD-Q4_K_XL: pp512 prefill tok/s
669.4 tok/s (pp512, per-request batch-1)Qwen 3 30B-A3B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-qwen3-30b-a3b-udq4-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Qwen 3 30B-A3B UD-Q4_K_XL: tg128 decode tok/s
58.5 tok/s (tg128, per-request batch-1)Qwen 3 30B-A3B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-qwen3-30b-a3b-udq4-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Shisa V2 70B i1-Q4_K_M: pp512 prefill tok/s
94.7 tok/s (pp512, per-request batch-1)Shisa V2 70B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-shisa-v2-70b-q4km-hip-pp512
verifiedmedium · 0.75
Strix Halo 8060S (HIP rocWMMA (ROCm)) Shisa V2 70B i1-Q4_K_M: tg128 decode tok/s
4.5 tok/s (tg128, per-request batch-1)Shisa V2 70B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-shisa-v2-70b-q4km-hip-tg128
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 0
1685.38 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d0
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 131072
160.67 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d128k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 16384
722.15 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d16k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 32768
469.91 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d32k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 4096
1314.26 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d4k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 65536
280.57 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d64k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 8192
1019.54 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d8k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 0
130.56 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d0
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 131072
27.32 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d128k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 16384
90.13 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d16k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 32768
68.54 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d32k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 4096
117.55 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d4k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 65536
45.54 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d64k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK (2025.Q2.1)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 8192
106.68 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d8k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan AMDVLK) dots1 UD-Q4_K_XL: pp512 prefill tok/s
182.0 tok/s (pp512, per-request batch-1)dots1 on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-dots1-udq4-amdvlk-pp512
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) dots1 UD-Q4_K_XL: tg128 decode tok/s
22.1 tok/s (tg128, per-request batch-1)dots1 on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-dots1-udq4-amdvlk-tg128
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) GLM 4.5 Air UD-Q4_K_XL: pp512 prefill tok/s
178.5 tok/s (pp512, per-request batch-1)GLM 4.5 Air on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-glm45-air-udq4-amdvlk-pp512
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) GLM 4.5 Air UD-Q4_K_XL: tg128 decode tok/s
22.6 tok/s (tg128, per-request batch-1)GLM 4.5 Air on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-glm45-air-udq4-amdvlk-tg128
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) gpt-oss 120B MXFP4: pp512 prefill tok/s
430.6 tok/s (pp512, per-request batch-1)gpt-oss 120B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-gpt-oss-120b-mxfp4-amdvlk-pp512
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) gpt-oss 120B MXFP4: tg128 decode tok/s
33.7 tok/s (tg128, per-request batch-1)gpt-oss 120B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-gpt-oss-120b-mxfp4-amdvlk-tg128
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) Hunyuan-A13B UD-Q6_K_XL: pp512 prefill tok/s
296.6 tok/s (pp512, per-request batch-1)Hunyuan-A13B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-hunyuan-a13b-udq6-amdvlk-pp512
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) Hunyuan-A13B UD-Q6_K_XL: tg128 decode tok/s
18.1 tok/s (tg128, per-request batch-1)Hunyuan-A13B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-hunyuan-a13b-udq6-amdvlk-tg128
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) Llama 2 7B Q4_0: pp512 prefill tok/s
1294.1 tok/s (pp512, per-request batch-1)Llama 2 7B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-llama2-7b-q4-0-amdvlk-pp512
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan AMDVLK) Llama 2 7B Q4_0: tg128 decode tok/s
47.7 tok/s (tg128, per-request batch-1)Llama 2 7B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strix-halo-8060s-llama2-7b-q4-0-amdvlk-tg128
verifiedmedium · 0.75
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 0
1539.61 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d0
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 131072
172.6 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d128k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 16384
710.06 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d16k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 32768
482.27 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d32k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 4096
1215.54 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d4k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 65536
299.29 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d64k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: pp512 at depth 8192
961.87 tok/s (pp512 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-pp512-d8k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 0
129.99 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d0
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 131072
39.83 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d128k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 16384
99.36 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d16k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 32768
82.79 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d32k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 4096
118.23 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d4k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 65536
60.92 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d64k
verifiedmedium · 0.72
Strix Halo 8060S (Vulkan RADV (Mesa 25.1.6)) Qwen3-30B-A3B UD-Q4_K_XL: tg128 at depth 8192
110.64 tok/s (tg128 at primed depth, per-request batch-1, 5-run average)Qwen3-30B-A3B (MoE, ~3B active) on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/wiki-strix-8060s-qwen3-30b-a3b-udq4-radv-tg128-d8k
verifiedmedium · 0.72
Strix Halo 8060S DwarfStar ROCm 10.0 Qwen3.8-Flash-Next Q4 64k-depth decode
19.3232 tok/s (per-request, concurrency 1)Qwen3.8-Flash-Next Q4 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-qwen38-flash-next-q4-d64k-decode
verifiedlow · 0.60
Strix Halo 8060S DwarfStar ROCm 10.0 Qwen3.8-Flash-Next Q4 depth-0 decode
19.5233 tok/s (per-request, concurrency 1)Qwen3.8-Flash-Next Q4 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-qwen38-flash-next-q4-d0-decode
verifiedlow · 0.60
Strix Halo 8060S DwarfStar ROCm 10.0 Qwen3.8-Flash-Next Q4 depth-0 prefill
504.5011 tok/s (per-request, concurrency 1)Qwen3.8-Flash-Next Q4 on amd-radeon-8060s-ryzen-ai-max via dwarfstarbenchmarks/llmb-8060s-dwarfstar-rocm10-qwen38-flash-next-q4-d0-prefill
verifiedlow · 0.60
Strix Halo 8060S llama.cpp b6702 Mesa: Qwen3-30B-A3B UD-Q4_K_XL pp512 50.11 tok/s (depth 130560)
50.11 tok/s per-request prefill (llama-bench pp512, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-radv-pp512-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 Mesa: Qwen3-30B-A3B UD-Q4_K_XL pp512 632.46 tok/s (depth 0)
632.46 tok/s per-request prefill (llama-bench pp512, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-radv-pp512
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 Mesa: Qwen3-30B-A3B UD-Q4_K_XL tg128 12.36 tok/s (depth 130560)
12.36 tok/s per-request decode (llama-bench tg128, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-radv-tg128-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 Mesa: Qwen3-30B-A3B UD-Q4_K_XL tg128 61.13 tok/s (depth 0)
61.13 tok/s per-request decode (llama-bench tg128, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-radv-tg128
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL pp512 40.35 tok/s (depth 130560)
40.35 tok/s per-request prefill (llama-bench pp512, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-hipblaslt-pp512-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL pp512 40.58 tok/s (depth 130560)
40.58 tok/s per-request prefill (llama-bench pp512, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-pp512-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL pp512 650.59 tok/s (depth 0)
650.59 tok/s per-request prefill (llama-bench pp512, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-pp512
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL pp512 651.93 tok/s (depth 0)
651.93 tok/s per-request prefill (llama-bench pp512, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-hipblaslt-pp512
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL tg128 4.97 tok/s (depth 130560)
4.97 tok/s per-request decode (llama-bench tg128, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-hipblaslt-tg128-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL tg128 4.98 tok/s (depth 130560)
4.98 tok/s per-request decode (llama-bench tg128, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-tg128-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL tg128 63.95 tok/s (depth 0)
63.95 tok/s per-request decode (llama-bench tg128, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-hipblaslt-tg128
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp b6702 ROCm: Qwen3-30B-A3B UD-Q4_K_XL tg128 64.17 tok/s (depth 0)
64.17 tok/s per-request decode (llama-bench tg128, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-tg128
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 prompt processing at depth 131072
186.14 tok/s (pp depth 131072, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d131072-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 prompt processing at depth 32768
618.3 tok/s (pp depth 32768, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d32768-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 prompt processing at depth 4096
929.11 tok/s (pp depth 4096, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d4096-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 prompt processing at depth 512
1014.35 tok/s (pp depth 512, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d512-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 token generation at depth 131072
25.26 tok/s (tg depth 131072, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d131072-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 token generation at depth 32768
44.22 tok/s (tg depth 32768, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d32768-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 token generation at depth 4096
50.33 tok/s (tg depth 4096, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d4096-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: gpt-oss-120b MXFP4 token generation at depth 512
51.18 tok/s (tg depth 512, per-request batch-1)gpt-oss-120b MXFP4 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-gptoss-120b-mxfp4-hip-d512-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL prompt processing at depth 131072
153.41 tok/s (pp depth 131072, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d131072-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL prompt processing at depth 32768
477.39 tok/s (pp depth 32768, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d32768-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL prompt processing at depth 4096
692.88 tok/s (pp depth 4096, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d4096-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL prompt processing at depth 512
807.78 tok/s (pp depth 512, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d512-pp
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL token generation at depth 131072
16.46 tok/s (tg depth 131072, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d131072-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL token generation at depth 32768
49.55 tok/s (tg depth 32768, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d32768-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL token generation at depth 4096
74.91 tok/s (tg depth 4096, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d4096-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp HIP b8156: Qwen3-30B-A3B UD-Q4_K_XL token generation at depth 512
88.5 tok/s (tg depth 512, per-request batch-1)Qwen3-30B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/strixwiki-8060s-qwen3-30b-a3b-udq4kxl-hip-d512-tg
verifiedlow · 0.55
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL pp512 659.07 tok/s (rocm-tuned, depth 130560)
659.07 tok/s per-request prefill (llama-bench pp512, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-tuned-pp512-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL pp512 741.60 tok/s (amdvlk, depth 0)
741.6 tok/s per-request prefill (llama-bench pp512, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL pp512 755.14 tok/s (amdvlk, depth 130560)
755.14 tok/s per-request prefill (llama-bench pp512, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-amdvlk-pp512-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL tg128 64.17 tok/s (rocm-tuned, depth 0)
64.17 tok/s per-request decode (llama-bench tg128, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-tuned-tg128
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL tg128 67.66 tok/s (rocm-tuned, depth 130560)
67.66 tok/s per-request decode (llama-bench tg128, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-rocm-tuned-tg128-d130560
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL tg128 81.79 tok/s (amdvlk, depth 0)
81.79 tok/s per-request decode (llama-bench tg128, batch 1, depth 0)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128
verifiedmedium · 0.72
Strix Halo 8060S llama.cpp: Qwen3-30B-A3B UD-Q4_K_XL tg128 85.11 tok/s (amdvlk, depth 130560)
85.11 tok/s per-request decode (llama-bench tg128, batch 1, depth 130560)Qwen3-30B-A3B UD-Q4_K_XL GGUF on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/shhl-8060s-qwen3-30b-a3b-udq4-amdvlk-tg128-d130560
verifiedmedium · 0.72
Strix Halo 8060S ROCm 10.0 DeepSeek-V4-Flash-0731 UD-IQ2_XXS 64k-depth decode
10.4007 tok/s (per-request, concurrency 1)DeepSeek-V4-Flash-0731 UD-IQ2_XXS on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-dsv4-flash-iq2xxs-d64k-decode
verifiedlow · 0.60
Strix Halo 8060S ROCm 10.0 DeepSeek-V4-Flash-0731 UD-IQ2_XXS depth-0 decode
13.3312 tok/s (per-request, concurrency 1)DeepSeek-V4-Flash-0731 UD-IQ2_XXS on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-dsv4-flash-iq2xxs-d0-decode
verifiedlow · 0.60
Strix Halo 8060S ROCm 10.0 DeepSeek-V4-Flash-0731 UD-IQ2_XXS depth-0 prefill
216.9735 tok/s (per-request, concurrency 1)DeepSeek-V4-Flash-0731 UD-IQ2_XXS on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-dsv4-flash-iq2xxs-d0-prefill
verifiedlow · 0.60
Strix Halo 8060S ROCm 10.0 Qwen3.6-35B-A3B UD-Q4_K_XL depth-0 decode
51.2575 tok/sQwen3.6-35B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-qwen36-35b-a3b-q4kxl-d0-decode
verifiedlow · 0.60
Strix Halo 8060S ROCm 10.0 Qwen3.8-Flash-Next UD-IQ4_XS 64k-depth decode
12.2638 tok/s (per-request, concurrency 1)Qwen3.8-Flash-Next UD-IQ4_XS on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-qwen38-flash-next-iq4xs-d64k-decode
verifiedlow · 0.60
Strix Halo 8060S ROCm 10.0 Qwen3.8-Flash-Next UD-IQ4_XS depth-0 decode
20.5757 tok/s (per-request, concurrency 1)Qwen3.8-Flash-Next UD-IQ4_XS on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-qwen38-flash-next-iq4xs-d0-decode
verifiedlow · 0.60
Strix Halo 8060S ROCm 10.0 Qwen3.8-Flash-Next UD-IQ4_XS depth-0 prefill
470.2468 tok/s (per-request, concurrency 1)Qwen3.8-Flash-Next UD-IQ4_XS on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-rocm10-qwen38-flash-next-iq4xs-d0-prefill
verifiedlow · 0.60
Strix Halo 8060S ROCm 7.2.3 Qwen3.6-35B-A3B UD-Q4_K_XL: pp512 tok/s
1120.0 tok/s (pp512, per-request batch-1)Qwen3.6-35B-A3B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/ky0-qwen36-35b-a3b-udq4-rocm-7-2-3-pp512
verifiedmedium · 0.70
Strix Halo 8060S ROCm 7.2.3 Qwen3.6-35B-A3B UD-Q4_K_XL: tg128 tok/s
51.53 tok/s (tg128, per-request batch-1)Qwen3.6-35B-A3B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/ky0-qwen36-35b-a3b-udq4-rocm-7-2-3-tg128
verifiedmedium · 0.70
Strix Halo 8060S Vulkan RADV gpt-oss-120b MXFP4: pp512 tok/s
719.91 tok/s (pp512, per-request batch-1)gpt-oss-120b on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/ky0-gpt-oss-120b-mxfp4-vulkan-radv-pp512
verifiedmedium · 0.70
Strix Halo 8060S Vulkan RADV gpt-oss-120b MXFP4: tg128 tok/s
56.61 tok/s (tg128, per-request batch-1)gpt-oss-120b on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/ky0-gpt-oss-120b-mxfp4-vulkan-radv-tg128
verifiedmedium · 0.70
Strix Halo 8060S Vulkan RADV LFM2.5-1.2B-Instruct BF16 depth-0 decode
74.4867 tok/sLFM2.5-1.2B-Instruct BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-vulkan-lfm25-1-2b-bf16-d0-decode
verifiedlow · 0.60
Strix Halo 8060S Vulkan RADV LFM2.5-1.2B-Instruct BF16 depth-0 prefill
2843.0933 tok/sLFM2.5-1.2B-Instruct BF16 on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-vulkan-lfm25-1-2b-bf16-d0-prefill
verifiedlow · 0.60
Strix Halo 8060S Vulkan RADV Qwen3.6-35B-A3B UD-Q4_K_XL depth-0 decode
60.6843 tok/sQwen3.6-35B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-vulkan-qwen36-35b-a3b-q4kxl-d0-decode
verifiedlow · 0.60
Strix Halo 8060S Vulkan RADV Qwen3.6-35B-A3B UD-Q4_K_XL depth-0 prefill
1218.7104 tok/sQwen3.6-35B-A3B UD-Q4_K_XL on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/llmb-8060s-vulkan-qwen36-35b-a3b-q4kxl-d0-prefill
verifiedlow · 0.60
Strix Halo 8060S Vulkan RADV Qwen3.6-35B-A3B UD-Q4_K_XL: pp512 tok/s
1113.64 tok/s (pp512, per-request batch-1)Qwen3.6-35B-A3B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/ky0-qwen36-35b-a3b-udq4-vulkan-radv-pp512
verifiedmedium · 0.70
Strix Halo 8060S Vulkan RADV Qwen3.6-35B-A3B UD-Q4_K_XL: tg128 tok/s
60.43 tok/s (tg128, per-request batch-1)Qwen3.6-35B-A3B on amd-ryzen-ai-max-395-framework-desktop, amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/ky0-qwen36-35b-a3b-udq4-vulkan-radv-tg128
verifiedmedium · 0.70
Strix Halo 8060S Vulkan/AMDVLK: gpt-oss-120b decode_tok_s (prompt 2048, gen 32, depth 0)
50.05 tok/s tg32 at context depth 0, batch 1, per-requestgpt-oss-120b on amd-radeon-8060s-ryzen-ai-max, amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/lhl-strix-8060s-gptoss120b-amdvlk-tg32-d0
verifiedmedium · 0.75
Strix Halo 8060S Vulkan/AMDVLK: gpt-oss-120b decode_tok_s (prompt 2048, gen 32, depth 32768)
31.54 tok/s tg32 at context depth 32768, batch 1, per-requestgpt-oss-120b on amd-radeon-8060s-ryzen-ai-max, amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/lhl-strix-8060s-gptoss120b-amdvlk-tg32-d32k
verifiedmedium · 0.75
Strix Halo 8060S Vulkan/AMDVLK: gpt-oss-120b prefill_tok_s (prompt 2048, gen 32, depth 0 (empty KV))
729.1 tok/s pp2048 at context depth 0, batch 1, per-requestgpt-oss-120b on amd-radeon-8060s-ryzen-ai-max, amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/lhl-strix-8060s-gptoss120b-amdvlk-pp2048-d0
verifiedmedium · 0.75
Strix Halo 8060S Vulkan/AMDVLK: gpt-oss-120b prefill_tok_s (prompt 2048, gen 32, depth 32768)
137.08 tok/s pp2048 at context depth 32768, batch 1, per-requestgpt-oss-120b on amd-radeon-8060s-ryzen-ai-max, amd-ryzen-ai-max-395-framework-desktop via llama-cppbenchmarks/lhl-strix-8060s-gptoss120b-amdvlk-pp2048-d32k
verifiedmedium · 0.75
Strix Halo retained-PM4 llama-server Qwen3.8-Flash-Next MTP decode
43.9 tok/s (per-request, parallel 1)Qwen3.8-Flash-Next (shared MTP head) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-strixllama-qwen38-flash-next-mtp-decode
verifiedmedium · 0.65
Strix Halo retained-PM4 llama-server Qwen3.8-Flash-Next MTP pp2048
1207.0 tok/s (per-request, parallel 1)Qwen3.8-Flash-Next (shared MTP head) on amd-radeon-8060s-ryzen-ai-max via llama-cppbenchmarks/kyuz0-8060s-strixllama-qwen38-flash-next-mtp-pp2048
verifiedmedium · 0.65
SyFI/VibeSys: 4x MI300A, bespoke agent-built engine, Qwen3.5-397B-A17B MXFP4, 96 sessions, goodput
2242.0 tok/s aggregate goodput (96 concurrent sessions, SLO-filtered)amd/Qwen3.5-397B-A17B-MXFP4 on amd-instinct-mi300a via vibesys-bespoke-enginebenchmarks/syfi-mi300a-vibesys-bespoke-qwen35-397b-mxfp4-96sess-goodput
verifiedmedium · 0.80
SyFI/VibeSys: 4x MI300A, tuned SGLang 0.5.18, Qwen3.5-397B-A17B MXFP4, 48 sessions, goodput
961.0 tok/s aggregate goodput (48 concurrent sessions, SLO-filtered)amd/Qwen3.5-397B-A17B-MXFP4 on amd-instinct-mi300a via sglangbenchmarks/syfi-mi300a-sglang-tuned-qwen35-397b-mxfp4-48sess-goodput
verifiedmedium · 0.80
TensorRT-LLM: Llama 2 70B FP8 on 8x H100 80GB, single-request latency at batch size 1
1700 ms per full request (batch 1; source states 'a single inference in 1.7 seconds')Llama-2-70b (TensorRT-LLM FP8 build) on nvidia-h100-sxm via tensorrt-llmbenchmarks/trtllm-llama2-70b-h100-fp8-batch1-request-latency
verifiedmedium · 0.80
TensorRT-LLM: Llama 3.1 405B FP8 on 8x H100 SXM 80GB (TP=8), ISL/OSL 1000/2000
2884.69 output tokens/s (aggregate, 8 GPUs)nvidia/Llama-3.1-405B-Instruct-FP8 on nvidia-h100-sxm via tensorrt-llmbenchmarks/trtllm-llama31-405b-fp8-h100-tp8-tps
verifiedhigh · 0.85
TensorRT-LLM: Llama 3.1 8B FP8 on 1x H100 SXM 80GB, ISL/OSL 1000/1000
14991.62 output tokens/s (aggregate, 1 GPU)nvidia/Llama-3.1-8B-Instruct-FP8 on nvidia-h100-sxm via tensorrt-llmbenchmarks/trtllm-llama31-8b-fp8-h100-1k1k-tps
verifiedhigh · 0.85
TensorRT-LLM: Llama 3.1 8B FP8 on 1x H200 SXM 141GB, ISL/OSL 1000/1000
17162.49 output tokens/s (aggregate, 1 GPU)nvidia/Llama-3.1-8B-Instruct-FP8 on nvidia-h200-sxm via tensorrt-llmbenchmarks/trtllm-llama31-8b-fp8-h200-1k1k-tps
verifiedhigh · 0.85
TensorRT-LLM: Llama 3.3 70B FP4 on 1x B200 180GB, ISL/OSL 1000/2000
6725.03 output tokens/s (aggregate, 1 GPU)nvidia/Llama-3.3-70B-Instruct-FP4 on nvidia-b200 via tensorrt-llmbenchmarks/trtllm-llama33-70b-fp4-b200-tp1-tps
verifiedmedium · 0.80
TensorRT-LLM: Llama 3.3 70B FP8 on 2x H100 SXM 80GB (TP=2), ISL/OSL 1000/2000
3708.93 output tokens/s (aggregate, 2 GPUs)nvidia/Llama-3.3-70B-Instruct-FP8 on nvidia-h100-sxm via tensorrt-llmbenchmarks/trtllm-llama33-70b-fp8-h100-tp2-1k2k-tps
verifiedhigh · 0.85
TensorRT-LLM: Llama 3.3 70B FP8 on 2x H200 SXM 141GB (TP=2), ISL/OSL 1000/2000
5790.36 output tokens/s (aggregate, 2 GPUs)nvidia/Llama-3.3-70B-Instruct-FP8 on nvidia-h200-sxm via tensorrt-llmbenchmarks/trtllm-llama33-70b-fp8-h200-tp2-1k2k-tps
verifiedhigh · 0.85
Tenstorrent Blackhole p150 2-card data-parallel Llama-3.1-8B aggregate (vendor b32)
1190.0 tok/s aggregate, 2x p150 data-parallel, batch-32Llama-3.1-8B on tenstorrent-blackhole-p150 via niche-tt-metalbenchmarks/comm-tt-blackhole-p150-2x-llama31-8b-agg
verifiedmedium · 0.75
Tenstorrent Blackhole p150 Llama-3.1-8B token-gen (vendor b32)
33.1 tok/s per user at batch-32; 1059 t/s aggLlama-3.1-8B on tenstorrent-blackhole-p150 via niche-tt-metalbenchmarks/comm-tt-blackhole-p150-llama31-8b-tg
verifiedmedium · 0.65
Tenstorrent Blackhole p150, tt-metal: Llama-3.1-8B decode tok/s per-user + aggregate (vendor-measured b32)
33.1 tok/s per user at batch-32; 1059 tok/s aggregateLlama-3.1-8B on tenstorrent-blackhole-p150 via niche-tt-metalbenchmarks/comm-tt-blackhole-p150-llama31-8b-decode
verifiedmedium · 0.78
Tenstorrent Wormhole n150 Falcon-7B token-gen (vendor b32)
18.5 tok/s per user at batch-32Falcon-7B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-falcon-7b-tg
verifiedmedium · 0.75
Tenstorrent Wormhole n150 Llama-3.1-8B token-gen (vendor b32)
26.0 tok/s per user at batch-32; 832 t/s aggLlama-3.1-8B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-llama31-8b-tg
verifiedmedium · 0.70
Tenstorrent Wormhole n150 Llama-3.2-1B token-gen (vendor b32)
80.5 tok/s per user at batch-32Llama-3.2-1B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-llama32-1b-tg
verifiedmedium · 0.75
Tenstorrent Wormhole n150 Llama-3.2-3B token-gen (vendor b32)
46.6 tok/s per user at batch-32Llama-3.2-3B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-llama32-3b-tg
verifiedmedium · 0.75
Tenstorrent Wormhole n150 Mistral-7B token-gen (vendor b32)
28.7 tok/s per user at batch-32; 918 t/s aggregateMistral-7B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-mistral-7b-tg
verifiedmedium · 0.75
Tenstorrent Wormhole n150, tt-metal: Falcon-7B decode tok/s per-user (vendor-measured b32)
18.5 tok/s per user at batch-32Falcon-7B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-falcon7b-decode
verifiedmedium · 0.80
Tenstorrent Wormhole n150, tt-metal: Llama-3.1-8B decode tok/s per-user + aggregate (vendor-measured b32)
26.0 tok/s per user at batch-32; 832 tok/s aggregateLlama-3.1-8B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-llama31-8b-decode
verifiedmedium · 0.80
Tenstorrent Wormhole n150, tt-metal: Mistral-7B decode tok/s per-user + aggregate (vendor-measured b32)
28.7 tok/s per user at batch-32; 918 tok/s aggregateMistral-7B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n150-mistral7b-decode
verifiedmedium · 0.80
Tenstorrent Wormhole n300 Qwen2.5-7B decode (vendor-measured, b32 TP2)
22.1 tok/s per user at batch-32; 707 tok/s aggregateQwen2.5-7B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/comm-tt-wormhole-n300-qwen25-7b-decode
verifiedmedium · 0.75
Tenstorrent Wormhole n300, tt-metal TP=2: Qwen2.5-7B decode tok/s per-user + aggregate (vendor-measured b32)
22.1 tok/s per user at batch-32; 707 tok/s aggregateQwen2.5-7B on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/comm-tt-wormhole-n300-qwen25-7b-tp2-decode
verifiedmedium · 0.80
Tenstorrent Wormhole N300, vLLM+TT-Metal: Llama-3.1-8B single-stream decode tok/s
30.0 tok/s (single-stream decode, batch 1, stabilized)Llama-3.1-8B on tenstorrent-wormhole-n300 via vllmbenchmarks/comm-tenstorrent-n300-llama31-8b-decode
verifiedlow · 0.55
Tesla V100 32GB Llama-3.1-8B Q4_K_M decode (homelab field guide claim)
60.0 tok/s CLAIMED ('60+' on Pop!_OS); vramguide models 106 - treat 60-100 as the envelopeLlama-3.1-8B Q4_K_M on nvidia-v100-sxm2-32gb, nvidia-v100-pcie-32gb via llama-cppbenchmarks/comm-v100-32gb-llama3-8b-q4-decode-field-guide
verifiedmedium · 0.75
tt-metal Llama-3.3-70B on Galaxy (32x Wormhole, TP=32): 72.5 tok/s/user, 2268.8 tok/s @b32
72.5 tok/s per user (batch-32, aggregate 2268.8 tok/s across 32-chip Galaxy; ISL=128)Llama-3.3-70B on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-wormhole-galaxy-llama33-70b-tp32-b32-tpsu
verifiedmedium · 0.78
tt-metal Llama-3.3-70B on Galaxy (TP=32): TTFT 53 ms (ISL=128, b32)
53.0 ms (time-to-first-token, ISL=128, batch-32)Llama-3.3-70B on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-wormhole-galaxy-llama33-70b-tp32-ttft
verifiedmedium · 0.78
tt-metal Mixtral-8x7B on QuietBox (8x Wormhole, TP=8): 24.9 tok/s/user, 796.8 tok/s @b32
24.9 tok/s per user (batch-32, aggregate 796.8 tok/s; ISL=128)Mixtral-8x7B on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-wormhole-n300-mixtral8x7b-tp8-b32-tpsu
verifiedmedium · 0.78
tt-metal Qwen2.5-72B on QuietBox (8x Wormhole, TP=8): 15.4 tok/s/user, 492.8 tok/s @b32
15.4 tok/s per user (batch-32, aggregate 492.8 tok/s; ISL=128)Qwen2.5-72B on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-wormhole-n300-qwen25-72b-tp8-b32-tpsu
verifiedmedium · 0.78
tt-metal Qwen2.5-72B on QuietBox (TP=8): TTFT 223 ms (ISL=128, b32)
223.0 ms (time-to-first-token, ISL=128, batch-32)Qwen2.5-72B on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-wormhole-n300-qwen25-72b-tp8-ttft
verifiedmedium · 0.78
TTFT, MPT-7B, 512-token prompt, batch size 1, 1x A100 40GB
46.0 milliseconds to first token (512-token prompt, batch size 1)MPT-7B (FP16) on ? via fastertransformerbenchmarks/mpt7b-a100-bs1-ttft-ms
verifiedhigh · 0.85
US average PUE for facilities serving AI equipment: 1.145 in 2024, modelled down to 1.136 in 2030
1.145 dimensionless PUE ratio (total facility energy / IT equipment energy), 2024. Not an accuracy score and not a speedup: 1.145 means the facility draws 14.5% more than the IT equipment it houses. PUE is a ratio by DEFINITION and carries no faster/slower reading, which is why it belongs to dimensionless_ratio rather than speedup_ratio.? on ? via ollamabenchmarks/pue-us-facilities-serving-ai-equipment-2024-1-145
verifiedhigh · 0.88
US national average PUE across all data center equipment: 1.45 in 2024, down from 1.55 in 2018; 1.36 projected for 2030 in the Reference Case
1.45 dimensionless PUE ratio (total facility energy / IT equipment energy). Not an accuracy score and not a speedup: 1.0 means the facility overhead is nil.? on ? via ollamabenchmarks/pue-national-average-us-all-equipment-2024-1-45
verifiedhigh · 0.90
Used RTX 3090 24GB, llama.cpp CUDA: Qwen3-8B Q4_K decode tok/s @4K ctx (Hardware Corner measured)
115.3 decode tok/s (TG, single-stream, batch-1, 4K context)Qwen3-8B Q4_K_M (GGUF) on nvidia-rtx-3090 via llama-cppbenchmarks/comm-rtx3090-qwen3-8b-q4-decode
verifiedhigh · 0.85
VibeSys bespoke agent-built engine on 4x MI300A: Qwen3.5-397B-A17B MXFP4 goodput 2242 tok/s @96sess
2242.0 tok/s goodput (aggregate over 96 concurrent sessions, SLO-bound)Qwen3.5-397B-A17B on amd-instinct-mi300a via vllmbenchmarks/syfi-mi300a-vibesys-bespoke-qwen35-397b-mxfp4-goodput
verifiedmedium · 0.75
vLLM 0.1.4 (EmbeddedLLM ROCm port) on amd-instinct-mi210: LLaMA2-13B 2x MI210 TP=2 offline throughput tok/s
595.11 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on amd-instinct-mi210 via vllmbenchmarks/embeddedllm-vllm-mi210-llama2-13b-2g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (EmbeddedLLM ROCm port) on amd-instinct-mi210: LLaMA2-70B 4x MI210 TP=4 offline throughput tok/s
436.4 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on amd-instinct-mi210 via vllmbenchmarks/embeddedllm-vllm-mi210-llama2-70b-4g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (EmbeddedLLM ROCm port) on amd-instinct-mi210: LLaMA2-7B single GPU TP1 offline throughput tok/s
718.06 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on amd-instinct-mi210 via vllmbenchmarks/embeddedllm-vllm-mi210-llama2-7b-1g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (upstream CUDA) on nvidia-a100-80gb-sxm4: LLaMA2-13B 2x A100-SXM-80G TP=2 offline throughput tok/s
730.01 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on nvidia-a100-80gb-sxm4 via vllmbenchmarks/embeddedllm-vllm-a100sxm80-llama2-13b-2g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (upstream CUDA) on nvidia-a100-80gb-sxm4: LLaMA2-70B 4x A100-SXM-80G TP=4 offline throughput tok/s
629.33 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on nvidia-a100-80gb-sxm4 via vllmbenchmarks/embeddedllm-vllm-a100sxm80-llama2-70b-4g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (upstream CUDA) on nvidia-l40s: LLaMA2-13B 2x L40 TP=2 offline throughput tok/s
491.16 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on nvidia-l40s via vllmbenchmarks/embeddedllm-vllm-l40-llama2-13b-2g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (upstream CUDA) on nvidia-l40s: LLaMA2-70B 4x L40 TP=4 offline throughput tok/s
399.01 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on nvidia-l40s via vllmbenchmarks/embeddedllm-vllm-l40-llama2-70b-4g-tps
verifiedmedium · 0.70
vLLM 0.1.4 (upstream CUDA) on nvidia-l40s: LLaMA2-7B single L40 offline throughput tok/s
532.26 tok/s aggregate (offline inference, 1000 questions; NOT per-request)LLaMA2 (HF bf16, unquantized) on nvidia-l40s via vllmbenchmarks/embeddedllm-vllm-l40-llama2-7b-1g-tps
verifiedmedium · 0.70
vLLM 0.12 on 4x Arc Pro B60 (TP=4): Qwen3-VL-30B chat 1K->1K TPOT (concurrency 16)
61 ms/tokenQwen/Qwen3-VL-30B-A3B-Instruct on intel-arc-pro-b60 via vllmbenchmarks/embeddedllm-b60x4-vllm-qwen3vl30b-chat-tpot
verifiedmedium · 0.75
vLLM 0.12 on 4x Arc Pro B60 (TP=4): Qwen3-VL-30B chat 1K->1K TTFT (concurrency 16)
2100 msQwen/Qwen3-VL-30B-A3B-Instruct on intel-arc-pro-b60 via vllmbenchmarks/embeddedllm-b60x4-vllm-qwen3vl30b-chat-ttft
verifiedmedium · 0.75
vLLM 0.12 on 4x Arc Pro B60 (TP=4): Qwen3-VL-30B peak aggregate throughput (c=64)
1000 tok/s aggregate (peak, c=64)Qwen/Qwen3-VL-30B-A3B-Instruct on intel-arc-pro-b60 via vllmbenchmarks/embeddedllm-b60x4-vllm-qwen3vl30b-peak-tps
verifiedmedium · 0.70
vLLM 0.20 TTFT for Qwen3.5-9B at 32K-256K context (dense FA2, chunk 4K)
2944 ms (TTFT, dense, 32K context, cold cache)Qwen3.5-9B (distilled indexer) on ? via vllmbenchmarks/vllm-qwen35-9b-ttft-32k-256k
verifiedmedium · 0.75
vLLM 0.26.0 prefill throughput for Command A W4A4 at 512-128K context on 2xH100
8080 tok/s (saturated prefill, 128K context, B=3, cold cache)CohereLabs/command-a-plus-05-2026-w4a4 (218B total / 25B active MoE, NVFP4 experts) on nvidia-h100-sxm via vllmbenchmarks/vllm-command-a-w4a4-prefill-tok-s-512-128k
verifiedhigh · 0.90
vLLM 0.26.0 TTFT for Command A W4A4 (218B MoE) at 512-128K context on 2xH100
16800 ms (TTFT p50, single-request, 128K context, cold cache)CohereLabs/command-a-plus-05-2026-w4a4 (218B total / 25B active MoE, NVFP4 experts) on nvidia-h100-sxm via vllmbenchmarks/vllm-command-a-w4a4-ttft-512-128k
verifiedhigh · 0.90
vLLM batch invariance is beta and its hardware support is bounded: NVIDIA compute capability 8.0+ and Intel XPU with Triton backend only - no ROCm/HIP, Metal, TPU or CPU claim
80.0 minimum NVIDIA compute capability (8.0) for which batch invariance is supported; the enum has no determinism-support metric so 'quality' is a placeholder? on ? via vllmbenchmarks/vllm-batch-invariance-hardware-support-scope-cc80-and-xpu-triton
verifiedhigh · 0.85
vLLM disaggregated PD serving: Qwen3.5-397B-A17B-NVFP4 on GB200 NVL72, 25K total TPS per GPU
25000 total tokens/s per GPU (aggregate across the rack)Qwen3.5-397B-A17B-NVFP4 on nvidia-gb200-nvl72 via vllmbenchmarks/vllm-qwen35-nvfp4-gb200-nvl72-tps-per-gpu
verifiedmedium · 0.75
vLLM disaggregated PD serving: Qwen3.8-2.4T-A95B NVFP4 on GB300 NVL72, 180 generated tokens/s per user
180 generated tokens/s per userQwen3.8-2.4T-A95B-NVFP4 on nvidia-b300 via vllmbenchmarks/vllm-qwen38-nvfp4-gb300-nvl72-interactivity
verifiedmedium · 0.80
vLLM disaggregated PD serving: Qwen3.8-2.4T-A95B NVFP4 on GB300 NVL72, 5000 total TPS per GPU
5000 total tokens/s per GPU (aggregate across the cluster)Qwen3.8-2.4T-A95B-NVFP4 on nvidia-b300 via vllmbenchmarks/vllm-qwen38-nvfp4-gb300-nvl72-tps-per-gpu
verifiedmedium · 0.80
vLLM on 1x MI250: Llama-2-7b-hf FP16 b1 128in/128out throughput
139.9 tok/s (per-request b1)Llama-2-7b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama27b-fp16-1x-b1-128io-tps
verifiedhigh · 0.85
vLLM on 1x MI250: Llama-2-7b-hf FP16 b64 128in/128out throughput
4477.1 tok/s (aggregate x64)Llama-2-7b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama27b-fp16-1x-b64-128io-tps
verifiedhigh · 0.85
vLLM on 4x MI250: Llama-2-70b-hf FP16 b1 128in/128out throughput
47.7 tok/s (per-request b1)Llama-2-70b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama270-fp16-4x-b1-128io-tps
verifiedhigh · 0.85
vLLM on 4x MI250: Llama-2-70b-hf FP16 b64 128in/128out throughput
1773.4 tok/s (aggregate x64)Llama-2-70b-hf on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-llama270-fp16-4x-b64-128io-tps
verifiedhigh · 0.85
vLLM on 4x MI250: Meta-Llama-3-8B FP16 b1 128in/128out throughput
171.2 tok/s (per-request b1)Meta-Llama-3-8B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-metallam-fp16-4x-b1-128io-tps
verifiedhigh · 0.85
vLLM on 4x MI250: Meta-Llama-3-8B FP16 b64 128in/128out throughput
5922.6 tok/s (aggregate x64)Meta-Llama-3-8B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-metallam-fp16-4x-b64-128io-tps
verifiedhigh · 0.85
vLLM on 4x MI250: Mixtral-8x7B-v0.1 FP16 b64 128in/128out throughput
3136.7 tok/s (aggregate x64)Mixtral-8x7B-v0.1 on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-mixtral8-fp16-4x-b64-128io-tps
verifiedhigh · 0.85
vLLM on 4x MI250: Qwen2-72B FP16 b64 128in/128out throughput
1708.6 tok/s (aggregate x64)Qwen2-72B on amd-instinct-mi250 via vllmbenchmarks/argonne-mi250-vllm-qwen272b-fp16-4x-b64-128io-tps
verifiedhigh · 0.85
vllm on intel-arc-pro-b70: Qwen3.5-27B fp8-e4m3 pp512 tok/s (bentech, dual B70+B60 vLLM FP8 pipeline-parallel)
1009.7 tok/s (pp512, batch 512)Qwen3.5-27B on intel-arc-pro-b70, intel-arc-pro-b60 via vllmbenchmarks/bentech-b70-qwen35-27b-fp8-dual-pp
verifiedmedium · 0.80
vllm on intel-arc-pro-b70: Qwen3.5-27B fp8-e4m3 tg128 tok/s (bentech, dual B70+B60 vLLM FP8 pipeline-parallel)
13.3 tok/s (tg128, batch-1 per-request)Qwen3.5-27B on intel-arc-pro-b70, intel-arc-pro-b60 via vllmbenchmarks/bentech-b70-qwen35-27b-fp8-dual-tg
verifiedmedium · 0.80
vLLM on single MI300X: Meta-Llama-3-8B-Instruct FP16 end-to-end latency, 512 input / 512 output, ROCm 6.2 untuned baseline
4600 ms (average end-to-end latency; source says 'over 4.60 seconds')Meta-Llama-3-8B-Instruct on amd-instinct-mi300x via vllmbenchmarks/vllm-mi300x-llama3-8b-fp16-latency-512in-512out
verifiedmedium · 0.70
vllm-ascend v0.17.0 on Huawei Atlas 300I Pro (Ascend 310P3, 24GB LPDDR4X): Qwen3-4B FP16, ~9 tok/s decode
9.0 ~tok/s decode (reporter's rounded figure)Qwen/Qwen3-4B (FP16, ~8GB weights) on huawei-atlas-300i-pro via niche-vllm-ascendbenchmarks/vllm-ascend-atlas300ipro-qwen3-4b-fp16-decode
verifiedlow · 0.55
vLLM-XPU on Max 1550: Llama-2-7b-hf FP16 b1 128io throughput
127.5 tok/s (per-request b1)Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-llama27b-fp16-1x-b1-128io-tps
verifiedhigh · 0.85
vLLM-XPU on Max 1550: Llama-2-7b-hf FP16 b64 128io throughput
2100.9 tok/s (aggregate x64)Llama-2-7b-hf on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-llama27b-fp16-1x-b64-128io-tps
verifiedhigh · 0.85
vLLM-XPU on Max 1550: Meta-Llama-3-8B FP16 b1 128io throughput
130.7 tok/s (per-request b1)Meta-Llama-3-8B on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-metallam-fp16-1x-b1-128io-tps
verifiedhigh · 0.85
vLLM-XPU on Max 1550: Meta-Llama-3-8B FP16 b64 128io throughput
2123.6 tok/s (aggregate x64)Meta-Llama-3-8B on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-metallam-fp16-1x-b64-128io-tps
verifiedhigh · 0.85
vLLM-XPU on Max 1550: Mistral-7B-v0.1 FP16 b1 128io throughput
126.4 tok/s (per-request b1)Mistral-7B-v0.1 on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-mistral7-fp16-1x-b1-128io-tps
verifiedhigh · 0.85
vLLM-XPU on Max 1550: Mistral-7B-v0.1 FP16 b64 128io throughput
2245.0 tok/s (aggregate x64)Mistral-7B-v0.1 on xpu-intel-dc-gpu-max-1550 via vllmbenchmarks/argonne-max1550-vllm-mistral7-fp16-1x-b64-128io-tps
verifiedhigh · 0.85
vLLM: Meta-Llama-3.1-405B-Instruct-FP8 on 8x H100 (TP=8), aggregate output throughput
291.53 output tokens/s (aggregate, 8 GPUs)meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 on nvidia-h100-sxm via vllmbenchmarks/vllm-llama31-405b-fp8-8xh100-tp8-output-tps
verifiedmedium · 0.80
vLLM: MTP speculative decoding makes latency 76.5% WORSE on Qwen3-Next-80B-A3B-Instruct-FP8 at TP=4 (4x H100)
894.0 ms average end-to-end latency, BASELINE with speculation DISABLED (0.894 s); the MTP-enabled figure is 1.578 s, i.e. +76.5%Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 on nvidia-h100-sxm via vllmbenchmarks/ev-vllm-mtp-regression-qwen3-next-80b-a3b
verifiedhigh · 0.85
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-27B decode at 2.9K prompt (u/neuromacmd)
31.8 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-27b-amdvlk-q4km-2k9-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-27B prefill at 2.9K prompt (u/neuromacmd)
177 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-27b-amdvlk-q4km-2k9-pp
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 109510 context (u/neuromacmd)
72.3 tok/s (single-request decode, batch 1, 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx109510-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 17635 context (u/neuromacmd)
116.8 tok/s (single-request decode, batch 1, 17635-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx17635-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 2.9K prompt (u/neuromacmd)
123.7 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-q4km-2k9-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 35577 context (u/neuromacmd)
100.0 tok/s (single-request decode, batch 1, 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx35577-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 71603 context (u/neuromacmd)
85.4 tok/s (single-request decode, batch 1, 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx71603-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 8824 context (u/neuromacmd)
124.5 tok/s (single-request decode, batch 1, 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx8824-tg
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 109510 context (u/neuromacmd)
380 tok/s (single-request prefill at 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx109510-pp
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 17635 context (u/neuromacmd)
1120 tok/s (single-request prefill at 17635-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx17635-pp
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 2.9K prompt (u/neuromacmd)
948 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-q4km-2k9-pp
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 35577 context (u/neuromacmd)
846 tok/s (single-request prefill at 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx35577-pp
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 71603 context (u/neuromacmd)
561 tok/s (single-request prefill at 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx71603-pp
verifiedlow · 0.50
W7900 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 8824 context (u/neuromacmd)
1422 tok/s (single-request prefill at 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-amdvlk-ctx8824-pp
verifiedlow · 0.50
W7900 llama.cpp RADV: Qwen3.5-27B decode at 2.9K prompt (u/neuromacmd, commit 48cda24)
31.8 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-27b-radv-q4km-2k9-tg
verifiedlow · 0.50
W7900 llama.cpp RADV: Qwen3.5-27B prefill at 2.9K prompt (u/neuromacmd, commit 48cda24)
726 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-27b-radv-q4km-2k9-pp
verifiedlow · 0.50
W7900 llama.cpp RADV: Qwen3.5-35B-A3B decode at 2.9K prompt (u/neuromacmd, commit 48cda24)
114.3 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-radv-q4km-2k9-tg
verifiedlow · 0.50
W7900 llama.cpp RADV: Qwen3.5-35B-A3B prefill at 2.9K prompt (u/neuromacmd, commit 48cda24)
2326 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-radv-q4km-2k9-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-27B decode at 2.9K prompt (u/neuromacmd)
24.4 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-27b-rocm-q4km-2k9-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-27B prefill at 2.9K prompt (u/neuromacmd)
434 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-27B (unsloth Q4_K_M, 16 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-27b-rocm-q4km-2k9-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 109510 context (u/neuromacmd)
61.2 tok/s (single-request decode, batch 1, 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx109510-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 17635 context (u/neuromacmd)
79.4 tok/s (single-request decode, batch 1, 17635-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx17635-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 2.9K prompt (u/neuromacmd)
78.9 tok/s (single-request decode, batch 1, ~2.9K prompt)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-q4km-2k9-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 35577 context (u/neuromacmd)
75.3 tok/s (single-request decode, batch 1, 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx35577-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 71603 context (u/neuromacmd)
67.7 tok/s (single-request decode, batch 1, 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx71603-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 8824 context (u/neuromacmd)
81.7 tok/s (single-request decode, batch 1, 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx8824-tg
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 109510 context (u/neuromacmd)
602 tok/s (single-request prefill at 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx109510-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 17635 context (u/neuromacmd)
1315 tok/s (single-request prefill at 17635-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx17635-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 2.9K prompt (u/neuromacmd)
1001 tok/s (single-request prefill, ~2900 prompt tokens)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-q4km-2k9-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 35577 context (u/neuromacmd)
1096 tok/s (single-request prefill at 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx35577-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 71603 context (u/neuromacmd)
808 tok/s (single-request prefill at 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx71603-pp
verifiedlow · 0.50
W7900 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 8824 context (u/neuromacmd)
1525 tok/s (single-request prefill at 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900 via llama-cppbenchmarks/nm-1s4bggo-w7900-35b-rocm-ctx8824-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 109510 context (u/neuromacmd)
48.3 tok/s (single-request decode, batch 1, two GPUs combined, 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx109510-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 143695 context (u/neuromacmd)
42.6 tok/s (single-request decode, batch 1, two GPUs combined, 143695-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx143695-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 215917 context (u/neuromacmd)
34.3 tok/s (single-request decode, batch 1, two GPUs combined, 215917-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx215917-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 35577 context (u/neuromacmd)
70.3 tok/s (single-request decode, batch 1, two GPUs combined, 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx35577-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 71603 context (u/neuromacmd)
59.4 tok/s (single-request decode, batch 1, two GPUs combined, 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx71603-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B decode at 8824 context (u/neuromacmd)
82.1 tok/s (single-request decode, batch 1, two GPUs combined, 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx8824-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 109510 context (u/neuromacmd)
563 tok/s (single-request prefill, two GPUs combined, 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx109510-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 143695 context (u/neuromacmd)
432 tok/s (single-request prefill, two GPUs combined, 143695-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx143695-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 215917 context (u/neuromacmd)
301 tok/s (single-request prefill, two GPUs combined, 215917-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx215917-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 35577 context (u/neuromacmd)
1380 tok/s (single-request prefill, two GPUs combined, 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx35577-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 71603 context (u/neuromacmd)
782 tok/s (single-request prefill, two GPUs combined, 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx71603-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp AMDVLK Vulkan: Qwen3.5-35B-A3B prefill at 8824 context (u/neuromacmd)
2072 tok/s (single-request prefill, two GPUs combined, 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-amdvlk-ctx8824-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 109510 context (u/neuromacmd)
58.0 tok/s (single-request decode, batch 1, two GPUs combined, 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx109510-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 143695 context (u/neuromacmd)
53.8 tok/s (single-request decode, batch 1, two GPUs combined, 143695-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx143695-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 215917 context (u/neuromacmd)
46.7 tok/s (single-request decode, batch 1, two GPUs combined, 215917-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx215917-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 35577 context (u/neuromacmd)
69.2 tok/s (single-request decode, batch 1, two GPUs combined, 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx35577-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 71603 context (u/neuromacmd)
63.2 tok/s (single-request decode, batch 1, two GPUs combined, 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx71603-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B decode at 8824 context (u/neuromacmd)
74.8 tok/s (single-request decode, batch 1, two GPUs combined, 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx8824-tg
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 109510 context (u/neuromacmd)
854 tok/s (single-request prefill, two GPUs combined, 109510-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx109510-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 143695 context (u/neuromacmd)
665 tok/s (single-request prefill, two GPUs combined, 143695-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx143695-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 215917 context (u/neuromacmd)
523 tok/s (single-request prefill, two GPUs combined, 215917-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx215917-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 35577 context (u/neuromacmd)
1679 tok/s (single-request prefill, two GPUs combined, 35577-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx35577-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 71603 context (u/neuromacmd)
1447 tok/s (single-request prefill, two GPUs combined, 71603-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx71603-pp
verifiedlow · 0.50
W7900+R9700 llama.cpp ROCm: Qwen3.5-35B-A3B prefill at 8824 context (u/neuromacmd)
2148 tok/s (single-request prefill, two GPUs combined, 8824-token context)Qwen3.5-35B-A3B (unsloth UD-Q4_K_M, 21 GB) on amd-radeon-pro-w7900, amd-radeon-ai-pro-r9700 via llama-cppbenchmarks/nm-1s4bggo-w7900-r9700-35b-rocm-ctx8824-pp
verifiedlow · 0.50
Wormhole n150 tt-metal: Mamba-2.8B batch-32 tok/s per user
14.1 tok/s per user at batch 32; 451.2 tok/s aggregateMamba-2.8B on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/ttmetal-n150-mamba-2p8b-b32-tpsu
verifiedmedium · 0.75
Wormhole n150 tt-metal: Whisper distil-large-v3 batch-1 tok/s per user
105.0 tok/s per user at batch 1 (equals aggregate at batch 1)Whisper distil-large-v3 on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/ttmetal-n150-whisper-distil-large-v3-b1-tpsu
verifiedmedium · 0.75
Wormhole n150 tt-metal: Whisper distil-large-v3 batch-1 TTFT
163.0 ms time-to-first-token, batch 1Whisper distil-large-v3 on tenstorrent-wormhole-n150 via niche-tt-metalbenchmarks/ttmetal-n150-whisper-distil-large-v3-b1-ttft
verifiedmedium · 0.75
Wormhole n300 tt-metal: Llama-3.2-11B-Vision TP2 batch-16 tok/s per user
18.1 tok/s per user at batch 16 across 2 chips (TP=2); 289.6 tok/s aggregateLlama-3.2-11B-Vision on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-n300-llama32-11b-vision-tp2-b16-tpsu
verifiedmedium · 0.75
Wormhole n300 tt-metal: Llama-3.2-11B-Vision TP2 batch-16 TTFT
714.0 ms time-to-first-token, batch 16, TP=2Llama-3.2-11B-Vision on tenstorrent-wormhole-n300 via niche-tt-metalbenchmarks/ttmetal-n300-llama32-11b-vision-tp2-b16-ttft
verifiedmedium · 0.75
16GB VRAM: Llama 3.1 8B Q8_0 decode tokens/sec
40.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.1 8B on ? via llama-cppbenchmarks/local-16gb-13b-q4-decode
deprecatedlow · 0.20
24GB VRAM: Llama 3.1 8B Q8_0 decode tokens/sec
50.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.1 8B on ? via llama-cppbenchmarks/local-24gb-13b-q8-decode
deprecatedlow · 0.20
8GB VRAM: Llama 3.1 8B Q4_K_M decode tokens/sec
25.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.1 8B on ? via llama-cppbenchmarks/local-8gb-7b-q4-decode
deprecatedlow · 0.20
M4 Max (128GB): Llama 3.3 70B Q4_K_M decode tokens/sec
18.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.3 70B on apple-m4-max via mlx-lmbenchmarks/local-m4-max-70b-q4-decode
deprecatedlow · 0.20
M5 Max (128GB): Llama 3.3 70B Q4_K_M decode tokens/sec
22.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.3 70B on local-apple-m5-max via mlx-lmbenchmarks/local-m5-max-70b-q4-decode
deprecatedlow · 0.20
M5 Ultra (512GB): Llama 3.3 70B Q4_K_M decode tokens/sec
35.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.3 70B on local-apple-m5-ultra via mlx-lmbenchmarks/local-m5-ultra-70b-q4-decode
deprecatedlow · 0.20
RTX 5070: Llama 3.1 8B Q4_K_M decode tokens/sec
120.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.1 8B on local-nvidia-rtx-5070 via llama-cppbenchmarks/local-rtx-5070-13b-q4-decode
deprecatedlow · 0.20
RTX 5080: Qwen3 32B Q4_K_M decode tokens/sec
65.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Qwen3 32B on nvidia-rtx-5080 via llama-cppbenchmarks/local-rtx-5080-32b-q4-decode
deprecatedlow · 0.20
RTX 5090: Llama 3.3 70B Q4_K_M decode tokens/sec
45.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.3 70B on nvidia-rtx-5090 via llama-cppbenchmarks/local-rtx-5090-70b-q4-decode
deprecatedlow · 0.20
RTX 5090: Qwen3 32B Q4_K_M decode tokens/sec
85.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Qwen3 32B on nvidia-rtx-5090 via llama-cppbenchmarks/local-rtx-5090-32b-q4-decode
deprecatedlow · 0.20
RTX PRO 6000 Blackwell (96GB): Llama 3.3 70B Q8_0 decode tokens/sec
55.0 tok/s - UNSOURCED, DO NOT CITE. Decommissioned 2026-10-04 (bq-): no cited source contains a decode-throughput measurement for this part and no run was performed. The number is retained only for history; the schema forbids nulling value/measured_by.Llama 3.3 70B on nvidia-rtx-pro-6000-blackwell via llama-cppbenchmarks/local-rtx-pro-6000-70b-q8-decode
deprecatedlow · 0.20

Most referenced

The records other records point at. Useful as an entry point: a heavily referenced accelerator or engine is one many decisions in this repo assume.

RecordReferenced byCites
Community-reported vLLM decode throughput on RTX 5090 (sm_120) under WSL2 with CUDA graphs working, Qwen3-14B AWQ44
Adding MI50s over PCIe makes decode WORSE, not better: gpt-oss-120B MXFP4 tg8192 falls 71.2 -> 33.9 -> 21.3 -> 14.3 tok/s from 2 to 4 to 6 to 8 cards under -sm tensor27
llama.cpp on 8xMI300X: DeepSeek-V3 671B Q4_K_M decode throughput at batch 1, 256 generated tokens26
llama.cpp on Radeon AI PRO R9700 (gfx1201, RDNA4): Bonsai-27B Q1_0 decode throughput, batch 124
llama.cpp under ROCm on 2x RX 7900 XTX (gfx1100): Qwen3.8-27B Q4_K_XL batch-1 tensor-split decode throughput25
REPORTED (unverified, Reddit): 16x MI50 32GB serving DeepSeek-V3.2 AWQ 4-bit at 10 tok/s decode and 2000 tok/s prefill at 69k context28
RTX 5070: Llama 3.1 8B Q4_K_M decode tokens/sec26
RTX 5090: Llama 3.3 70B Q4_K_M decode tokens/sec25