inference/research
draftlow · 0.50updated 2026-10-03

llama.cpp on Radeon AI PRO R9700 (gfx1201, RDNA4): Bonsai-27B Q1_0 decode throughput, batch 1

61.66 tok/s (tg64, mean of two interleaved runs: 61.97 and 61.34)
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
benchmarks/llamacpp-radeon-r9700-gfx1201-q1-0-decode-tok-s
Metric
decode_tok_s
Value
61.66
Unit
tok/s (tg64, mean of two interleaved runs: 61.97 and 61.34)
Model
Bonsai-27B (Q1_0, 3.53 GiB)
Methodology
PER-REQUEST decode throughput at batch 1: -p 0 (no prompt) -n 64 -r 5 (five repetitions). Two interleaved readings taken on the same hardware and the same base commit: unpatched master 29.69 and 29.11 t/s; patched build 61.97 and 61.34 t/s. Mean 29.402 -> 61.657 t/s, i.e. +109.7%. The 61.66 stored here is the mean of the two patched readings. Hardware: 1x Radeon AI PRO R9700 (gfx1201), ROCm/TheRock 7.13. Correctness was separately verified exhaustively over all 65536 possible q values with bit-identical output to the reference, plus test-backend-ops MUL_MAT. OMITTED BY SOURCE: VRAM size, GPU clock, power limit, host CPU, driver version beyond 'ROCm/TheRock 7.13', KV-cache size, and thread count. The author explicitly notes no NVIDIA hardware was available, so the CUDA branch of the patch is unmeasured.
Measured by
third_party
Reproducible
yes
notes
Recorded for two reasons. First, it is the best-documented RDNA4 llama.cpp decode measurement I could find with a real command line, exact quantization, mean-of-repeats, and a baseline measured on the same machine in the same session - the methodology most people actually want from a community PR. Second, and more important for a knowledge base: it is a worked example of an AMD-SPECIFIC SILENT PENALTY. HIP has no hardware __byte_perm, so it lowers to a software routine, and the Q1_0 unpack was calling it 20 times per 32-element chunk, making Q1_0 decode VALU-bound on AMD rather than bandwidth-bound. Replacing it with one hardware v_perm_b32 per nibble doubled decode. The general lesson: sub-2-bp quantization formats can be pathologically slow on AMD through software lowering rather than through any hardware limit, so a llama.cpp tok/s figure on AMD is a statement about the build, not just the GPU. The PR's own caveat is preserved: Q1_0 still will not beat Q2_0 on this hardware even after the fix - that appears to be a property of sub-2-bp formats rather than of any implementation. ACCELERATOR ID CAVEAT: no accelerators/amd-radeon-ai-pro-r9700 record exists in this repo; planned id, validator will warn. measured_by is third_party and status draft because the PR author self-reports and discloses that an AI agent built the trees and ran the benchmarks - good-faith, not independently audited. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): format_id holds '2bit', a bit width rather than a record id. What was actually measured is llama.cpp Q1_0 on Bonsai-27B, a 2-bit trellis format; data/quantization/ has no Q1_0 record, and mapping it to gguf-iq1-s or to ternary would misstate what the number describes. Left unchanged. This record also already carries its own accelerator-id caveat in notes about a planned id. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-radeon-ai-pro-r9700). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface. FORMAT_ID (2026-10-14 backfill): source names the format 'Q1_0' verbatim in methodology ('Q1_0 unpack'); that is llama.cpp's Q1_0 type, NOT TQ1_0 (ternary, different block struct) and NOT IQ1_S. No data/quantization record exists for plain Q1_0, so format_id is null rather than guessed - the dangling '2bit' slug it previously carried was wrong (Q1_0 is ~1.25 bpw, not 2-bit).

Notes

Recorded for two reasons. First, it is the best-documented RDNA4 llama.cpp decode measurement I could find with a real command line, exact quantization, mean-of-repeats, and a baseline measured on the same machine in the same session - the methodology most people actually want from a community PR. Second, and more important for a knowledge base: it is a worked example of an AMD-SPECIFIC SILENT PENALTY. HIP has no hardware __byte_perm, so it lowers to a software routine, and the Q1_0 unpack was calling it 20 times per 32-element chunk, making Q1_0 decode VALU-bound on AMD rather than bandwidth-bound. Replacing it with one hardware v_perm_b32 per nibble doubled decode. The general lesson: sub-2-bp quantization formats can be pathologically slow on AMD through software lowering rather than through any hardware limit, so a llama.cpp tok/s figure on AMD is a statement about the build, not just the GPU. The PR's own caveat is preserved: Q1_0 still will not beat Q2_0 on this hardware even after the fix - that appears to be a property of sub-2-bp formats rather than of any implementation. ACCELERATOR ID CAVEAT: no accelerators/amd-radeon-ai-pro-r9700 record exists in this repo; planned id, validator will warn. measured_by is third_party and status draft because the PR author self-reports and discloses that an AI agent built the trees and ran the benchmarks - good-faith, not independently audited. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): format_id holds '2bit', a bit width rather than a record id. What was actually measured is llama.cpp Q1_0 on Bonsai-27B, a 2-bit trellis format; data/quantization/ has no Q1_0 record, and mapping it to gguf-iq1-s or to ternary would misstate what the number describes. Left unchanged. This record also already carries its own accelerator-id caveat in notes about a planned id. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (amd-radeon-ai-pro-r9700). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface. FORMAT_ID (2026-10-14 backfill): source names the format 'Q1_0' verbatim in methodology ('Q1_0 unpack'); that is llama.cpp's Q1_0 type, NOT TQ1_0 (ternary, different block struct) and NOT IQ1_S. No data/quantization record exists for plain Q1_0, so format_id is null rather than guessed - the dangling '2bit' slug it previously carried was wrong (Q1_0 is ~1.25 bpw, not 2-bit).

Cross-references