Rockchip RK3588 NPU, RKLLM w8a8: ChatGLM3-6B decode tok/s (vendor benchmark table)
4.98 tok/s (w8a8, 128-in/64-out; largest model Rockchip benchmarks)
Fields
id
benchmarks/comm-rk3588-chatglm3-6b-decodeMetric
decode_tok_s
Value
4.98
Unit
tok/s (w8a8, 128-in/64-out; largest model Rockchip benchmarks)
Model
ChatGLM3-6B (w8a8)
Methodology
VENDOR-MEASURED benchmark table (airockchip/rknn-llm, RK3588, w8a8 quant, 128-token input / 64-token output). ChatGLM3-6B = 4.98 tok/s - the LARGEST model they benchmark, already ~5 tok/s. Full table: Qwen2-0.5B 41.6; Qwen2.5-1.5B 16.7; Phi3-3.8B 7.45; ChatGLM3-6B 4.98. A real 7B (Llama-3.1-8B, Qwen2.5-7B) lands ~3-4 tok/s decode and wants ~5-6 GB for weights+KV at w8a8 - fits a 16 GB board only just.
Measured by
vendor
Reproducible
yes
notes
TRAP for 7B+. RKLLM is real and actively maintained with a wide model-conversion list, but the measured benchmark table makes the wall obvious. Fine for a sub-4B on-device assistant or VLM demo; a trap if the goal is 7B+.
Notes
TRAP for 7B+. RKLLM is real and actively maintained with a wide model-conversion list, but the measured benchmark table makes the wall obvious. Fine for a sub-4B on-device assistant or VLM demo; a trap if the goal is 7B+.