A100 40GB SXM4 llama.cpp b10808: Ornith-1.5-35B-A3B Q4_K_M (MoE, 3B active) decode
148 tok/s decode (tg128, batch-1, ctx 0.5k, FA on, kv q8_0); prefill 3291 t/s
Fields
id
benchmarks/llamabench-a100sxm40-ornith15-35b-a3b-q4km-tg128Metric
decode_tok_s
Value
148
Unit
tok/s decode (tg128, batch-1, ctx 0.5k, FA on, kv q8_0); prefill 3291 t/s
Model
Ornith-1.5-35B-A3B Q4_K_M (MoE, 3B active)
Methodology
llamabench.ai community row, reporter @wmeddie: llama.cpp build b10808, flash-attention on, KV cache q8_0, 0.5k context, no speculative decoding. tg128 batch-1 decode / site prefill. n=1.
Measured by
third_party