BigDL-LLM self-speculative decoding on Intel GPU Max 1550 (1 tile): ~1.35x FP16 next-token-latency speedup
1.35 x vs non-speculative FP16 baseline (mean of Llama-2-7B/13B, Mistral-7B rows)
Fields
id
benchmarks/intel-bigdl-max1550-fp16-selfspec-speedupMetric
speedup_ratio
Value
1.35
Unit
x vs non-speculative FP16 baseline (mean of Llama-2-7B/13B, Mistral-7B rows)
Model
Llama-2-7B/13B + Mistral-7B (avg)
Methodology
VENDOR-MEASURED (Intel BigDL-LLM article 816988, tests Jan 2024). Next-token latency, batch 1, greedy, input 1024 + output 128 tokens, INT4 and FP16, BigDL-LLM 2.5.0b20240111/20240123, PyTorch 2.1.0a0, IPEX 2.1.10+xpu, oneAPI 2024.0, driver 775_20, Ubuntu 22.04.2. Host: 2S Xeon 8480+; Flex 170 system = 256GB DDR5-4800, Max 1100 = 512GB, Max 1550 single tile = 1024GB. Absolute latencies are published only as bar charts - values here are NOT transcribed; the record captures the methodology and the text-stated deltas. Vendor text: 'Self-Speculative Decoding brings about 35% improvements for FP16 on next token latency' on Max 1550 single tile; ~40%+ on Max 1100.
Measured by
vendor
Reproducible
yes
notes
Vendor text-stated ~35% latency improvement for FP16 with self-speculative decoding, measured on Max 1550 single tile (~40%+ on Max 1100). The same article's INT4 chart includes Flex 170, but its absolute values are graph-only and are NOT transcribed here. RATIO record only - no tok/s implied.
Notes
Vendor text-stated ~35% latency improvement for FP16 with self-speculative decoding, measured on Max 1550 single tile (~40%+ on Max 1100). The same article's INT4 chart includes Flex 170, but its absolute values are graph-only and are NOT transcribed here. RATIO record only - no tok/s implied.