SpecExec massively parallel speculative execution on RTX 4090 with RAM offload: Llama-2-70B batch 1
8.3 speedup_ratio: 8.3x speedup over sequential offloaded autoregressive decoding (throughput increased to 5.66 tok/s with generation rate 13.46 accepted tokens per iteration). BASELINE: standard autoregressive decode with RAM offloading on RTX 4090.
Fields
id
benchmarks/sd-specexec-offload-llama2-70b-b1Metric
speedup_ratio
Value
8.3
Unit
speedup_ratio: 8.3x speedup over sequential offloaded autoregressive decoding (throughput increased to 5.66 tok/s with generation rate 13.46 accepted tokens per iteration). BASELINE: standard autoregressive decode with RAM offloading on RTX 4090.
Model
meta-llama/Llama-2-70b-chat-hf
Methodology
Batch size 1 (interactive decode), RAM offloading over PCIe. Target model: Llama-2-70B GPTQ 4-bit; draft model: Llama-2-7B. Speculative budget: 256 candidate tokens. Hardware: single NVIDIA GeForce RTX 4090 24GB. Dataset: OpenAssistant (OASST).
Measured by
third_party
Reproducible
yes
notes
Table 3 in arXiv:2406.02532. Highlights the extreme win of massive speculation in RAM offloading settings: when target forward passes take seconds due to PCIe transfer, evaluating a tree of 256 candidate tokens in parallel yields 13.46 accepted tokens per step, reaching 5.66 tokens/s (8.3x speedup). On 16-bit unquantized Llama-2-70B with budget 2048, speedup reaches 18.7x (3.12 tok/s).
Notes
Table 3 in arXiv:2406.02532. Highlights the extreme win of massive speculation in RAM offloading settings: when target forward passes take seconds due to PCIe transfer, evaluating a tree of 256 candidate tokens in parallel yields 13.46 accepted tokens per step, reaching 5.66 tokens/s (8.3x speedup). On 16-bit unquantized Llama-2-70B with budget 2048, speedup reaches 18.7x (3.12 tok/s).