llama.cpp PR #25532: Backend sampling multi output (extends -bs to speculative decoding)
Fields
sources/flop3-llamacpp-pr-25532-backend-samplingNotes
Shows backend sampling is the on-device counterpart of the CPU sampler: one backend-sampling graph row per output rather than per sequence, graph reservation and node budgeting for replicated sampler graphs, and an explicit backend_reset hook so graph-owned tensor references are cleared before graph reconstruction. Reports ~8% improvement on RTX 5090 with Qwen3.6-35B Q4_K_M under MTP speculative decoding, with identical acceptance ratio between CPU and backend sampling (rate 0.567-0.856 per workload, tok/s 308.9-437.1 backend vs 308.9-401.1 CPU). Also notes the multi-output path needs a temporary RNG so rejected tokens do not advance the rng - i.e. the rejection sampler and the sampler share RNG state, which is a real coupling when the verify path is moved on device.
Cross-references
This record references no other records.