inference/research
verifiedhigh · 0.85updated 2026-10-03

llama.cpp src/llama-sampler.cpp (master) - host-side sampler chain and full-vocabulary candidate walk

repo

Fields

id
sources/flop3-llamacpp-sampler-source
Publisher
ggml-org/llama.cpp
Kind
repo
Accessed
2026-10-03
notes
Primary source for the STRUCTURE of a CPU sampler, which is what makes it O(n_vocab) regardless of top_k. llama_token_data_array_partial_sort_inplace cur_p->data onto std::partial_sort when npartial <= 128; otherwise it falls back to llama_token_data_array_partial_sort, which makes TWO passes over the full array to build a 128-bucket histogram over logits clamped to [-10,10] and then sorts only the top buckets. llama_sample_dist builds a std::discrete_distribution<int> over a probs_iterator spanning the candidate array. llama_sampler_temp_impl scans the whole array for the max when temp <= 0. So: the candidate array is walked in full at every step, and the cost is a function of n_vocab, not of top_k. Matches the measurement in issue #27050 that top_k=1 changes nothing.

Notes

Primary source for the STRUCTURE of a CPU sampler, which is what makes it O(n_vocab) regardless of top_k. llama_token_data_array_partial_sort_inplace cur_p->data onto std::partial_sort when npartial <= 128; otherwise it falls back to llama_token_data_array_partial_sort, which makes TWO passes over the full array to build a 128-bucket histogram over logits clamped to [-10,10] and then sorts only the top buckets. llama_sample_dist builds a std::discrete_distribution<int> over a probs_iterator spanning the candidate array. llama_sampler_temp_impl scans the whole array for the max when temp <= 0. So: the candidate array is walked in full at every step, and the cost is a function of n_vocab, not of top_k. Matches the measurement in issue #27050 that top_k=1 changes nothing.

Cross-references

This record references no other records.