Host-side sampling loop: full-logits D2H transfer plus the CPU sampler chain

This is the same V-wide tensor the sampling record describes, but measured in HOST time rather than device bytes, and that changes the answer by four orders of magnitude. On the device the logit read is negligible - 4*128256 = 501.0 KiB per token at fp32 against a Llama-3-8B batch-1 decode step that streams 2*8.03e9 = 16.06 GB of weights, i.e. 0.0031% of the step's bytes, and ~0.15 us of H100 time at 3350 GB/s. If a sampler reads those logits from HBM on the device and never ships them anywhere, it is free. But llama.cpp's default server path moves the ENTIRE logits row to the host every decode step: at one slot the issue measures 608.3 KB per D2H copy and identifies it as exactly 152064 vocab x 4 B = 608,256 B, one complete fp32 logits row, occurring once per step (938 copies in a 10 s window at ~94 decode steps/s); at 32 slots the same per-step copy measures 16,157.4 KB, which is at most 32 rows (32 x 608,256 B = 19,464,192 B) and in practice fewer because slots are not all resident every step. Then the CPU walks it. CPU sampling of on-CPU samples landing inside >=1 ms GPU idle gaps at 32 slots attributes 45.2% to common_sampler_sample() and 12.7% to llama_token_data_array_partial_sort_inplace(), 58.6% of in-gap host time to the sampler in total. The C++ source explains why top_k does not save you: llama_token_data_array_partial_sort_inplace always walks the full candidate array (a std::partial_sort for npartial <= 128, otherwise a two-pass 128-bucket histogram over logits clamped to [-10,+10]), llama_sampler_temp_impl scans the whole array for the max even at temp = 0, and llama_sample_dist builds a std::discrete_distribution over an iterator spanning the array. Measured confirmation: top_k=1 changes throughput by nothing (711 vs 717 tok/s at 32 slots) while partial_sort is still 12.7% of in-gap CPU. So the cost model is O(n_vocab) per step, times batch, times fp32 width - with NO arithmetic-intensity saving anywhere, because the array is walked twice (once for the histogram/partition, once for the distribution) at 4 bytes per element on a single core.

CLASSIFICATION ARGUMENT: filed under class 'sampling' with bound_by 'launch_overhead' rather than 'memory', because the arithmetic intensity that makes sampling cheap is computed on the wrong side of the PCIe bus. The device-side AI for logit processing is 1-2 flop/byte and the payload is 0.0031% of a Llama-3-8B decode step - essentially free. The host-side chain has no roofline at all: it is a single-threaded O(n_vocab) walk plus a synchronous 608 KB transfer, and it is not overlappable because the next step cannot start until the host has read the row it just synchronised on. That is a launch/dispatch-shaped cost - CPU-side, per-step, independent of how much GPU work the step contains - which is why launch_overhead is the correct bound and why the existing custom-kernel-launch-overhead record is the natural neighbour (both are 'the GPU is idle waiting for the host'). Requested enum addition, for the schema owner: a distinct 'host_logit_transfer' bound_by value would separate this from per-kernel launch overhead, since the two have different cures (device sampling vs CUDA Graphs). DESIGN CONSEQUENCE, and the part worth remembering: this is the ONLY per-token class in the taxonomy whose cost rises with batch size in absolute terms while its share of useful GPU work rises too. decode_gemm gets cheaper per token as B grows (AI = B); decode_attention and sampling's device side are batch-invariant; this class multiplies. So on a many-slot server the fix for throughput is not more FLOPS or more bandwidth - it is removing the host round trip. The same issue's control experiments make the boundary precise: disabling FlashAttention, changing perf timers, raising slot count to 64 with 64K context, and doubling HTTP threads each moved throughput by <10%, while removing the D2H+CPU sampler moved it 48%. HARDWARE READING: absolute HBM bandwidth still sets the device floor (a 5090 at 1792 GB/s beats a 4090 at 1008 GB/s at batch 1 regardless of this class), but host-interface quality - PCIe topology, single-core speed, pinned-memory paths - sets the ceiling, and a 2.5x faster GPU with a worse host path can lose. CAVEAT on the single-slot case: the same issue measured no effect at 1 slot (100.1 -> 101.4 tok/s, TPOT 9.65 -> 9.43 ms) and attributes the sampler's 5.1% there to a much smaller idle budget (0.185 s in 10 s). So this is a concurrency-regime problem, and a single-user benchmark is structurally blind to it. CAVEAT on attribution: with -bs the issue explicitly declines to quote a GPU-busy figure (cudaGraphLaunch calls rose 144 -> 502 in the same window, so the node-level trace does not cover both configurations equally). Throughput, TPOT and memcpy volume are direct measurements; the GPU-utilisation percentages are only quoted for the default path. HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.