inference/research

06 — Glossary

One line per term, defined the way this repo uses it, with the records where the repo actually discusses it. Every id below resolves (python tools/query.py get <id>); no number appears here without a record carrying it.

This is a practitioner vocabulary, not a textbook. Where the industry uses a term loosely, the definition given is the one the repo's records are consistent with, and the record list is where the definition comes from.

Serving metrics

Prefill — the prompt-processing phase that runs the whole input through the model once before any token is emitted; compute-bound, and its rate is what sets time to first token. Attention over the prompt (prefill / training-style causal attention) Dense weight GEMMs in the decode step (the per-token matmul tax)

Decode — the token-generation phase, run one step at a time; memory-bound at small batch, which is why it is the phase that rewards batching and KV-cache engineering. Dense weight GEMMs in the decode step (the per-token matmul tax) Attention over the KV cache in the decode step

TTFT (time to first token) — wall-clock latency from request submission to the first output token. Dominated by prefill, and the one metric tensor parallelism reliably improves. TTFT, MPT-7B, 512-token prompt, batch size 1, 1x A100 40GB

ITL (inter-token latency) — the interval between consecutive output tokens after the first. In this repo some itl_ms records hold end-to-end time-to-last- token instead; read the unit field, not the metric name. MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8 batch 1: end-to-end time-to-last-token at 128-in/2048-out

TPOT (time per output token) — mean per-token generation time, i.e. steady- state ITL. See "terms with no coverage yet".

TPS — tokens per second. Meaningless unqualified: it is either per user or aggregate across all in-flight requests, and the two differ by the concurrency. This repo has one hardware configuration measured both ways — aggregate 13.9x higher at batch 64 while per-user falls 4.6x. Per-user decode tok/s, MPT-7B static batch 1, 1x A100 Aggregate output throughput, MPT-7B static batch 64, 1x A100 Per-user decode tok/s, MPT-7B static batch 64, 1x A100

TTLT (time to last token) — end-to-end request latency, i.e. TTFT plus all decode steps. MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8 batch 1: end-to-end time-to-last-token at 128-in/2048-out

Goodput — throughput measured only for requests meeting a latency SLO, which is the metric disaggregated serving is designed to raise. Discussed in DistServe (research prototype); no benchmark record reports a measurement.

Scheduling and cache management

Paged KV / PagedAttention — storing the KV cache in fixed-size blocks that map to non-contiguous device memory, so allocation is on demand and sequence length does not have to be reserved in advance. KV cache movement: block copy, prefix sharing, prefill-to-decode handoff, and offload

Radix cache / prefix cache — a prefix tree over token sequences so a request sharing a system prompt or few-shot prefix with a previous one reuses that KV instead of recomputing it. Multi-turn chat with prefix caching: two flop profiles from one workload (cache hit vs miss)

Prefix-cache hit rate — the fraction of a request's KV that came from cache rather than recomputation. Speculative decoding measurably erodes it. EAGLE/MTP speculative decoding costs 6-10 points of prefix-cache hit rate, and hybrid layouts make the recompute 100x larger

Chunked prefill — splitting a long prompt into pieces that interleave with decode steps, so a large prefill cannot monopolize the batch and starve running sequences. Chunked prefill and prefill/decode interleaving: deciding when to interrupt a prefill for a decode step

Continuous batching — adding and removing requests from the running batch at every step, rather than waiting for a fixed batch to drain. The single largest throughput lever in a serving engine. Iteration-level scheduling: the overlap of prefill and decode in one batch

Disaggregated serving (prefill/decode disaggregation) — running prefill and decode on separate pools of GPUs so each phase uses hardware suited to it and KV is transferred between them. Raises throughput, adds KV-transfer cost. Prefill/decode disaggregation: the KV handoff tax that separates a compute-bound phase from a memory-bound phase DistServe (research prototype)

KV-cache quantization — storing K and V in fewer bits than the weights. Independent of weight quantization and often applied to the same model. LLM-QAT KV cache quantization (4-bit KV inside a QAT regime)

KV eviction — discarding cached KV under pressure, keeping a predicted-useful subset. H2O (heavy-hitter KV eviction) - NOT KV quantization

Attention variants

MHA — multi-head attention, one KV head per query head; largest KV footprint.

GQA (grouped-query attention) — several query heads share one KV head, cutting KV cache proportionally. YOCO, NSA, DSA, RetNet, lightning attention, HGRN2, Kimi Linear: attention replacements that change the complexity class GPT-OSS 20B (8 KV heads across 24 layers)

MQA (multi-query attention) — all query heads share a single KV head; the extreme case of GQA, smallest KV cache. FlashDecoding / split-KV attention: turning a single-token decode into a bandwidth-saturating GEMM

MLA (multi-head latent attention) — compressing KV into a low-rank latent space and caching that instead of the full K/V, so KV cache shrinks far more than GQA can at comparable quality. Multi-head Latent Attention: KV cache compressed to a single low-rank latent vector per token DeepSeek-V3 (kv_lora_rank 512, 128 heads expanded from latent)

Sliding-window attention — each token attends only to the previous W tokens, so KV cost is independent of context length. Usually interleaved with a few full- attention layers, which is what makes a hybrid model necessary rather than merely optimized. Sliding-window and interleaved local/global attention: local layers are O(L*w) in prefill and O(1) in cache

Cross-attention KV sharing — in encoder-decoder and multimodal models, sharing or caching the encoder-side KV across decoder steps so it is computed once. Whisper large-v3 Parakeet CTC 1.1B

Split-KV decode / FlashDecoding — splitting the KV dimension across SMs in decode so the bandwidth-bound phase uses more of the machine. FlashDecoding / split-KV attention: turning a single-token decode into a bandwidth-saturating GEMM

Attention sink — the tendency of a small set of early positions to absorb disproportionate attention; the basis of most KV-sparsity methods. DuoAttention (head-wise retrieval/streaming KV sparsity) - NOT KV quantization H2O (heavy-hitter KV eviction) - NOT KV quantization

AMD multi-die terms

These are the terms AMD's own architecture notes use, and they appear throughout the MI2xx/MI3xx records.

GCD (GPU Complex Die) — the accelerator compute tiles on an XCD.

XCD (eXpandable Complex Die) — one GCD plus its HBM stacks and I/O; an individual accelerator complex. AMD Instinct MI300X AMD Instinct MI300A

GIL (GCD-to-Infinity Link) — the die-to-die interconnect within a package, distinct from XGMI which runs between packages. AMD Instinct MI300X AMD Instinct MI355X

Sparsity and interconnect

Dense vs 2:4-sparse — 2:4 keeps 2 of every 4 adjacent weights in a fixed pattern the tensor cores can skip; a 2:4 figure and a dense figure are not comparable and vendor material conflates them. NVIDIA B200 AMD Instinct MI210

Structured sparsity — a sparsity pattern with hardware-visible regularity (2:4 blocks, block-sparse rows), as opposed to unstructured pruning a general GEMM cannot exploit. AMD Instinct MI210 NVIDIA GeForce RTX 5090

NVLink / NVSwitch — NVIDIA's scale-up domain; NVLink is the link, NVSwitch the switch that makes many links act as one. NVIDIA NVLink 3.0 NVIDIA NVLink 4.0 NVIDIA NVLink 5.0 NVIDIA NVLink 6.0 NVIDIA GB200 NVL72 (per-accelerator)

XGMI — AMD's between-package die-to-die/coherent interconnect. AMD xGMI (Socket/Inter-Chip Global Memory Interconnect) AMD Instinct MI300X

HCCS — Huawei's chip-to-chip scale-up interconnect. Huawei HCCS (Hierarchical Chip-to-Chip Scale)

ICI — Google TPU's interconnect fabric, its scale-up and scale-out domain. Google Cloud TPU Inter-Chip Interconnect (ICI)

PCIe — the host-attached link; much slower than a scale-up fabric, which is why form factor (SXM vs PCIe) changes performance and not just price. PCI Express 5.0 NVIDIA H100 PCIe

Quantization notation

W4A16 — 4-bit weights, 16-bit activations: weight-only, the usual choice because it costs no accuracy and needs no activation calibration. AWQ (activation-aware weight quantization) GPTQ (one-shot approximate second-order PTQ)

W4A4 — 4-bit weights and activations; much harder, and mostly research kernel work rather than a silicon path. GGUF Q4_K_M (4-bit K-quant, medium)

W8A8 — 8-bit both ways, via outlier migration rather than naive rounding. SmoothQuant (W8A8 via offline outlier migration)

MXFP4 / NVFP4 — microscaling 4-bit float formats with per-block scales; MXFP4 shares one scale across 32 elements, NVFP4 uses 16 plus a per-tensor scale. MXFP4 (OCP microscaling FP4, E8M0 scale) NVFP4 (Blackwell block-scaled E2M1 + E4M3)

GGUF — llama.cpp's container and quantization format; the "K-quants" are per-tensor recipes, not global bpw targets. GGUF Q4_K_M (4-bit K-quant, medium)

imatrix (importance matrix) — a calibration matrix of per-input-channel activation magnitudes used by llama.cpp to decide which weights matter most, so higher importance can get more bits. GGUF imatrix (importance matrix) - the calibration file that determines WHICH weights get more bits

The roofline model

Roofline — attainable performance = min(peak FLOPs, memory bandwidth x arithmetic intensity). Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures LLM Inference Unveiled: Survey and Roofline Model Insights

Arithmetic intensity — FLOPs per byte moved; the x-axis position of a kernel on the roofline. Attention over the prompt (prefill / training-style causal attention)

Ridge point / balance point — the intensity at which a kernel becomes compute-bound rather than bandwidth-bound; set by the machine's FLOPs-to-bytes ratio. Dense weight GEMMs in the decode step (the per-token matmul tax) NVIDIA H100 SXM

Memory-bound vs compute-bound — whether a kernel is limited by bytes or by FLOPs. Decode is memory-bound at small batch; prefill is compute-bound. Dense weight GEMMs in the decode step (the per-token matmul tax) Attention over the prompt (prefill / training-style causal attention)

Memory

HBM — high-bandwidth memory on-package with the GPU; the reason datacenter accelerators have both high FLOPs and high bandwidth. NVIDIA H100 SXM (HBM3, 80GB, 3350 GB/s) AMD Instinct MI300X (hbm3, 192GB, 5300 GB/s)

GDDR — graphics DRAM, cheaper and lower-bandwidth than HBM; the consumer and lower-end accelerator path. Intel Arc Pro B60

LPDDR — low-power DRAM, for APUs, Apple silicon and Grace-class memory. Apple M4 Max (MacBook Pro 14-inch, 2024)

Unified memory / UMA — one address space for CPU and accelerator, removing an explicit copy at the cost of bandwidth. Apple M4 Max (MacBook Pro 14-inch, 2024) AMD Instinct MI250X

MIG — Multi-Instance GPU, hardware partitioning of one GPU into isolated smaller instances with their own memory and compute. NVIDIA H100 SXM

Parallelism

Tensor parallelism (TP) — splitting individual matrix multiplications across GPUs; requires all-reduce per layer, so it is interconnect-bound. Multi-GPU tensor-parallel inference: the per-layer all-reduce and the batch at which it matters

Pipeline parallelism (PP) — splitting layers across GPUs; needs a micro-batch in flight to avoid bubbles. vLLM

All-reduce — the collective every tensor-parallel layer ends with; on the critical path once per layer. Multi-GPU tensor-parallel inference: the per-layer all-reduce and the batch at which it matters

Expert parallelism (EP) — the MoE analogue of TP, distributing experts across GPUs. SGLang

MoE

MoE (mixture of experts) — a model with many parallel expert FFNs of which only a few run per token. Mixtral 8x22B

Active parameters — the parameters actually computed per token. For a 671B DeepSeek-style model it is ~37B, so the serving problem is KV and expert placement, not dense FLOPs. DeepSeek-V3

Expert routing — selecting which experts a token goes to; top-k selection is the standard mechanism. MoE gating: router GEMM, softmax/top-k, and the token scatter-gather around it

Shared expert — an always-on expert applied to every token, which adds to per-token active parameters and is not optional. MoE expert GEMMs (the sparse feed-forward compute)

MTP (multi-token prediction) — training the model to predict several future tokens per forward pass, used as a zero-draft speculative scheme. Enabling MTP speculative decoding together with prefix caching silently costs ~20% accuracy SGLang

Speculative decoding — drafting several tokens cheaply then verifying them in one forward pass; a bandwidth win that is not always a win, and is not lossless on a quantized target. Draft-model speculative decoding is not lossless at temperature 0 when the target model is quantized

Execution-level terms

CUDA graph — capturing a launch sequence so the CPU submits it once instead of per-step, removing launch overhead. Degrading to PIECEWISE silently costs ~16% decode. Enabling speculative decoding with FlashInfer silently downgrades CUDA graphs to PIECEWISE, costing ~16% decode

Kernel launch overhead — per-launch CPU cost; material at small batch where the GPU is not the bottleneck. Per-kernel launch and dispatch overhead at batch 1

Warp specialization — dedicating warps to producer vs consumer roles so TMA copies overlap with math. FlashAttention-3 (Hopper): warp specialization, TMA, and FP8 to overlap matmul with softmax

Arithmetic intensity of attention — the S x S score-matrix traffic that tiling exists to remove. FlashAttention IO-aware tiling (FA1): eliminating the S x S HBM round trip

Terms with no coverage yet

Used in real benchmark methodology, discussed nowhere in this repo. Do not cite a record for these.

  • TPOT as a distinct metric — the repo reports itl_ms and tok_s_per_user instead; no record defines or measures TPOT on its own.
  • Goodput as a measurement — the term appears in the DistServe material, but no benchmark record reports goodput against an SLO.
  • P99 / tail latency as a recorded metric — the phrase appears inside interconnect records and NVFP4 gotchas, never as a benchmark metric value.
  • Long-context quality benchmarks (RULER, needle-in-a-haystack) — long-context behavior is covered from the performance side (Hybrid Gated DeltaNet models keep fast prompt processing but collapse ~25x in decode past ~80K context) but no retrieval-quality-at-length record exists.
  • Prefix-cache hit rate as a benchmark record — the concept is documented in a gotcha, but hit rate itself is never a measured record.
  • TPOT-style streaming/chunking artifacts — the effect of --stream-interval on measured ITL is recorded inside vLLM disaggregated PD serving: Qwen3.5-397B-A17B-NVFP4 on GB200 NVL72, 25K total TPS per GPU rather than as its own term.

Related