01 — Hardware selection by workload

How to read this document. It is organised by workload, not by part. For each workload it names (a) the binding constraint, (b) which accelerator records fit that constraint, (c) which gotcha records will bite you there, and (d) — where it matters — what the records cannot answer.

The single structural fact underneath every recommendation here is that batch-1 decode is memory-bound and prefill is compute-bound, with the crossover between them quantified per part in 02-flop-map.md. A "fast GPU" is a fast compute part or a fast memory part; they are rarely the same part, and the workload decides which one you need.

Read NVIDIA H100 SXM alongside this document for the worked example: 80 GB HBM3 at 3,350 GB/s and 989.5 dense bf16 TFLOPS gives a dense-bf16 roofline balance point of 295.4 flop/byte, so a batch-1 decode GEMM (~1 flop/byte) sits roughly 295x below the ridge. That arithmetic, not the headline TOPS figure, is what each recommendation below turns on.

One warning from the records that governs how the whole table set is read: Dense weight GEMMs in the decode step (the per-token matmul tax) notes that "an RTX 4090 has a LOWER bf16 ridge than an H100 (163.9 vs 295.4) and is still dramatically worse at decode. The ridge tells you WHERE the memory wall ends, not HOW WIDE it is. Wall width is what matters at batch 1, and there the ranking is set by absolute bandwidth (MI300X 5.3 TB/s > H100 3.35 TB/s > RTX 4090 1.008 TB/s) with no reference to FLOPS at all." Do not rank decode candidates by ridge point.

1. Single-user local chat (laptop or desktop, one stream)

Binding constraint: memory capacity, then memory bandwidth. Not FLOPS.

A single stream is a batch-1 decode loop. The records are unambiguous that this is bandwidth-bound at every batch size: decode GEMM has arithmetic intensity exactly equal to the batch size Dense weight GEMMs in the decode step (the per-token matmul tax), so at batch 1 it is ~1 flop/byte, and the hardware consequence stated in that record is that "memory bandwidth, not peak TOPS, sets single-stream decode throughput, so a part with high FLOPS and modest bandwidth is a throughput part, not a latency part." At batch 1 there is also a second, non-bandwidth tax: per-kernel launch overhead Per-kernel launch and dispatch overhead at batch 1, which sets a floor on inter-token latency that no FLOPS improvement moves.

Fits, and why:

Apple capacity-vs-bandwidth warning, stated by the record itself. The M3 Ultra record refuses to collapse these into one ranking: 512 GB at 819 GB/s versus the RTX PRO 6000 Blackwell's 96 GB at 1792 GB/s gives "5.3x capacity, 0.46x bandwidth". For a model that fits on both, the record's conclusion is that the discrete card "is roughly 2x faster per stream". Apple publishes no GPU TFLOPS for any of these parts — the flops array is empty by choice, not by gap, and "any 'M-series TFLOPS' number in circulation is a third-party die-level extrapolation, not an Apple figure."

Gotchas that bite here:

• NVFP4 models with --dtype bfloat16 produce garbled output on any GPU without native FP4 — blocker, silent wrong output. Serving any NVFP4 model with --dtype bfloat16 on SM < 100 (RTX 4090 SM89, V100 SM70) "loads, starts and serves, but every response is garbled." Workaround: --dtype float16 on pre-Blackwell-datacenter parts, because the FP16 path maps the 5-bit exponent directly.
• Setting VLLM_MARLIN_INPUT_DTYPE=fp8 enables Marlin W4A8-FP8 on sm_121a where it silently produces garbage — blocker, silent wrong output, and the corrupted path is "~2.5% FASTER end-to-end, so throughput benchmarking selects for the bug." Affects NVIDIA GB10 (DGX Spark, sm_121a).
• On gfx1151 the llama.cpp HIP backend produces corrupt output where Vulkan is correct with byte-identical flags, so cross-backend comparison is the only clean detector — major. On gfx1151 (Strix Halo APU, ROCm 7.2.4) "the HIP backend is wrong and Vulkan is right", verified same machine, same build, byte-identical flags. Treat Vulkan as the reference there.
• On gfx1201 the Vulkan backend loses 4.7-6.7x decode throughput to HIP at hidden_size >= 4096, and effective bandwidth collapses to ~70-100 GB/s on a 640 GB/s card — major. On gfx1201, a hidden-size cliff in the Vulkan decode path; the record's advice is to compute effective decode bandwidth from file size before concluding a card is slow.
• A llama.cpp ROCm commit raised WikiText-2 perplexity from 7.72 to 3024 on a Q4_0 model with no error, and only llama-per perplexity caught it — major, silent. llama-perplexity on ROCm/gfx1151 returns a PPL jump "of two orders of magnitude" vs the same model on another backend. Treat a PPL jump of two orders of magnitude as a hard stop.
• Hybrid Gated DeltaNet models keep fast prompt processing but collapse ~25x in decode past ~80K context — major. Hybrid GDN GGUF models on this stack: decode falls from ~33 t/s at 68K KV position to 1.4 t/s at 91K, even with the model fully resident on GPU. Cap -c below roughly 80K for these models.
• mlx-lm --kv-bits passes its startup check, then crashes on the first request for hybrid sliding-window models — major. The one Apple-Silicon trap in the repo: mlx_lm.server passes /health then the first request raises NotImplementedError: RotatingKVCache Quantization NYI on hybrid-attention models (Gemma 4 uses sliding-window attention on 35 of 42 layers at 26B-A4B).

What the records cannot answer: whether a given consumer card's llama.cpp or MLX/Vulkan path is fast for a specific model, and any per-part tok/s ranking. The benchmark records that do exist are heavily model-and-backend specific and mostly draft status (e.g. llama.cpp on Radeon AI PRO R9700 (gfx1201, RDNA4): Bonsai-27B Q1_0 decode throughput, batch 1 is draft, MLX-LM on MacBook Pro 14-inch M3 Pro 18GB: Qwen3-4B-Instruct-2507 4-bit generation throughput is draft). There is no controlled consumer-card decode comparison in the repo.

2. A 24 GB consumer card — the specific 24 GB decision

Binding constraint: capacity at exactly 24 GB, which is why this deserves its own section. Four records in the repo carry 24 GB and they are not interchangeable.

The 3090's record states the trap explicitly: "The two 24 GB cards are not interchangeable on bandwidth; recommendations that quote '24 GB RTX 3090' from a Ti benchmark silently overstate this card." The 3090 Ti is also the only GeForce in the repo with NVLink (2-way SLI was the only consumer multi-GPU path on Ampere), and the 3090 family carries the NVLink bridge on the product page.

The A10 is the memory-cheap datacenter option: 150 W and 600 GB/s for the same 24 GB. Its record notes 125 dense bf16 TFLOPS, which is "roughly a tenth" of H100 SXM's FLOPS for roughly a fifth of the bandwidth — it is a capacity-and-power part.

Cross-tier comparison from the NVIDIA GeForce RTX 4080 (16 GB) record, which states the 16 GB position explicitly: "16 GB at 716.8 GB/s is ~71% of the 4090's 1008 GB/s. But the binding constraint at 16 GB is capacity, not bandwidth." So within the consumer line, below ~24 GB capacity binds first and above ~24 GB bandwidth binds — but see NVIDIA GeForce RTX 5090 (32 GB, 1792 GB/s) for where the crossover actually lands today.

What the records cannot answer: whether 24 GB is enough for a specific model at a specific context length. That calculation needs the KV-cache figures from KV cache movement: block copy, prefix sharing, prefill-to-decode handoff, and offload and Multi-head Latent Attention: KV cache compressed to a single low-rank latent vector per token and is model-specific; the repo has model records but no per-model capacity planner.

3. Long-context serving (32k+ context, KV cache is the budget)

Binding constraint: memory capacity for the KV cache, and memory bandwidth to stream it. Interconnect matters only if you shard. KV cache movement: block copy, prefix sharing, prefill-to-decode handoff, and offload states this most directly: "KV movement is the class that decides whether a long-context service is viable at all, and it is the one that most rewards hardware with fast local memory and low-latency scale-up links rather than fast FLOPs."

Two structural facts from the flop records that should drive the architecture choice:

• Decode attention has AI ~1 flop/byte at every context length and every batch size (Attention over the KV cache in the decode step): flops = 4Ld and bytes = 4Ld, both linear in L, so "growing the context does NOT raise arithmetic intensity, and batching does NOT raise it either." Cost per token grows linearly with context and no kernel fixes that.
• Prefill attention crosses from memory-bound to compute-bound at roughly L ~ 300 tokens on H100 SXM dense bf16, and is "about 27x the ridge point" at L=8k (Attention over the prompt (prefill / training-style causal attention)). So long-context serving is a two-constraint workload: a compute-bound prefill phase and a bandwidth-bound decode phase over a capacity-hungry cache.

Architecture choices that change the KV budget — all from flop records, not speculation:

• Multi-head Latent Attention: KV cache compressed to a single low-rank latent vector per token: DeepSeek-V3's MLA cache is 68.6 KiB/token at bf16 versus 5,856 KiB/token for the MHA equivalent — "an 85.3x reduction, or 1.17% of the MHA cache" — and it is rated stronger than MHA in the same table, unlike GQA/MQA.
• Sliding-window and interleaved local/global attention: local layers are O(L*w) in prefill and O(1) in cache: Gemma 3 at w=1024 with 5 local layers per 1 global; local-layer cache is bounded at w and "does not grow with context at all."
• SSM / linear-RNN decode: the constant-size recurrent state update that replaces the KV cache entirely: SSM/recurrent layers have no KV cache at all; the state "is CONSTANT in context length, so unlike KV attention its byte count never grows." Nemotron-H-8B is 48 Mamba / 4 attention.
• YOCO, NSA, DSA, RetNet, lightning attention, HGRN2, Kimi Linear: attention replacements that change the complexity class: YOCO stores the global cache once instead of per layer; NSA/DSA reduce the work set but "still pay a full-context scoring pass."
• FlashMLA decode: FP8/FP4 KV cache with scales inlined per token (DeepSeek MLA): FlashMLA V4.1 stores 528 bytes/token at fp8; the fp4 variant 288 bytes/token.

Fits, and why:

The MI300X/MI250 trap that affects capacity planning: MI250X's 128 GB is "the WHOLE OAM module, i.e. 2 x 64 GB HBM2e" and you must "NOT read '128 GB' as one 128GB device, and do NOT read it as two 64GB devices either" AMD Instinct MI250X. The MI300X record makes the same point for 192 GB across 8 XCDs, and MI325X for 256 GB. This is a shared address space across compute dies, not a device count.

Gotchas that bite here:

• vLLM's --gpu-memory-utilization is a fraction of TOTAL VRAM, so KV cache block allocation fails even with memory visibly free — major. "No available memory for the cache blocks." The KV budget is a first-class constraint and the error message names the fix (--gpu_memory_utilization, --max-num-batched-tokens, --max-model-len).
• An FP8 KV cache write path in Python is graph-unsafe and silently corrupts attention under CUDA graphs while --enforce-eager looks fine — blocker, silent corruption. 8x A800-80GB (SM80, no native FP8 tensor cores) with --kv-cache-dtype fp8_e5m2, TP=8: the server runs and returns wrong output.
• EAGLE/MTP speculative decoding costs 6-10 points of prefix-cache hit rate, and hybrid layouts make the recompute 100x larger — major. Multi-round and repeated-prompt workloads lose ~6-10 points of prefix-cache hit rate (97.5% → 86.9-91.7%) when speculative decoding is on. Measure hit rate with and without speculative_config.
• Enabling MTP speculative decoding together with prefix caching silently costs ~20% accuracy — blocker. On a finetuned Qwen3.6 35B-A3B with MTP (num_speculative_tokens=2) plus --enable-prefix-caching, internal classification accuracy drops ~20% versus the identical setup without.
• Enabling the V1 engine shrinks usable context several-fold on the same hardware and model — major. Same model and flags: vLLM 0.6.x ran at 12K context on 4x RTX 3070; 0.7.0 with V1 enabled cannot exceed ~3K. Re-tune for the V1 engine rather than carrying V0 values.
• On RDNA, models with head_dim != 128 silently fall back to Triton paged attention and decode collapses at long context — major. Qwen3.6-27B on gfx1100: 12.1 tok/s at 518 tokens of context but 4.2 tok/s at 32K. Only the 16 full-attention layers degrade. Check head_dim against 128 and the measured KV block size against 16.

What the records cannot answer: the KV-cache budget for any specific model at a specific context length, as a number you can plug into a sizing decision. The ingredients exist (KV cache movement: block copy, prefix sharing, prefill-to-decode handoff, and offload gives the per-token formula, Multi-head Latent Attention: KV cache compressed to a single low-rank latent vector per token and Sliding-window and interleaved local/global attention: local layers are O(L*w) in prefill and O(1) in cache give architecture multipliers) but no record composes them into a table. This is the single highest-value gap in the repo for this workload.

4. High-concurrency multi-user server

Binding constraint: it moves. At low concurrency, bandwidth and KV capacity. At high concurrency, FLOPS — because batching is the only lever that raises decode arithmetic intensity, and its ceiling is the roofline ridge point.

The mechanism, in the records' own terms: decode GEMM AI = B, so "the entire throughput story of an inference service is 'what is the average effective B'", and continuous batching is the mechanism that keeps B high (Dense weight GEMMs in the decode step (the per-token matmul tax), Iteration-level scheduling: the overlap of prefill and decode in one batch). The crossover the records give you: you need batch on the order of 150-300 for bf16 to be compute-bound on H100 SXM, and roughly half that in fp8. For an MoE the bar is dramatically higher — see below.

The MoE number is the one that should drive a multi-user MoE server decision. MoE expert GEMMs (the sparse feed-forward compute): with E experts and top-k, AI = B*(k/E) flop/byte, because you must stream every expert weight to do k/E of the work. Worked against DeepSeek-V3's own config (256 routed experts, 8 activated, k/E = 1/32): "an MoE decode step needs a batch of about 32*295 = 9,440 to reach compute-bound where the same-width dense model needs ~295." The record's design consequence: "a MoE model with the same parameter count as a dense model is not faster to serve — it is slower."

Fits, and why:

Scale-up fabric matters here, and the records disagree about what it costs. AMD Instinct MI300X and MI325X carry 8 Infinity Fabric links at 128 GB/s; the interconnect record is contested on direction — AMD's MI300X page states 128 GB/s "with no direction stated", and the record retains both readings (64 GB/s per-direction bidirectional, or 128 GB/s per-direction), noting a 64 GB/s per-direction figure is "roughly 7x below NVLink 4 per direction" (AMD Infinity Fabric link (4th generation)). By contrast NVIDIA H100 SXM exposes NVLink 4 at 900 GB/s bidirectional per GPU and NVIDIA NVLink Switch 4 900 GB/s between GPUs. Both figures are retained; neither record resolves the direction question. Treat "MI300X multi-GPU bandwidth headroom is tighter than H100's" as contested, not settled.

Where the measured numbers actually land (benchmark records, with the methodology the schema demands): Per-user decode tok/s, MPT-7B static batch 1, 1x A100 57.6 tok/s at batch 1 versus Per-user decode tok/s, MPT-7B static batch 64, 1x A100 12.5 tok/s per user at batch 64 with aggregate 800 tok/s — the cleanest illustration in the repo that aggregate throughput and per-user latency trade directly.

AMD can beat NVIDIA on aggregate throughput — with caveats you must quote. On Llama 3.1 405B FP8 at TP=8, MI300X vs H100, vLLM, Llama 3.1 405B FP8 TP8: output token throughput at 128-in/4096-out (MI300X wins 1.62x) records 3171 output tok/s aggregate for MI300X against H100 reference column for MI300X vs H100, vLLM/TRT-LLM, Llama 3.1 405B FP8 TP8: output token throughput at 128-in/4096-out 1957 for H100 — and the record's own note says MI300X "wins all four" 405B shapes in that table (1.25x, 1.62x, 1.13x, 1.28x; geomean 1.31x). But on 70B the same table's record says the opposite: MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out 15105 against H100 reference column for MI300X vs H100, vLLM/TRT-LLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out 15810, i.e. 0.96x, geomean 0.94x across four shapes. So the honest summary is: MI300X is at parity or slightly behind on 70B and clearly ahead on 405B, and the record attributes the crossover to "the bandwidth-to-capacity balance at 8-way TP for large weights," not to CDNA versus Hopper.

Two caveats the records insist on, which must travel with any quote of these numbers: the NVIDIA half was "measured by AMD, not by NVIDIA" and the two sides used different engines (vLLM on AMD, TensorRT-LLM on NVIDIA), so "the delta blends silicon and software"; and the comparison is not memory-capacity-neutral (MI300X has 192 GB against H100's 80 GB). Batch size, KV-cache dtype, GEMM tuning state and driver versions were all omitted by the source.

Gotchas that bite here:

• vLLM 0.30.0's FA4 kernel ignores num_splits on SM90, costing up to 48% decode throughput — major. Decode on H100 (SM90) is 48% slower per token than on v0.26.0 at batch 1 and 36% slower at batch 4, while TTFT is flat. The fix is backend selection, not hardware: attention_backend=TRITON_ATTN recovers ~70%.
• Enabling speculative decoding with FlashInfer silently downgrades CUDA graphs to PIECEWISE, costing ~16% decode — major. A single logger.warning announces a CUDA-graph downgrade; decode throughput 47.5 tok/s on FLASHINFER vs 55.2 on FLASH_ATTN (+16%), changing only --attention-backend.
• Draft-model speculative decoding is not lossless at temperature 0 when the target model is quantized — major. Under greedy sampling, --spec-type draft-dspark or draft-mtp produces different text than the same server without it, when the target is quantized. n-gram speculation is verified lossless.
• On GPUs in NVIDIA Confidential Computing mode, vLLM's V2 model runner serves silent garbage — blocker. On H100 NVL in CC mode inside a TDX guest, vLLM 0.29.0 with the default V2 model runner serves and every response is garbage (token id 0 repeated). VLLM_USE_V2_MODEL_RUNNER=0 fixes it.
• FP8 on RDNA3 blows the 600 s engine-ready timeout on first start even though weights load in seconds — major. vLLM on gfx1100 with an FP8 checkpoint aborts during startup on the engine-ready timeout (default 600 s); measured first init far exceeds it. Set VLLM_ENGINE_READY_TIMEOUT_S=1800 for the first cold start; once ~/.triton/cache is warm, init drops to ~49 s.
• vLLM's AITER integration gate silently excludes RDNA3 (gfx1100) — major. On gfx1100 Radeon, vLLM logs "AITER is not found or not supported on the current platform" and falls back to emulation. (Note this gotcha's affects list names amd-radeon-pro-w7900 and amd-radeon-rx-7900-xtx, which have no accelerator records — a dangling reference.)

What the records cannot answer: sustained tokens-per-dollar. The supply records give per-GPU-hour prices but no benchmark ties a specific price to a specific measured tokens/s on the same part with the same engine, so any "$/Mtok" figure would be a fabrication. See 06 below and the cost section.

5. Batch / offline throughput

Binding constraint: FLOPS, at high enough batch that decode becomes compute-bound. This is the only workload where peak dense TFLOPS is the right metric — and it is exactly the workload where the dense-vs-sparse trap is most expensive.

The dense/sparse trap is the first thing to check on any part you are buying for batch throughput. NVIDIA's own printed figures are sparse-marked and dense is one half. From NVIDIA B200: "NVIDIA's Blackwell datasheet states 'Specifications in sparse. Dense is one-half of the sparse spec shown'" — so a '10 PFLOPS FP8' claim for B200 is sparse, and dense FP8 is 5 PFLOPS. From NVIDIA H100 SXM: NVIDIA prints 989 TF32 / 1,979 BF16+FP16 / 3,958 FP8 all "WITH SPARITY", and dense is exactly half — and the record notes "anyone reading the sheet without the footnote halves it twice." The repo's stored dense values: B200 2500 bf16 / 5000 fp8 dense; H100 SXM 989.5 / 1979 dense.

Two cases where the convention differs, both retained:

• NVIDIA L40S prints explicit dense|sparse pairs (FP16 "362.05 | 733*"), so the left column is dense — and the record notes "the marketing headline for L40S is 733 'FP8 TFLOPS' and that is already the DENSE number, while other vendors' 733 figures are sparse."
• NVIDIA B300 (Blackwell Ultra) is contested (confidence 0.6): the Blackwell Ultra datasheet prints FP4 as '20 PFLOPS | 15 PFLOPS' under a footnote reading "Specification in Sparse | Dense" — a sparse|dense pair, the opposite convention from the other rows — so dense FP4 is 15 PFLOPS, not 10. The record also flags the INT8 row as a probable broken text-layer extraction (330/307 TOPS against ~10 PFLOPS implied) and does not record INT8.
• Intel Arc Pro B60 is the counterexample where labelling is good: the Intel Arc B580 record notes Intel defines its 233 TOPS as "the peak throughput when running XMX workloads with INT8 datatype and DENSE models."

Fits, and why:

Two accuracy warnings that must travel with any batch-throughput number:

• AMD Instinct MI350X / AMD Instinct MI355X: TF32 is software-emulated on CDNA4 (AMD's own footnote). "Do not carry a MI300X TF32 figure forward to MI355X" — MI300X has 653.7 dense TF32.
• NVIDIA H20 is draft, confidence 0.45, and is the lowest-confidence record in the accelerator set: NVIDIA has never published an H20 datasheet, every figure is from a third-party aggregator with vendor_claim: false. Its shape is the interesting part — 148 dense FP16 TFLOPS against 96 GB at 4.0 TB/s, i.e. "roughly 2/3 of H100 SXM's dense 989 TFLOPS but a HIGHER bandwidth-per-FLOP ratio, so H20 is a memory-rich, compute-poor Hopper." Do not quote it as verified.

Where measured throughput lands — MLPerf-family benchmark records, all with full methodology: MI355X, MLPerf Inference v6.0 Offline: Llama 2 70B WMXFP4 on 8xMI355X, aggregate output throughput 103,480 output tok/s aggregate on WMXFP4 weights with FP8 KV cache; MI355X, MLPerf Inference v6.0 Offline: gpt-oss-120b (MXFP4/FP8 MoE) on 8xMI355X, aggregate output throughput 95,004 on a MoE; SGLang: DeepSeek-V3 671B FP8 on 96x H100 with PD disaggregation + large-scale EP, 22300 output tokens/s per node 22,300 on 96x H100. These are not comparable to each other — different models, formats, node counts and concurrency. Cite them only with their model and node count attached.

Gotchas that bite here:

• The ROCm Triton w4a16 path asserts on GPTQ models with a non-zero-group qzeros layout — major. Serving GPTQ-Int4 fails on the Triton w4a16 path with a qzeros.shape assertion on ROCm.
• Marlin weight-only INT4 GEMM: repacking so dequant is free and tensor cores stay saturated's stated limitation: Marlin "is not yet optimized for Hopper" — on Hopper Marlin is the fallback, not the fast path.
• Forcing the FlashInfer attention backend fails on head_dim 256/512 models — major. Engine init aborts with "head_size not supported" for large head dims on RTX PRO 6000 Blackwell; the same model runs under TRITON_ATTN.

6. Cost-per-token-sensitive

Binding constraint: capacity-to-bandwidth ratio per dollar, and — more often than not — the fact that the records give you prices and throughputs in different units, so the join has to be done by hand and is not yet done anywhere in the repo.

The supply records are careful about this, and their own warnings should be carried forward:

• Rental bands (all observed 2026-10-03, per-GPU-hour, verified hosts): RTX 3090 median $0.1498, RTX 4090 $0.4289, RTX 5090 $0.4896 (Peer-to-peer rental index: consumer GeForce (4090 / 3090 / 5090 per-GPU-hour)); L40S $0.7085 and RTX 6000 Ada $0.6289 (Peer-to-peer rental index: datacenter PCIe L40S and RTX 6000 Ada (per-GPU-hour)); H100 SXM $2.4704 (Peer-to-peer rental index: NVIDIA H100 SXM5 (per-GPU-hour, verified hosts)); H200 SXM $5.0005 (Peer-to-peer rental index: NVIDIA H200 SXM (per-GPU-hour, verified hosts)); B200 $7.9695 (Peer-to-peer rental index: NVIDIA B200 (per-GPU-hour, verified hosts)).
• Rental and purchase prices do not move together. The consumer rental record states the ratio explicitly: the RTX 3090 rents at ~0.35x the RTX 4090 rate while its asking purchase price is ~0.46x. "Neither number can stand in for the other."
• The cheapest credible accelerator hour in the slice is a TPU, not a GPU, and it is not a drop-in substitute: v5e at $1.20 per chip-hour, less than AWS g6's L4 at $1.67, but "XLA/JAX only" and quota-gated rather than payment-gated (Google Cloud TPU v4 / v5e / v5p (per chip-hour, in a pod)).
• Hyperscaler Hopper is mostly allocation-only. AWS EC2 P5 (NVIDIA H100 SXM, on-demand + Capacity Blocks) is the "scarcity record for this slice" — AWS's own price list carries p5.48xlarge at an explicit $0.00/hour alongside the real $55.04. Same sentinel on p4d and p5en. Microsoft Azure ND H200 v5 (ND96isr, 8x H200) records a 0% spot discount on H200 v5 — "$84.80, byte-identical to the on-demand rate... A zero-discount spot price is what a sold-out fleet looks in a price API."
• A stale belief this repo explicitly corrects: Oracle Cloud Infrastructure BM.GPU.H100.8 / BM.GPU.H200.8 records that the historical "OCI is dramatically cheaper for GPU" claim "no longer holds" for Hopper — $10.00/GPU-hour against AWS p5's $6.88. What Oracle still has is the bare-metal shape with no vCPU/RAM bundling.
• The 48 GB Ada tier is the value tier, per the rental record: L40S and RTX 6000 Ada "rent within ~13% of each other... while sitting ~6.5x below H100 SXM and ~1.6x above RTX 4090."
• Refurbished consumer is the cheapest 24 GB per dollar in the whole slice: a Micro Center refurbished RTX 3090 24GB at $1,099.99 is called "the most cost-effective 24GB-per-dollar option in this entire slice" (Micro Center in-store refurbished GeForce RTX 3090 / 3090 Ti 24GB tier). The channel is store-inventory only — "it will not ship."
• Do not buy used Intel silicon. Newegg Refreshed (first-party refurbished) Intel Arc B580 12GB tier records refreshed B580 at $199.99-$259.99 versus used $290-$499 and new $329.99-$590: "Buying USED Intel silicon is strictly dominated."

Fits: for a cost-sensitive single-user or small-batch deployment the value ordering the records support is RTX 3090 (rental $0.1498/h, Peer-to-peer rental index: consumer GeForce (4090 / 3090 / 5090 per-GPU-hour); refurb purchase $1,099.99, Micro Center in-store refurbished GeForce RTX 3090 / 3090 Ti 24GB tier) → L40S at $0.7085/h or RTX 6000 Ada at $0.6289/h, both 48 GB (Peer-to-peer rental index: datacenter PCIe L40S and RTX 6000 Ada (per-GPU-hour)) → H100 SXM ($2.4704/h rental, Peer-to-peer rental index: NVIDIA H100 SXM5 (per-GPU-hour, verified hosts); $6.88/h AWS on-demand when you can get it at all, AWS EC2 P5 (NVIDIA H100 SXM, on-demand + Capacity Blocks)). For cost-sensitive high-concurrency serving, the cheapest H100 hour in the slice is Runpod at $3.49/GPU-hour (Runpod Pods (dedicated H100 / H200 / A100 GPU instances)), below Lambda AI cloud GPU instances (NVIDIA H100 SXM, self-serve)'s $3.99 and well below AWS's $6.88.

Gotchas that bite here: none of the gotcha records are about price. The price traps are in the supply records themselves and are quoted above.

What the records cannot answer — and this is the biggest honest gap in this document: cost per token. There is no record that joins a per-GPU-hour price to a measured tokens/s for the same part, engine, model and precision. Every figure above is either a price or a throughput, never both. A reader wanting $/Mtok must either run the benchmark themselves or accept a hand-built ratio whose error bars nobody has written down. MARKET SUMMARY: GPU price bands by segment, observed 2026-10-03 (new / used / rental) holds the aggregate bands and is deliberately price_usd: null on the grounds that "one point estimate for 'the GPU market' would be a fabrication" — that same discipline is why no cost-per-token record exists.

7. Embedding / reranking workloads (compute-bound, the inverse of everything above)

Binding constraint: FLOPS, at batch 1. This is the one workload class where a bandwidth-bound consumer card is the wrong purchase.

The records are unanimous and give the arithmetic. Cross-encoder reranking: N independent bidirectional forwards, compute-bound at batch 1: "Reranking N passages is N INDEPENDENT forward passes with no KV-cache reuse, so it is prefill-shaped and compute-bound even at batch 1 — the exact opposite of the generative decode regime." Its AI is ~t flop/byte from weight amortization alone, "t times HIGHER than the ~1 flop/byte of a decode GEMM."

Embedding / batch encoding: BERT-style bi-encoder over documents, compute-bound at batch 1 gives the threshold explicitly: AI = B·t, so compute-bound once B·t ≥ 295.4 on H100 SXM — B ≥ 3 at t = 128. Corpus work pushes B far above that.

Corpus cost shape decides the bill:

• Embedding: linear in corpus size — "each document is encoded exactly once and the vector is cached."
• Reranking: linear in N but N is top-k of first-stage retrieval, so "rerank top-20 vs top-100 is a 5x difference in a bill that nothing else can reduce."

Fits: the same dense-FLOPS parts as batch serving — NVIDIA B200 (2500 dense bf16) and AMD Instinct MI355X (2500 dense bf16 matrix) lead, with NVIDIA H100 SXM (989.5) as the reference. Notably the 48 GB Ada tier is well-matched here rather than capacity-constrained: NVIDIA L40S at 362 dense bf16 TFLOPS and NVIDIA RTX 6000 Ada Generation at 48 GB.

The ColBERT inversion is worth knowing. ColBERT reference implementation: late interaction "INVERTS the usual arithmetic-intensity intuition for a GPU... Hardware selection therefore needs BOTH: a high-FLOPS part for encoding the corpus, and high-bandwidth/high-capacity RAM or storage for the index, which is the opposite mix from what an LLM decode part wants." Index size is on the order of ~100x a single-vector index.

What the records cannot answer: encoder throughput per GPU-second by part. The records say the engine matters more (HuggingFace Text Embeddings Inference (TEI) "the reference implementation most teams reach for first, and it is what Hugging Face itself deploys"; vLLM pooling models (embedding, rerank, late interaction)'s own docs disclaim a performance advantage) but there is no benchmark record for any encoder on any accelerator. Only two Apple decode benchmarks exist at all, and neither is an encoder.

What the records cannot answer, by workload

Honest negatives, collected in one place because they are the next research priorities:

1. Cost per token (all workloads). No price↔throughput join exists. Highest value.
2. KV-cache budget per model per context length (§3). The formula exists in KV cache movement: block copy, prefix sharing, prefill-to-decode handoff, and offload, the architecture multipliers exist, the composition does not.
3. A controlled consumer-card decode comparison (§1). Existing benchmarks are backend- and model-specific and mostly draft; there is no apples-to-apples ranking.
4. Sustained multi-user throughput per dollar for AMD vs NVIDIA at equal model and precision (§4). MI300X vs H100, vLLM, Llama 3.1 70B FP8 TP8: output token throughput at 128-in/2048-out is one such comparison and is valuable precisely because there are so few.
5. Anything about Huawei, Biren, Enflame, MetaX, Horizon or Moore Threads as a purchase. Huawei Ascend 910B is contested at confidence 0.45 with "no primary source at all"; Biren BILI 166 series (166C inference card / 166M / 166L OAM modules), Enflame S60 (SCORPIO-AO) AI accelerator, MetaX MXC500 / MXN100 GPGPU (training and inference), Horizon Robotics Journey 6P (Nash BPU automotive SoC), Moore Threads MTT S4000 LLM training/inference accelerator and Moore Threads MTT S5000 (PH100, Pinghu architecture) are all draft, several with no memory or compute figure at all. Huawei Ascend 910D is "DELIBERATELY ALMOST EMPTY" because only existence is substantiable, not shipping. Horizon Robotics Journey 6P (Nash BPU automotive SoC) records an explicit negative finding that it is not a plausible source of LLM inference hardware.
6. Volta and anything below sm75 as a 2026 purchase. vLLM requires compute capability 7.5+, HuggingFace Text Embeddings Inference (TEI) excludes V100 and Titan V explicitly, and FlashInfer is unsupported below sm75.
7. Non-NVIDIA, non-Apple parts as first-class multi-user serving. The ROCm caveats in vLLM and SGLang are specific and severe enough that "AMD works" is not a supportable statement; see 03-engine-selection.md.

Related documents

• 02-flop-map.md — why each workload above binds the way it does.
• 03-engine-selection.md — which engine can actually run on these parts.
• 04-quant-selection.md — the lever that changes the capacity column in every table above.
• 05-known-traps.md — every gotcha record, grouped by severity.