Gotchas (475)

The traps, ordered by severity. The blocker-severity entries come first because they are the only ones that produce confident, plausible, wrong answers rather than a crash. Read before deploying, not after.

contested 9draft 14verified 452→ sources (2562)

Filter

class

severity

status

sort

A KGW-style watermark is designed for stochastic sampling - under temperature 0 the signal it relies on is not produced

[major] securityThe same watermarked model returns detectable text at temperature 1.0 and undetectable text at temperature 0. Any batch that goes all-greedy contributes no watermark signal at all, which also corrupts the detection statistic for a mixed batch.gotchas/sec-watermark-greedy-decoding-defeats-the-signal

contestedmedium · 0.75

A logits processor that reports is_argmax_invariant() is never called under greedy decoding - the sampler returns before that stage

[major] securityYou ship a safety logits processor on a model served with temperature 0. It passes tests on sampled traffic. Callers who use greedy decoding get unfiltered output, and the only symptom is that the filter appears to work most of the time rather than failing loudly.gotchas/sec-logit-mask-argmax-invariant-skipped-under-greedy

contestedmedium · 0.80

CXL 3.x and 4.0 records carry nearly 2x the PCIe figure for the same physical layer, because the pass that 'corrected' CXL for direction applied the fix to the wrong state

[major] hardwareTwo records that describe the SAME physical layer hold different numbers, and each record's own prose says so. cxl-3-x.bandwidth_gbps is 236.0 while its own bandwidth_basis asserts verbatim: "The recorded value now equals pcie-gen6 exactly, which is correct since CXL 3.x IS the P…gotchas/w8r-cxl-family-broke-the-pcie-direction-convention-it-copied-fro

contestedmedium · 0.80

CXL 3.x and 4.0 records carry nearly 2x the PCIe figure for the same physical layer, because the pass that 'corrected' CXL for direction applied the fix to the wrong state

contestedmedium · 0.80

On gfx906 llama.cpp, Flash Attention is often SLOWER than leaving it off - the community FAQ says do not enable -fa 1 if you can avoid it, while vLLM users on the same cards insist on the Triton FA path

[major] kernelllama.cpp, Llama-3-8B Q6_K, 1 MI50: with -fa 1, PP512 = 332 t/s and TG128 = 54 t/s; with -fa 0, PP512 = 576 t/s and TG128 = 58 t/s. In diffusion the same inversion: quad-cross-attention 7.2 s, split-cross-attention 7.3 s, pytorch-cross-attention 7.6 s, but flash-attention 10.4 s …gotchas/comm-gfx906-flash-attn-slower-than-off

contestedlow · 0.45

On gfx906, float16 activations overflow to inf and vLLM emits runs of '!!!!' instead of tokens; bf16 weights are rejected for MXFP4 and fp32 activation compute is the only reported fix, at a large throughput cost

[blocker] kernelGeneration degrades into runs of the literal character '!' - described in the community as output that 'spits !!!!' - rather than a crash or a clean error. Reported on: Qwen3.5 (AWQ, on 4x MI50) where 'using dtype fp16 generated garbage "!!!!"' until fp32 activations with fp16 KV…gotchas/comm-gfx906-fp16-garbage-bang-output

contestedlow · 0.50

On MI50 the choice of VBIOS - not the driver - decides whether you get a 32 GB BAR, ReBAR, P2P, the 225 W or 178 W power cap, and whether your cards hang; a Dell T7820 will not even allocate more than 1 GB BAR without a flash

[major] hardwareDistinct named MI50 VBIOSes behave differently in ways that matter for inference. Reported: (a) BAR size - a user with 3x MI50 in a Dell T7820 could not make room for more than 1 GB BAR until flashing the V420 BIOS with ReBAR support; another confirms the post-flash state as 'Mem…gotchas/comm-gfx906-vbios-flash-selects-bar-power-and-stability

contestedmedium · 0.70

Router-side admission control observes load without reserving capacity, and fails open on stale telemetry — so N routers each admit a request that all N replicas then must hold

[major] operationsAdding a second router replica makes overload worse rather than better. Each replica independently decides a worker is healthy and admits a request, and the N simultaneously-admitted requests are dispatched to replicas that are all near capacity — so the effective admission limit…gotchas/ops-gateway-admission-has-no-reservation

contestedmedium · 0.70

Whether a watermark survives paraphrasing is genuinely contested in the literature - the pro side and the con side both have primary sources

[major] securityTwo papers in the same literature reach opposite conclusions on the same question. One finds watermarks survive human and machine paraphrasing; another shows a handful of black-box generations from the watermarked model is enough to reverse-engineer the scheme and defeat it. Citi…gotchas/sec-watermark-detection-fragile-to-paraphrasing-contested

contestedmedium · 0.70

A 24GB consumer card can be the harder card to run a given model on than an 80GB datacenter card - capacity is not the only axis, and no source in this mining pass documents the comparison

[major] hardwareThe awkward case practitioners keep describing: a 24GB RTX 4090 cannot run a model that a used A100 80GB runs comfortably. Nothing here is a mystery of driver behaviour - it is the arithmetic of 24GB versus 80GB forcing the consumer owner down the quantization ladder until qualit…gotchas/comm-4090-24gb-can-run-less-than-used-a100-80gb

draftlow · 0.30

Community consensus on where the quantization quality cliff is: below Q4 is unusable, and below Q8 is unusable for code - but this is forum opinion, and the one measurement in the thread does not test it

[minor] formatThe recurring practitioner judgement in r/LocalLLaMA, stated plainly by timschwartz: 'Quantization is lossy compression. It shaves off a few IQ points in exchange for making it smaller. Stuff below Q4 is usually too smooth-brained to be useful.' The sharper version, from Randomma…gotchas/comm-community-quant-quality-floor-opinion

draftlow · 0.40

Diffusion LM serving shape is fundamentally different from AR

[major] configDiffusion language models cannot be served with standard AR inference machinery. Continuous batching, paged KV, speculative decode, and FlashDecoding all assume one forward pass per token. Diffusion LMs do S full forward passes per generation, with no KV cache reuse across steps.gotchas/diffusion-lm-serving-shape-gotcha

draftlow · 0.60

Dynamic resolution makes serving cost unpredictable

[blocker] configServing cost is unpredictable because the token count depends on the input image size, which is attacker-controlled. A malicious user can send a 4096x4096 image to cause a 65,536-token prefill, exhausting GPU memory.gotchas/dynamic-resolution-makes-serving-cost-unpredictable

draftlow · 0.50

Hybrid attention-SSM batch-size regime crossover

[major] configThe optimal batch size for a hybrid attention-SSM model is set by the SSM state footprint, not by available KV memory. At small batch and long context, the hybrid wins (SSM state is smaller than KV cache). At large batch and short context, pure attention wins (paged KV can pack s…gotchas/hybrid-attn-ssm-batch-regime

draftlow · 0.60

llama-server's prompt-cache save path aborts the whole process when the RPC backend is active, and it presents as a random crash under load

[major] frameworkWith `--rpc` set, the server starts fine, answers a short first request, then aborts the entire process on the first request large enough to make the slot's prompt eligible for prompt-cache save. The stack goes through `rpc_buffer_get_tensor`, which hits an abort inside the RPC b…gotchas/comm-rpc-backend-prompt-cache-abort

draftmedium · 0.65

llama.cpp on the Vulkan backend beats HIP by ~20% on gfx906 at fa=0 and short context, and the reported cause is AMD's aggressive Vulkan power management idling cards between layers in a layer-split

[minor] driverOne member builds llama.cpp from ROCm 7.14 and separately from the Vulkan backend and reports: 'the PP and TG is roughly 20 pc faster than rocm 7.14 when fa=0. note that my card is capped at 175 w. but rocm still performs better with -fa 1 when context is longer.' The Vulkan devi…gotchas/comm-gfx906-vulkan-20pct-over-hip-short-ctx

draftlow · 0.35

MoE all-to-all communication is the serving bottleneck

[major] hardwareThe all-to-all communication in MoE expert parallelism is the serving bottleneck, not the expert GEMM. At decode batch B, the dispatch volume per step per layer is B*k*d*2 bytes outbound and the same inbound on combine. This sits on the interconnect, not HBM, so it shows up as la…gotchas/moe-alltoall-serving-bottleneck

draftmedium · 0.70

Practitioners' engine choice tracks the hardware axis, not the feature axis: llama.cpp by default, vLLM when MoE tensor parallelism or throughput matters, TensorRT-LLM only where its FP8 block-scaled kernels actually exist

[minor] frameworkA practitioner running dual RTX 3090s states the split plainly: 'My daily LLM inference server is llama.cpp, which is far more user friendly and flexible than vLLM, however vLLM's performance is usually superior as such for especially heavy or performance sensitive workloads I so…gotchas/comm-engine-choice-consumer-vs-datacenter-practice

draftlow · 0.50

SSM state cannot be paged or evicted like KV cache

[major] configAn SSM layer's recurrent state cannot be paged, evicted, or compressed like KV cache. It is a fixed-size per-sequence tensor that must be resident in HBM for the entire decode. Standard KV cache management (paged attention, eviction, prefix caching) does not apply to SSM layers.gotchas/ssm-state-no-paging-eviction

draftmedium · 0.70

TurboQuant KV cache (k8v4) is unusable on Ampere consumer/prosumer cards with hybrid MoE models: a Triton FP8 path reports supported below SM 89, gets selected, then fails at runtime

[major] driverA user on 2x NVIDIA RTX A5000 24GB (SM86) running Qwen/Qwen3-35B-A3B-FP8 with kv_cache_dtype=turboquant_k8v4 and max_model_len=163840 hit a chain of failures, starting with the model refusing to start at all: 'NotImplementedError: TurboQuant KV cache is not supported for hybrid (…gotchas/comm-turboquant-kv-ampere-arch-gate

draftlow · 0.60

Vision encoder not quantized in mixed-precision serving

[minor] configThe vision encoder runs in BF16 even when the LLM decoder is quantized to FP8 or INT8, leading to a precision mismatch and wasted memory.gotchas/vision-encoder-not-quantized-in-mixed-precision-serving

draftlow · 0.50

Vision encoder re-encoding waste in multi-turn chat

[major] configIn multi-turn image/video chat, the vision encoder re-encodes the same image every turn, wasting compute. For a 448x448 image with ViT-L/16, this is ~763 GFLOPs per turn. Over 10 turns, that's 7.6 TFLOPs of wasted compute.gotchas/vision-encoder-re-encoding-waste-in-multi-turn-chat

draftlow · 0.50

Vision encoder runs on separate schedule from decoder

[major] configThe vision encoder and LLM decoder run on separate schedules, leading to underutilization. While the encoder is running, the decoder is idle, and vice versa.gotchas/vision-encoder-runs-on-separate-schedule-from-decoder

draftlow · 0.50

'Up to 90-91% off' is a ceiling across all SKUs, the published grid is a regional floor, and spot is explicitly excluded from committed-use discounts - so no blended spot rate exists

[major] configYou cost an inference deployment at the advertised spot discount and get a number 40-80% below on-demand, then discover three separate reasons the number is wrong. (1) The discount is a CEILING over all EC2 SKUs, not an H100 figure - nobody has paid 90% off for p5, and the concre…gotchas/rent-spot-discount-is-not-a-budget-number

verifiedhigh · 0.88

'vLLM falls back to emulation' is wrong for MXFP4, MXFP8 and NVFP4 on NVIDIA — the weight-only Marlin kernel comes first in the registry and wins by default

[major] kernelAn MXFP4, MXFP8 or NVFP4 checkpoint loads on Turing through Blackwell and runs, with the expected memory saving, at a fraction of the expected compute throughput. Nothing errors. The only clue is a logger.warning_once line that is easy to miss, or the 'MarlinMxFp4LinearKernel is …gotchas/mx-nvfp4-default-fallback-is-marlin-not-emulation

verifiedhigh · 0.90

"Nobody publishes this price" is usually "no OEM publishes this price" - and that distinction hides a closable gap

[blocker] measurementA whole hardware generation is recorded as unpriceable, and the finding is repeated until it becomes load-bearing. docs/08-cost-per-token.md states 'Every AMD MI300X/MI325X/MI355X benchmark is unpriceable' with the reason 'no supply record prices AMD Instinct per GPU-hour', names…gotchas/w5s-a-negative-about-one-channel-is-not-a-negative-about-the-mar

verifiedhigh · 0.90

"Nobody publishes this price" is usually "no OEM publishes this price" - and that distinction hides a closable gap

verifiedhigh · 0.90

/health returns 200 before the model is loaded, so a liveness-style readiness probe routes traffic to a replica that will fail every request for minutes

[blocker] operationsAfter a rollout, a fraction of requests fail with connection errors or immediate 500s against newly-created pods that Kubernetes reports as Ready. The failure window is the model load time and it varies per pod, so the error rate looks stochastic. Meanwhile the pods are Ready by …gotchas/ops-health-endpoint-lies-about-readiness

verifiedhigh · 0.90

128K context capacity numbers are quoted without concurrency, and one 8-GPU node holds single-digit 128K sequences for most production models

[major] hardwareA capacity plan reads '128K context, 8x H100' and concludes a healthy batch size. The arithmetic disagrees: on an 8xH100 80GB node with bf16 KV, Llama-3.1-70B at TP4 holds 3.8 concurrent 128K sequences, Qwen2.5-72B 3.7, Mistral-Large-2411 1.2, Command-R-plus-0824 2.7. With fp8 KV…gotchas/ctx-gotcha-128k-concurrency-is-single-digit

verifiedmedium · 0.80

16 papers in this repo were live under two slugs each, with contradictory venues - the affiliation field is what exposed them

[blocker] toolchainSixteen papers each had TWO live records under different slugs, so every venue-, category- and adoption-filtered query returned some papers twice. Six of the sixteen carried contradictory venues: s4 held iclr while s4-structured-state-spaces held iclr plus a notes sentence claimi…gotchas/w6p-one-paper-two-slugs-sixteen-times-and-the-field-that-exposed

verifiedhigh · 0.95

16 papers in this repo were live under two slugs each, with contradictory venues - the affiliation field is what exposed them

verifiedhigh · 0.95

2026 DRAM supply crisis inflates big-memory CPU builds

[major] operationsA 384GB DDR5 RDIMM kit now costs $4,000-5,200+ alone - more than four used RTX 3090s - pushing a single-socket Genoa build to ~$7,000-8,000.gotchas/gotcha-2026-dram-supply-crisis-cpu-build-cost

verifiedmedium · 0.80

35 arXiv preprints are filed kind:blog at confidence 0.5 in data/sources while 392 arXiv records are filed kind:paper at 0.9+ - the split tracks whether anyone read past the abstract, not the source's reliability

[major] measurementdata/sources holds 35 records whose URL is an arxiv.org/abs page and whose confidence is 0.5, ALL of them with kind='blog' - the pap-eff-* and pap-* family, plus blip-2, flamingo and navit. The same directory holds 392 records whose URL is an arxiv.org page sitting at 0.9-0.95, 2…gotchas/w10-thirty-five-arxiv-preprints-are-kind-blog-at-0-5-while-392-a

verifiedhigh · 0.90

A 4.3x per-GPU gap between two Llama 2 70B Offline records is arithmetically consistent, not a transcription error - and it is the gap a cross-record outlier sweep is most likely to report as a finding

[minor] measurementSweep data/benchmarks/ for records that disagree with their own siblings and one group fires immediately: gaudi2-mlperf-v4-0-llama2-70b-offline-tokens at 8034.85 and bench-mlperf-v5-1-cisco-h200-tgp700w-llama2-70b-offline at 34548.2. Both are Llama 2 70B, both fp8-e4m3, both MLPe…gotchas/bench-cross-version-mlperf-looks-like-an-outlier-under-every-nor

verifiedhigh · 0.85

A benchmark record whose value is a speedup ratio but whose metric says decode_tok_s is selected by every filter that matters and read as a rate

[major] measurementYou pick hardware by filtering data/benchmarks/ on metric: decode_tok_s and taking the largest value, because that is the field a capacity planner reaches for. Three records in that slice are not throughputs at all. sd-medusa-batch32-degradation carries value 0 with unit speedup_…gotchas/bench-metric-enum-hides-a-dimensionless-ratio

verifiedhigh · 0.85

A benchmark shorter than the queue ramp measures the idle-to-loaded transition, not the steady state - the MLPerf traces show ~15 minutes of sub-3 kW ramp before the plateau

[major] measurementTwo runs of the identical configuration on the identical node report different tokens/s, and the shorter run reports the HIGHER number. Mean power and joules per token come out too low in the same runs. There is no error, no warning, and the LoadGen-style harness declares the run…gotchas/pwr-short-run-measures-ramp-not-steady-state

verifiedhigh · 0.90

A block format's declared block size is not its scale group, and a library default is not a format constant - reading either wrong silently corrupts the memory arithmetic

[major] formatTwo wrong-but-plausible weight_group_size values, both of which change every memory and quality number computed from them. (1) SEVEN GGUF records carried 256, which is the K-quant SUPER-BLOCK (QK_K) rather than the group over which one scale is shared: the true scale groups are 1…gotchas/qzgw-block-size-is-not-the-super-block-and-a-default-is-not-a-co

verifiedhigh · 0.90

A block format's declared block size is not its scale group, and a library default is not a format constant - reading either wrong silently corrupts the memory arithmetic

verifiedhigh · 0.90

A cache hit produces different output than a cold run, with no error anywhere in the stack

[blocker] frameworkThe disaggregated deployment looks completely healthy and still returns wrong text. Both vLLM servers start, return HTTP 200, stay healthy after the requests, report that LMCache P/D mode is enabled, report the NIXL/UCX transfer channel active, and report COMPLETE prompt KV-cache…gotchas/pd-cache-hit-silently-differs-from-cold-run

verifiedhigh · 0.90

A Chinese vendor's own site has no spec table, an export-controlled part has no public datasheet, an industry survey is members-only, and a hosted model has no config.json - four different vendor silences that produce the same 0.5 and only one of them is fixable

[major] measurementFive source records carry confidence 0.3-0.5 and every one of them is at that number because the first-party document does not exist in public. cn-cambricon-mlu590 sits at 0.3, the lowest non-probe value in data/sources, because as its own note states 'Cambricon has NOT published…gotchas/w10-vendors-who-publish-nothing-export-control-paywall-and-aggre

verifiedhigh · 0.90

A citation goes dead-looking three different ways - deleted file, moved file, and wrong-repo file - and a single probe of a single hypothesis cannot tell them apart

[major] complianceA source record's URL stops resolving and gets written off as DEAD, when the document is alive at a different address. Three distinct mechanisms produce the identical symptom, and the fix differs in each case. (1) TRULY DELETED: the Pallas README was removed from jax-ml/jax main …gotchas/w6s-citation-rot-three-instrument-false-deaths

verifiedhigh · 0.95

A closed-loop autoscaler on scraped telemetry overshoots because the signal is stale by the time the decision is made — several replicas are added for a burst that one replica would have absorbed

[major] operationsA traffic burst triggers a scale-up to several replicas, then a scale-down, then another scale-up — the classic flapping pattern — while the burst itself was short enough that the original replica count could have served it. Prometheus shows the gauge rising and falling well afte…gotchas/ops-autoscaler-telemetry-lag-thundering-herd

verifiedhigh · 0.85

A defect one agent cannot fix is not a finding - it is a debt with an owner, and without a ledger it is re-discovered under a new name every wave

[minor] operationsThe same defect is reported by three different agents in three different waves, each with a different name and each believed to be a fresh discovery. Measured in this repo across waves 4, 5 and 6: docs/08's AMD coverage claim, the AMD cost gap, the tools/index.py bare-id emitter,…gotchas/w7l-a-defect-out-of-scope-gets-renamed-every-wave

verifiedhigh · 0.90

A derived number can be written into kv_cache_bytes_per_token with its derivation expression written correctly beside it in the notes, and the two still disagree because the expression was never actually evaluated - the field is not self-checking

[major] configFour of the eight Whisper records overstated kv_cache_bytes_per_token: whisper-tiny 30720 vs 6144 (5.00x), whisper-base 40960 vs 12288 (3.33x), whisper-small 81920 vs 36864 (2.22x), whisper-medium 131072 vs 98304 (1.33x). The four large/turbo records were correct. The tell that i…gotchas/mdl-gotcha-derived-figures-field-disagrees-with-its-own-note

verifiedhigh · 0.92

A DGX H100/H200 node continues to run at REDUCED performance with half its PSUs depopulated - the degradation is silent, not an alert

[major] hardwareAfter a data centre power-distribution or PSU failure takes three of the six PSUs offline, the system 'continues to function, but at a reduced performance level'. There is no crash, no error and no degraded-state alarm in normal monitoring: the node serves traffic, passes health …gotchas/dgx-psu-depop-silent-degraded-throughput

verifiedhigh · 0.92

A dropped KV-transfer completion notification hangs the receiver forever at 100% CPU instead of erroring

[blocker] frameworkThe decode (GEN) worker goes silent after model init with no error and no crash. The stack sits in check_gen_transfer_status, the GPU is idle, and the process spins at 100% CPU. Reproduced 100% of the time on one rack, worked around by switching that rack to the UCX backend, then…gotchas/pd-completion-notification-drop-hangs-receiver

verifiedhigh · 0.90

A failed RDMA KV transfer is reported as success: HTTP 200, valid envelope, garbage tokens

[blocker] hardwareOn a disaggregated vLLM deployment over InfiniBand, the served model returns garbage tokens with no error at all: 'The request returns HTTP 200 with a well-formed OpenAI response envelope, populated usage, and a finish_reason. No error is returned to the client.' Two consecutive …gotchas/pd-rdma-transfer-failure-returns-http-200-garbage

verifiedhigh · 0.90

A fleet at low utilisation can cost MORE in power than the same fleet at high utilisation - the fixed power floor makes naive cost-per-token arithmetic wrong

[major] measurementEnergy per unit of work is U-shaped in the operating point rather than monotonic, so the cheapest-to-run configuration is not the one that runs hardest. Measured: continuous batching improves system-level energy efficiency 3.2x from 1 to 16 requests per second because GPU power P…gotchas/idle-and-underbatched-fleet-can-cost-more-than-busy-fleet

verifiedhigh · 0.85

A flops[] row could carry a number with no derivation: the schema gap that let eight generated values survive review

[blocker] measurementThe accelerator schema's flops[] items required exactly two fields - precision and tflops - and permitted five optional ones (dense, vendor_claim, source_id, notes, unit). A row could therefore state a TFLOPS figure, set vendor_claim:false, leave source_id null, and carry its ent…gotchas/flops-row-had-no-basis-field

verifiedhigh · 0.92

A gateway hop costs one to two orders of magnitude less latency than the model call it fronts, but it adds a failure domain, a compatibility ceiling and a throughput ceiling - the overhead number is the least of the three

[minor] configThe question 'how much latency does the gateway add' is asked with the expectation of a bad number, and the measured answers are small: roughly 2 ms of data-plane overhead under sustained enterprise LLM load (independent Broadcom/VMware validation, ~0.01% of end-to-end latency, f…gotchas/dep-gateway-hop-cost-and-value

verifiedhigh · 0.85

A GPU pod is bound to its device, so any reschedule onto another node loses the weights, the KV cache, the compiled graphs and the page cache together - and re-warming costs the full measured cold start on a node that has never run the image

[major] hardwareAn LLM serving replica is not a stateless app, so the ordinary Kubernetes failure response - delete the pod, let the scheduler place a new one - is a multi-minute cold start, and there is no scheduling trick that avoids it. Three properties compound. The pod is bound to a specifi…gotchas/dep-gpu-pod-rescheduling-destroys-all-warm-state

verifiedhigh · 0.90

A high KV cache usage percentage does not mean you are close to the wall, and a low one does not mean you have room - the same number means different things on different engines

[major] measurementThe classic 80% KV-usage alert fires constantly on a healthy server, or a server sits at 60% 'utilization' while rejecting requests. Two replicas of the same deployment on different backends need different thresholds for the same alert. A dashboard shows usage near 100% with plen…gotchas/obs-kv-cache-usage-percentage-is-not-a-headroom-signal

verifiedhigh · 0.88

A histogram whose top bucket sits below your real latency makes p99 report the ceiling, not the data - and it looks like a stable measurement

[major] measurementAn inter-token-latency p99 sits at exactly 2.0 seconds, day after day, and does not move. The dashboard looks healthy - a round, plausible number with no variance - while users report stutters of five and ten seconds. Every percentile below the ceiling varies normally, so the cha…gotchas/w5e-histogram-ceiling-pins-p99-at-the-ceiling

verifiedhigh · 0.93

A hosting platform is not a vendor - putting huggingface.co in the vendor registry inflates Tier A and reads as rigor

[None] measurementA provenance classifier that asks 'is this domain a vendor' and has huggingface.co in the vendor table returns Tier A (primary vendor document) for EVERY file on the platform - including unsloth's re-publications of Nous Research's and Meta's weights, and individual community fin…gotchas/w7r-the-tier-a-number-is-only-as-wide-as-the-hub-you-lookup

verifiedhigh · 0.95

A hybrid model's KV cost depends entirely on how many layers are full attention, and when config gives a GROUP SIZE rather than an index list, reading the group size as the layer count inflates the KV figure several-fold

[blocker] configling-3-0-flash recorded kv_cache_bytes_per_token 23040, overstating the real figure 8064 by 2.86x (20 attention layers used where 7 are real). The record had attention_layer_indices and layer_types left null, which is the visible symptom: an author could not find the layer list i…gotchas/mdl-gotcha-hybrid-attention-layer-count-from-a-rule-not-the-name

verifiedhigh · 0.93

A JIT cache keyed on more than the kernel shape recompiles inside a single process: FlashInfer's micro-kernel cache key includes batch size AND workspace size, so cache growth invalidates entries it already paid for

[major] configWithin one running server process, a kernel that compiled successfully at startup recompiles later, several times, as real traffic arrives. FlashInfer issue #4317 documents the mechanism in a single sentence: the micro-kernel 'cache_key includes both m and workspace.max_rows (0.6…gotchas/ev-jit-cache-key-invalidation

verifiedhigh · 0.85

A kernel JIT compile that runs on a request path stops the world: FlashInfer's own issue documents ~17.5 s stop-the-world stalls with every in-flight request frozen, including mid-decode

[major] kernelA serving engine that has been warm for days suddenly freezes for 15-40 seconds in the middle of traffic. FlashInfer issue #4317 measured the mechanism rather than inferring it: the Blackwell sm12x fused-MoE path compiles all of its CuTe-DSL kernels through explicit cute.compile(…gotchas/ev-jit-compile-freezes-the-engine-mid-request

verifiedhigh · 0.85

A llama.cpp ROCm commit raised WikiText-2 perplexity from 7.72 to 3024 on a Q4_0 model with no error, and only llama-per perplexity caught it

[major] measurementOn Ryzen AI Max+ 395 (iGPU Radeon 8060S, gfx1151), ROCm 7.2.4, Llama-3.2-3B-Instruct Q4_0, wikitext-2-raw/wiki.test.raw, -c 32768 -ngl 999 --chunks 1: llama-perplexity returns PPL = 7.7241 at b10038 and 5839ba35, then PPL = 3024.3587 at b10040, and 2177.3497 on latest master bb4c…gotchas/llamacpp-rocm-ppl-explosion-silent-corrupt

verifiedhigh · 0.90

A mixed-arch AMD GPU box initialises its RCCL communicator fine over P2P and then aborts on the first collective with 'invalid device function' on the older card

[major] driverRCCL 2.30.4 in a process spanning one gfx1201 and one gfx1200 device: communicator init succeeds - channels connect via 'P2P/direct' - and then the first ncclAllReduce raises '[FATAL ERROR]: HIP failure: \'invalid device function\'' on the gfx1200 device. The identical binary pas…gotchas/mgpu-rccl-heterogeneous-amd-gpu-archs-fail-first-collective

verifiedhigh · 0.85

A model's advertised context window is a license, not a capability: the gap runs 2x to 256x and nobody publishes the ratio

[blocker] measurementA model loads, accepts a long prompt, and returns fluent text that is wrong. No error, no truncation warning, no OOM - just a confidently incorrect answer about something that was in the prompt. The specific figures: Yi-34B advertises 200K and its RULER aggregate effective length…gotchas/ctx-gotcha-claimed-vs-effective-context-gap

verifiedhigh · 0.88

A multi-column vendor spec table lets a record silently inherit every number from the wrong product column, and the error survives review because one field in each group is right

[major] hardwareA record named for one product carries another product's memory capacity, bandwidth and FLOPS from the same datasheet, while its power figure is correct - so no internal cross-check flags it. Concretely: nvidia-b200 stored 186 GB / 8 TB/s / fp32 80 / tf32 1250 / fp4 10000, every …gotchas/aa-gotcha-spec-table-column-mixing

verifiedhigh · 0.90

A multi-tier KV existence query can report a prefix as available when no tier can actually serve it

[major] frameworkHiCacheFile.batch_exists_v2() reports a hybrid prefix as a cache hit even when a required auxiliary pool cannot restore that prefix. With SWA valid endpoints {3} and Mamba valid endpoints {2}, the reported shared endpoint is 2 where the expected value is 0 (no common endpoint). W…gotchas/pd-kv-tier-existence-query-over-reports-prefix

verifiedhigh · 0.85

A paginated price API that ignores its own filter will hand you a global sample and let you invent a price error

[blocker] measurementA price record is re-verified against what appears to be a targeted query, the queried meter is absent from the returned rows, and the record is about to be 'corrected' upward. The correction is wrong. On 2026-10-05 an intermediate probe of prices.azure.com/api/retail/prices, fil…gotchas/w5s-api-page1-is-not-the-catalogue

verifiedhigh · 0.95

A part named only in prose has no record, and no check in the repo can see it

[major] measurementFour parts that the corpus talks about constantly had no accelerator record at all, and every mention of them lived in free text: Cerebras CS-4 / WSE-3 Turbo (named in five records, including by the CS-3 record itself, which says 'A separate record for CS-4 / WSE-3 Turbo should b…gotchas/w7z-a-part-named-in-prose-has-no-record

verifiedhigh · 0.90

A PCIe-only multi-GPU box serving a large MoE saturates interconnect rather than compute, so adding GPUs or SMs does not scale and utilization can look healthy while throughput is capped

[major] measurementMeasured on 8x RTX PRO 6000 Blackwell Max-Q (SM120, 96 GB, PCIe-only, no NVLink) serving DeepSeek-V4.1-Flash: 128K cold prefill 6,853 tok/s (re-run 6,765, -1.3%), 32K prompt ~7,196 tok/s, at ~87% GPU utilization drawing 220 W of a 600 W cap - i.e. the box is communication-bound a…gotchas/mgpu-pcie-only-multigpu-is-communication-bound

verifiedhigh · 0.85

A peak-FLOPS specification filed under metric=tps_aggregate is returned verbatim by a capacity planner's tokens/s filter, as four numbers that are not tokens per second

[major] measurementFour records in data/benchmarks/ carry metric=tps_aggregate while their value is a peak COMPUTE figure, not a rate of tokens: alt-aws-inf2-fp16-tps is 2280 with unit 'TFLOPS FP16', alt-aws-trn2-fp8-tps is 20.8 with unit 'PFLOPS FP8', alt-tpu-v5e-bf16-tps is 197 with unit 'TFLOPS …gotchas/bench-tflops-filed-as-tps-aggregate-survives-the-tok-s-filter

verifiedhigh · 0.90

A per-GPU-hour rate is not a comparable number until you also know which instance shape it came from - the same GPU differs 2x per-GPU between an 8-GPU and a 1-GPU shape

[major] measurementYou collect per-GPU-hour figures from several providers to compare cost, divide each node price by its GPU count, and produce a clean table. The table looks fine and it is wrong in a specific way: AWS charges $6.88 per H100-hour for p5.48xlarge (8x H100, $55.04/instance-hr) AND e…gotchas/rent-per-gpu-hour-needs-the-shape-too

verifiedhigh · 0.90

A persistent per-pod /dev/shm converts one scheduler crash into a permanent crashloop, because a container restart does not reclaim leaked segments

[blocker] frameworkMeasured fleet-wide on ~700 H20 GPUs with TP4 pods: a single traffic burst produced scheduler crashes, and affected pods went from 4 to 101 in about 30 minutes with none recovering without pod deletion, each cycle producing a core dump. Two verbatim startup failure modes after th…gotchas/mgpu-devshm-leak-wedges-pod-permanently

verifiedhigh · 0.92

A plain HTTP probe from this host gets 403 from a commercial vendor and looks like a dead source - five sources in data/sources were rated 0.5 for exactly that, and the correct rating is KNOWN_LIMITATION not UNCHECKED

[major] complianceFive source records sit at confidence 0.5 because a plain HTTP GET from an unauthenticated datacenter IP returns HTTP 403 'Access Denied'. They are sup-buy-intel-gaudi (intel.com), sup-buy-supermicro-gpu-servers and sup-buy-supermicro-sys-521ge (supermicro.com), sup-buy-nvidia-ma…gotchas/w10-antibot-403-is-not-a-dead-link-five-sources-block-a-scripted

verifiedhigh · 0.90

A pre-set PROMETHEUS_MULTIPROC_DIR makes vLLM's metrics wrong across restarts — and autoscaling reads them

[major] operationsAfter a pod restart the vLLM gauges that should reset to zero instead carry the previous process's values: vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc all read high on a pod that has served nothing. Any autoscaler or alert reading them acts o…gotchas/ops-stale-multiprocess-metrics-dir

verifiedhigh · 0.90

A record can cite a source_id that exists but is absent from its own sources[], and no check in the repo can see it

[major] toolchainAGENTS.md rule 2 says every claim must have at least one sources entry and every entry must resolve, and tools/validate.py implements exactly that: it resolves each sources[] entry against data/sources/ and errors on a dangling one. The invariant is one-directional. A record whos…gotchas/w10r-cited-source-id-is-not-a-declared-source

verifiedhigh · 0.90

A record whose bandwidth_basis correctly says the figure is bidirectional and must be halved, while its typed bandwidth field stores the unhalved aggregate - the prose was right and the number was 2x wrong, for two months

[major] hardwareThe nvswitch-1 record stated in its own bandwidth_basis that 'the recorded per-direction value is 150 GB/s per GPU' and simultaneously stored bandwidth_gbps 300. Two fields of one record, one prose sentence, disagreement of exactly 2x, and the typed field is the one every downstr…gotchas/w7n-nvswitch-1-stored-the-bidirectional-aggregate-as-per-directi

verifiedhigh · 0.92

A right-to-erasure request has to reach logs, traces and caches, not just the primary store - and 164.316 wants the audit documentation kept for 6 years

[major] complianceA deletion request is honoured in the application database. The same prompt is still in the log aggregator, in a trace span, in object storage, and in a multi-turn conversation's cached prefix, and the processor cannot demonstrate erasure across all of them.gotchas/sec-right-to-erasure-has-to-reach-derived-artefacts

verifiedhigh · 0.85

A router scrape that recognizes zero metric names returns an all-zero stats object that is truthy, so load-aware routing silently degrades to a random pick with no warning anywhere

[major] frameworkThe deployment looks correct from every signal it exposes. Every backend reports healthy, get_health() stays true, the router logs a normal scrape, dashboard gauges render, and traffic keeps flowing. But load-aware placement is not working: the router is picking roughly at random…gotchas/dep-router-metric-name-parse-miss-looks-healthy

verifiedhigh · 0.92

A separate guard model is a second LLM in the request path, and its cost is bounded below by seconds - not by the size of the flag you set

[major] securityExpectation is that 'add Llama Guard' is a policy toggle. Reality is a model forward pass on every request, and even the smallest published guard (Llama Guard 3-1B-INT4, 440 MB, about 7x smaller than the fp 1B) reports TTFT of 2.5 s or less and at least 30 tok/s on a commodity An…gotchas/sec-guard-model-is-a-second-llm-in-the-path

verifiedhigh · 0.85

A short benchmark is not representative of a 24-hour serving run: the same probe spans ~4.8x depending on thermal/clock state, and GPU-only telemetry hides 41-45% of system energy

[major] measurementTwo failures compound. First, short probes measure a cold or parked machine, not the steady state a 24-hour serving run reaches - the same bf16 canary read 25.0 TFLOPS idle and 119.3 TFLOPS after a heavy load on one machine, and an 8-minute sustained workload ran at full speed im…gotchas/short-benchmark-not-representative-of-sustained-serving

verifiedhigh · 0.88

A single llama.cpp commit regressed Vulkan Q4_K_M decode speed with no prompt-processing change

[minor] kernelBisected to first bad commit adc5dd92e8aea98f5e7ac84f6e1bc15de35130b5 on Windows 11 with dual Radeon PRO W7800, Vulkan SDK 1.3.283, Qwen2.5-14B-Instruct-Q4_K_M at -ngl 99. Decode (tg) fell 42.12 -> 36.38 tok/s (23.74 -> 27.49 ms/token) while prompt processing and sampling were fl…gotchas/vulkan-q4km-speed-regression

verifiedmedium · 0.80

A source citation pointing at a file that upstream moved looks first-party and cannot be followed - and the reported move is often not a move at all

[minor] toolchainA source record cites a specific file inside an upstream repo - the natural way to cite code, because a permalink to a source file is the strongest evidence available. Upstream refactors, the file moves, and the citation silently becomes unfollowable. There is no error anywhere: …gotchas/w6s-stale-repo-path-citation-looks-first-party

verifiedhigh · 0.90

A tensor-parallel size that does not divide the model's attention-head count fails at startup, and the message does not tell you what to use instead

[blocker] configServing a model with --tensor-parallel-size N raises at config validation. The vLLM string is: 'Total number of attention heads (40) must be divisible by tensor parallel size (6)' (reported verbatim on Qwen-1.5-32B with TP=6), and in current form: 'Value error, Total number of at…gotchas/mgpu-tp-size-must-divide-attention-heads

verifiedhigh · 0.90

A throughput number without batch size, concurrency, input/output length, and warmup is meaningless

[major] measurementA vendor claims '1000 tokens/sec' but doesn't specify whether that's at batch 1 or batch 128, with 128-token or 8192-token inputs, with or without warmup. The actual serving throughput at your workload could be 10x lower.gotchas/meth-throughput-number-omits-load-bearing-details

verifiedhigh · 0.85

A tok/s figure obtained by dividing a spec sheet's memory bandwidth by a guessed bytes-per-token is not a measurement, and it passes the load-condition rule because it inherits the right condition words

[major] measurementA record states a clean decode figure - 25 tok/s on an 8GB card, 120 tok/s on an RTX 5070, 22 tok/s on an M5 Max - and states its load conditions too: batch size 1, 512-token prompt, 128-token output, a named precision, a named backend. It looks like the model answer to 'does thi…gotchas/bench-tok-s-derived-from-memory-bandwidth-is-not-a-benchmark

verifiedhigh · 0.90

A translating gateway drops output-affecting request fields and returns 200: a pinned JSON-schema const arrives as type:object, stop sequences never arrive at all, and nothing in any log says so

[major] frameworkThe request succeeds, the response looks right, and the constraint you set is not in force. Six concrete cases reported against LiteLLM v1.101.0 against real provider APIs: a JSON-schema const is forwarded to Gemini as {'type':'object'}, so any object satisfies it and a pinned va…gotchas/dep-gateway-silently-drops-params-with-http-200

verifiedhigh · 0.92

A vendor 'PCIe peak bandwidth' number may be per-direction or bidirectional aggregate, and the two differ by exactly 2x, so a spec that does not state which one silently halves or doubles every host-bandwidth number

[major] hardwareTwo failure directions, both already seen in this repo. (a) DOUBLE-HALVING: a correct aggregate figure is halved because the reader assumes the sheet quotes per-direction, or is halved twice - once for a 'sparse' reading it does not need and once for a 'per-direction' reading it …gotchas/aa-gotcha-pcie-peak-bandwidth-direction-convention

verifiedhigh · 0.85

A vLLM SIGTERM aborts every in-flight request by default — a rolling update fails requests that were seconds from completing, with no error unless you set --shutdown-timeout

[blocker] operationsEvery rolling update or scale-down fails a batch of in-flight requests. The failures land as error-status responses to clients mid-generation, typically on the pods being replaced, and cluster-level error rates spike in proportion to in-flight concurrency at the moment of termina…gotchas/ops-restart-aborts-in-flight-by-default

verifiedhigh · 0.95

A100 hardware relevance cited without memory capacity (Defect D6)

[minor] hardwareThirteen records in data/papers/ cite hardware_relevance 'nvidia-a100-sxm', generating 13 dangling reference warnings under tools/validate.py against the budgeted debt.gotchas/gap-a100-no-capacity-hardware-relevance

verifiedhigh · 0.95

Accelerator FLOPS rows generated by repeatedly doubling a base figure instead of being read off the spec table produce a whole internally-plausible record that is wrong at every row

[major] hardwareA record whose flops array is a clean geometric ladder - fp32 29.4, fp16 58.7 (2x), fp8 117.4 (4x), fp4 234.8 (8x) - which is exactly the shape NVIDIA's real tables have, so it passes a 'does this look sane' review, while every value is wrong. nvidia-rtx-5070 stored 7 rows, all 7…gotchas/aa-gotcha-accelerator-flops-derived-by-doubling-not-read

verifiedhigh · 0.90

Across the non-NVIDIA, non-AMD scale-up fabrics audited together, every vendor who states a direction states an AGGREGATE one, the per-link rate is usually never published at all, and three different null bandwidths mean three different things

[major] hardwareAuditing every non-NVIDIA, non-AMD scale-up fabric in this repo at once (2026-10-05) surfaces three traps that a record-by-record reading does not, because each fabric hides exactly one of them. TRAP 1 - THE CORPUS-WIDE CONVENTION IS AGGREGATE, AND EVERY VENDOR WHO SAYS SO SAYS …gotchas/w7f-non-nv-am-vendor-conventions

verifiedhigh · 0.85

Affiliation fields filled without a byline invent plausible universities - Rice University appears in two records where no author is there

[blocker] measurementTwo paper records in this repo credit institutions that appear nowhere on the papers. distflashattn (arXiv 2310.03294) listed 'Lehigh University', 'Rice University' and 'Texas A&M University' - the paper's byline lists UC Berkeley, University of Washington, UCSD, CMU and USC, so …gotchas/w6p-affiliation-fields-invented-from-plausibility-rice-universit

verifiedhigh · 0.92

Affiliation fields filled without a byline invent plausible universities - Rice University appears in two records where no author is there

verifiedhigh · 0.92

AMD 'Infinity Fabric bandwidth' means three different numbers depending on which table row you read (per-link scale-up vs GPU-to-GPU total vs host/IO path)

[major] hardwareA part's 'Infinity Fabric bandwidth' is quoted as a single number and used to size the wrong path. The concrete case already seen in this repo: the MI355X record's '7 IF links @ 153 GB/s' was at risk of being read as a host or total figure when it is INFINITY FABRIC PER-LINK SCAL…gotchas/w7a-amd-infinity-fabric-scale-up-vs-host-io

verifiedhigh · 0.92

AMD MI100 multi-GPU segfaults on bare-metal ROCm - needs PCIe-passthrough VM + HSA_AMD_SVM

[major] driverAdding a second (or more) MI100 card causes segfaults and instability on bare-metal Linux with standard ROCm - the canonical llama.cpp report (#2824) could only get 2x MI100 working by passing the cards through to a VM. Single-card works; multi-card is the landmine.gotchas/gotcha-mi100-multigpu-rocm-segfault

verifiedmedium · 0.80

AMD Ryzen Strix Point (gfx1150) APU accelerator gap distinct from Strix Halo (gfx1151)

[minor] hardwaredata/flops/embedding.json cites affected_by_hardware 'amd-ryzen-strix-point', which has no matching accelerator record in data/accelerators/. This reference is flagged by tools/validate.py as a dangling reference warning (budgeted debt).gotchas/gap-amd-ryzen-strix-point

verifiedhigh · 0.95

AMD Strix Halo (gfx1151) requires ttm.pages_limit boot parameter to expose >100GB unified memory to ROCm, avoiding false OOM errors

[blocker] configOn AMD Strix Halo APUs (Ryzen AI Max+ 395 with Radeon 8060S, 128 GB LPDDR5X), attempting to allocate large models in llama.cpp with ROCm fails during startup with 'ROCm error: out of memory' or 'hipStreamCreateWithFlags', even when the system has over 100 GB of free RAM. Converse…gotchas/amd-strix-halo-ttm-pages-limit-unified-memory

verifiedhigh · 0.90

AMD Strix Halo is an MoE part, not a dense-70B part - the ~256 GB/s wall caps dense decode at ~5 tok/s

[major] hardwareA Strix Halo 128GB box bought to run a dense 70B model delivers only ~5 tok/s - far slower than the marketing 'runs 70B/120B' implies. The same box runs gpt-oss-120b MoE at ~34 tok/s, which is the number that gets quoted.gotchas/gotcha-strix-halo-moe-not-dense

verifiedhigh · 0.85

An ACL ARR submission record is not a negative venue finding - the paper can still be published, and was (TurboRAG -> EMNLP 2025)

[major] toolchainVerifying a paper's venue by searching OpenReview finds 'ACL ARR 2025 May Submission' and, correctly, refuses it: an ARR submission is a REVIEW BATCH, not a publication, so a paper can sit in a batch indefinitely without ever appearing in a proceedings volume. Recording that as '…gotchas/w6p-negative-acl-arr-submission-is-not-a-negative-venue-finding

verifiedhigh · 0.93

An API key on an open-weights endpoint protects your serving capacity and quota - it never protected the model, and the weights are the thing worth stealing

[major] securityA team serves open weights behind vLLM with an API key and considers the model protected. An API key on your endpoint limits who can reach YOUR capacity. It does not limit what an attacker learns from the weights, which are downloadable from the model host by anyone with a Huggin…gotchas/sec-api-key-protects-capacity-not-the-weights

verifiedhigh · 0.85

An Azure Ultra managed disk at 28,000 GiB costs about $4.59/hour - 37% of the $12.29 H100-hour it is attached to, so a durable-disk inference node is never priced by its GPU rate

[major] configYou pick a GPU instance by its per-GPU-hour rate, ship to production, and the bill is 30-40% higher than the estimate with no obvious line item. On Azure ND96isr H100 v5 - 8x H100 at $12.29 per GPU-hour on-demand, and a 28,000 GiB temp storage field on the instance - the durable-…gotchas/rent-persistent-disk-can-cost-more-than-the-gpu-next-to-it

verifiedhigh · 0.85

An empty quantization.native_support means three different things - no silicon path, no hardware format at all, or unchecked - and the field cannot tell them apart

[minor] format72 of the 103 quantization records in this repo carry native_support: [], and the field is defined in SCHEMA.md as 'ids where the format has a hardware path'. A reader taking that definition literally concludes the same thing about all 72: that no accelerator in this repo runs th…gotchas/quant-native-support-empty-means-three-different-things

verifiedmedium · 0.80

An engine-core worker death loses every in-flight request — there is no retry, no replication and no request log; the client gets a bare error with no indication the work was partially done

[blocker] operationsA worker dies and every request it was serving fails at once, with no retry and no failover: the client receives a connection or 5xx error even though other healthy replicas exist, and the work already done (prefill, partial decode) is gone. Error-rate alerts show a sharp spike c…gotchas/ops-inflight-requests-are-lost-not-retried

verifiedhigh · 0.90

An engine-side KV layout change silently invalidates every external KV cache written by the previous layout

[blocker] frameworkAfter upgrading vLLM to 0.26.0 with LMCache 0.5.2, a request served from the LMCache CPU tier produces different output than the same request computed from scratch, with no exception and no warning - generations are simply wrong. The in-process connector silently returns wrong KV…gotchas/pd-packed-kv-layout-change-silently-breaks-external-caches

verifiedhigh · 0.90

An FP8 KV cache write path in Python is graph-unsafe and silently corrupts attention under CUDA graphs while --enforce-eager looks fine

[blocker] framework8x A800-80GB (SM80, no native FP8 tensor cores), TP=8, MiniMax-M3-AWQ-INT4 (weight-only int4), launched with --kv-cache-dtype fp8_e5m2 --max-model-len 1048576 --attention-backend TRITON_ATTN --block-size 128 --enable-chunked-prefill: the server boots, accepts the 1M-token window,…gotchas/fp8-kv-cache-cudagraph-python-scatter-corruption

verifiedhigh · 0.90

An idle 8xH200 node still draws 41.7% of its loaded watts, so per-token energy RISES as utilisation falls and naive cost-per-token arithmetic understates cost without bound

[major] measurementCost per token improves when the fleet gets BUSIER, which is backwards from the intuition that sparser workloads are cheaper per request. A utilisation-based cost model built as (power_at_full_load x hours x price) / tokens under-reports the cost of every low-traffic deployment, …gotchas/pwr-idle-node-draw-defeats-naive-cost-per-token

verifiedhigh · 0.85

An MLA model's config publishes four different head widths that disagree with each other, so any single head_dim field on such a record is a guess about which one is meant

[major] configglm-5-3 recorded head_dim 192, a value that appears nowhere in its config as a general head width and that corresponds to no single geometry in the stack. config publishes head_dim 64, qk_nope_head_dim 192, qk_rope_head_dim 64, qk_head_dim 256 and v_head_dim 256 simultaneously. T…gotchas/mdl-gotcha-mla-single-head-dim-field-hides-four-widths

verifiedhigh · 0.90

An SW power cap silently reduces clocks below requested with no error, so a capped fleet looks like a slow model

[major] configWith nvidia-smi --power-limit set below the card's Max TDP, the GPU runs below its requested clocks and every workload gets slower, with nothing in the application, the engine logs, or a benchmark run reporting an error. The card still passes ECC checks, still returns correct out…gotchas/sw-power-cap-silent-clock-reduction

verifiedhigh · 0.93

Apple Silicon has better bandwidth-per-watt than discrete GPUs, but lower absolute bandwidth

[minor] hardwareApple Silicon delivers more memory bandwidth per watt than discrete GPUs, but the absolute bandwidth is lower. A Mac can run a model longer on battery, but a discrete GPU will decode faster when both are plugged in.gotchas/local-bandwidth-per-watt-apple-vs-discrete

verifiedmedium · 0.80

Apple Silicon unified memory enables 70B+ model execution on consumer devices, but autoregressive decode speed is strictly bounded by LPDDR package bandwidth

[minor] hardwarePractitioners purchasing 64GB-128GB Macs frequently expect high decode throughput for 70B models based on the large unified pool, but observe single-user batch-1 decode speeds of only 8-15 tokens/second, which feels underwhelming compared to multi-GPU enterprise setups.gotchas/gotcha-apple-unified-memory-bandwidth-decode-ceiling

verifiedhigh · 0.95

Apple Silicon unified memory GPU allocations are capped at recommendedMaxWorkingSetSize (~75-85% of physical RAM), failing hard rather than paging cleanly

[major] hardwareInference jobs allocating near the physical memory capacity of Apple Silicon crash with Metal errors such as kIOGPUCommandBufferCallbackErrorOutOfMemory or libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insuff…gotchas/gotcha-apple-unified-memory-working-set-cap

verifiedhigh · 0.90

Applying the batch-1 decode roof to an aggregate record makes a compliant 800 tok/s result read as 7.2x impossible, because the roof that bounds one sequence does not bound sixty-four

[major] measurementRun a whole-corpus roofline sweep that ignores batch size and it will condemn the repo's own best-documented measurement. The MPT-7B trio on 1x A100 40GB: MPT-7B in FP16 is 14 GB of weights, and the A100 40GB SXM4 streams 1555 GB/s, so the batch-1 decode roof is 1555e9/14e9 = 111…gotchas/bench-batch-size-above-one-makes-the-batch-1-roofline-illegally

verifiedhigh · 0.90

Applying the batch-1 decode roof to an aggregate record makes a compliant 800 tok/s result read as 7.2x impossible, because the roof that bounds one sequence does not bound sixty-four

verifiedhigh · 0.90

Automatic prefix caching turns KV-cache reuse into an API-visible timing oracle, and prefix hit/miss alone can reconstruct another user's prompt

[blocker] privacyA cache hit is measurably faster than a cache miss. On a shared multi-tenant endpoint that latency difference is client-visible, and an attacker who controls their own queries can walk a victim's private prefix one token at a time by watching which of their own guesses hits. Conf…gotchas/sec-kv-cache-timing-side-channel-reconstructs-private-prefixes

verifiedhigh · 0.85

Average tokens/sec is not an SLO for LLM serving and will not detect the failure users actually report; the two that matter are TTFT and TPOT

[major] measurementA dashboard is green because output throughput meets target while users report the service 'feels stuck'. Raising concurrency to hit the throughput target makes TTFT worse without moving the throughput number off target in a way anyone notices. On the other side, a capacity revie…gotchas/obs-serving-slos-ttft-tpot-not-tok-s

verifiedhigh · 0.88

Bare SXM/OAM accelerator modules cannot run without a carrier/baseboard

[blocker] hardwareeBay 'cheap' V100/A100/GAUDI2 modules arrive with no PCIe edge connector and no cooler - they will not power on in a workstation.gotchas/gotcha-sxm-bare-module-not-usable-alone

verifiedmedium · 0.80

Batch-invariant mode is not part of any KV-offload namespace or connector compatibility check, so a numerics-mode mismatch across prefill and decode is undetectable

[major] configNo observed failure yet - this is the mechanism, documented from the code by the reporter, who is explicit: 'I have not reproduced a divergence.' The risk is that an instance running with VLLM_BATCH_INVARIANT=1 can load KV blocks that were produced under a different numerical con…gotchas/pd-batch-invariant-mode-absent-from-kv-namespace

verifiedmedium · 0.80

Benchmark contamination: if a model has seen the test set in training, the result is meaningless

[blocker] measurementA model reports a benchmark score that looks impressive but is actually measuring memorization, not capability. The model's perplexity on the test set is artificially low because it has seen the exact text during training.gotchas/meth-benchmark-contamination-invalidates-results

verifiedhigh · 0.85

Between two tenants sharing one vLLM process there is no isolation primitive at all - the granularity that exists is MIG, a container, or a process

[blocker] securityA deployment describes itself as isolated between tenants. It authenticates every request and scopes every KV block correctly, and both statements are true, and neither means tenants are isolated from each other: they share weights, one CUDA context, one KV pool, one scheduler an…gotchas/sec-tenant-isolation-granularity-is-process-or-mig-not-request

verifiedhigh · 0.85

Bi-encoder encodes corpus once; cross-encoder costs O(corpus) per query

[major] configReranking the full corpus with a cross-encoder is prohibitively expensive; the standard pattern is to retrieve 100 candidates with a cheap bi-encoder and rerank only those 100 with an expensive cross-encoder.gotchas/emb-bi-encoder-vs-cross-encoder-cost

verifiedhigh · 0.95

BlueField DPU offloads network processing but not GPU compute or NCCL collectives

[minor] hardwareBlueField DPU is sometimes described as offloading 'AI workloads' or 'collective communication', but it actually offloads network, storage, and security processing - not GPU compute or NCCL collectives.gotchas/nic-bluefield-dpu-offload-limits

verifiedhigh · 0.85

Cache-aware routing has no way to rebuild its map of which KV blocks live where: the placement view is built from a live event stream, so any router or replica restart leaves it permanently wrong, and the result is silently worse routing rather than an error

[minor] frameworkPrefix-cache hit rates drop after a restart and never fully recover, with no error and no unhealthy worker. This is the expected behaviour of the current design, not a bug report: SGLang's own RFC enumerates the cases where the router's placement view cannot be recovered. The rou…gotchas/dep-kv-aware-router-view-cannot-be-rebuilt-after-restart

verifiedhigh · 0.90

Canarying an inference service is harder than canarying a web service because a request's cost is not knowable in advance and the comparison is non-deterministic, so there is no clean 'new version is worse' signal

[major] operationsA canary at 5% traffic produces no usable verdict. Error-rate comparison is meaningless because both versions return 200; latency comparison is confounded because a different replica mix behind the canary has a different queue depth and therefore different latency for reasons unr…gotchas/ops-canary-on-inference-is-harder-than-web

verifiedhigh · 0.85

Cheap exotic accelerator surplus (Cambricon, Ascend, Coral, FPGA, Grayskull) has no load-a-GGUF-and-go LLM stack

[blocker] toolchainA tempting cheap used accelerator (Cambricon MLU270 ~$315, Ascend 310P ~$265, Coral ~$50-130, Alveo FPGA ~$200-2000, Grayskull e150 ~$300-500) cannot run an open-weight 7B+ model - there is no llama.cpp/vLLM/GGUF path. The silicon is real and the price is real, but the software i…gotchas/gotcha-exotic-silicon-dead-end-stack

verifiedhigh · 0.85

Cheap exotic silicon with no open LLM stack is a paperweight (Cambricon/Ascend/Iluvatar/Coral/Grayskull/FPGA)

[blocker] frameworkCambricon MLU270/370, Huawei Ascend, Iluvatar MR-V100, Google Coral, Tenstorrent Grayskull and FPGA cards appear cheap on eBay but have no llama.cpp/vLLM/GGUF path - they cannot run an open 7B model without writing your own kernels.gotchas/gotcha-closed-vendor-stack-paperweight

verifiedmedium · 0.80

Cheapest-per-GB VRAM is not value: two join artifacts produced a 60x and an 80x false ranking, and the real answer is that only five accelerators in this corpus can be ranked at all

[major] measurementAsked for the cheapest configuration that can genuinely serve VRAM, the corpus produced a table where a $310 supply record paired with a 96GB RTX PRO 6000 Blackwell and came out as the best value in the database at $5,636 per 1000 tok/s, and where the RTX 4090 came out as the wor…gotchas/value-cheapest-usable-vram-and-what-it-actually-runs

verifiedhigh · 0.90

Cold start cost of loading a model makes reactive autoscaling structurally late

[blocker] configNew replicas take 2-10 minutes to start because multi-gigabyte weights must load from storage. By the time the new replica is ready, the traffic spike has passed. The fleet is always either over-provisioned (wasting money) or under-provisioning (dropping SLOs).gotchas/serve-cold-start-model-loading-blocks-autoscaling

verifiedhigh · 0.85

Colocation PUE is ~1.9, not the 1.2 often quoted: the 1.2 figure is hyperscale-only, and most third-party inference runs in colocation where the LBNL 2014 baseline shows midtier at 1.9 with cooling alone contributing 0.63

[major] measurementA carbon or energy calculation for an inference deployment uses PUE = 1.2, producing an emissions estimate that is 35-40% too low for a colocation facility. The 1.2 figure is real — it is the LBNL 2014 baseline for hyperscale facilities — but it is the BEST case, not the typical …gotchas/pue-colocation-1-9-not-1-2

verifiedhigh · 0.85

compressed-tensors MXFP4 weight-only checkpoints are dispatched to a W4A4 kernel on SM100+, costing ~7.6 average points across AIME/GSM8K/MATH-500

[major] formatA compressed-tensors MXFP4 checkpoint with input_activations: null (i.e. MXFP4 weights, group_size=32, no activation quantization) served on B200 with --dtype bfloat16 logs 'Using FlashInferMxfp4LinearKernel for MXFP4 GEMM' and then scores far below a weight-only quant should: Qw…gotchas/mxfp4-w4a16-misrouted-to-w4a4-kernel

verifiedhigh · 0.85

Concurrent collectives on 2-3 RCCL communicators deadlock 50-100% of runs on gfx90a, which is the exact shape FSDP2+tensor-parallel produces

[major] kernelRCCL 2.30.4 (HIP 7.14) on 4x gfx90a GCD of an MI250 system deadlocks 50-100% of runs when a process uses 2-3 NCCL communicators concurrently. Upstream PyTorch tests reproduce it directly: TestFullyShardNDTraining.test_2d_mlp_with_nd_mesh ~50-60% hang, and TestFullyShardPerParamMe…gotchas/mgpu-rccl-multiple-communicators-deadlock-gfx90a

verifiedhigh · 0.85

config.json's declared context length disagrees with the vendor's advertised one, sometimes by 3x, and it can be a different key entirely

[major] configA server sized from config.json rejects prompts the vendor's API and model card accept, or a service advertises a context length its own weights were never trained for. Three concrete instances found 2026-10-04: (1) Qwen3-235B-A22B and Qwen3-8B both ship max_position_embeddings =…gotchas/ctx-gotcha-config-context-length-differs-from-claim

verifiedhigh · 0.85

Configuring llama.cpp's SYCL backend after installing oneAPI fails with 'unrecognized command-line option -fsycl'

[blocker] buildAfter installing intel-oneapi-base-toolkit from Intel's apt repository (2025.2/2025.3 packages, including intel-oneapi-compiler-dpcpp-cpp), a GGML_SYCL=ON configure/build aborts with: c++: error: unrecognized command-line option '-fsycl'. The compiler being invoked is not the Int…gotchas/sycl-build-unrecognized-fsycl

verifiedhigh · 0.85

Consensus with a real justification: SGLang is preferred over vLLM specifically for multinode setup and for prefix-heavy structured workloads, and its simpler deployment path is the cited reason

[minor] toolchainThe multinode case is stated bluntly by a practitioner who has run both at tp=16: 'For multinode, I much prefer SGLang's simple setup vs Ray - the docs for vLLM are barely adequate for setting up Ray w/ slurm. I would probably have burnt days on this without the help of Claude an…gotchas/sent-sglang-multinode-and-prefix-consensus

verifiedmedium · 0.70

Consensus with a reservation: TensorRT-LLM is recommended only when you own the hardware and can absorb the build and operational complexity, and even enthusiasts concede the complexity

[minor] buildTwo practitioner sources, four years apart, agree on the shape and differ on the tone. In 2023 a commenter in the launch thread leads with the gating friction: TensorRT-LLM at launch required that 'you must be registered in the NVIDIA Developer Program.' In r/LocalLLaMA a comment…gotchas/sent-tensorrt-llm-worth-the-build-step

verifiedmedium · 0.70

Consensus: llama.cpp is the default recommendation for one user, especially on consumer or non-NVIDIA hardware, and the reasons given are dependency count and quant coverage rather than speed

[minor] toolchainIn a thread asking for the fastest engine for a single personal assistant, the top answer is the bare name: '"llama-server" from "llama.cpp"' (24 upvotes). The second, at 17 upvotes, gives the reason: 'llama.cpp is the way to go if you don't want to mess with lots of Python depen…gotchas/sent-llamacpp-single-user-consensus

verifiedmedium · 0.70

Consensus: vLLM is what production teams reach for by default on datacenter NVIDIA, because of Day-1 model support and ecosystem size - not because it benchmarks best

[minor] toolchainAsked what stacks the industry actually uses, the highest-scoring answer (63 upvotes) from a working ML engineer says: 'Infra for inference wise most production apps use Nvidia TensorRT LLM or vLLM, combined with nvidia triton server usually... vLLM feels less professional but it…gotchas/sent-vllm-enterprise-default-consensus

verifiedmedium · 0.70

Consumer GPUs have no NVLink: multi-GPU inference requires PCIe or model sharding

[major] hardwareMulti-GPU inference on consumer GPUs is limited by PCIe bandwidth, not NVLink. Two RTX 4090s connected via PCIe 4.0 x16 have ~64 GB/s of interconnect bandwidth, compared to 900 GB/s on NVLink. This makes tensor parallelism across consumer GPUs much less efficient than on datacent…gotchas/local-consumer-gpu-no-nvlink

verifiedmedium · 0.80

Consumer NVIDIA cards report no peer access because the driver disables P2P, not because the hardware lacks DMA paths - and NCCL_P2P_DISABLE=1 does not restore it

[blocker] driverOn two RTX 5090s the reporter could not run any multi-GPU inference in either vLLM or SGLang; on 4x RTX 3090 the same wall appeared. The community diagnostic is `nvidia-smi topo -p2p r` returning NS (Not supported) between the cards. The trap that costs people days: setting NCCL_…gotchas/comm-consumer-p2p-disabled-driver-not-hardware

verifiedhigh · 0.85

Contested paper audit: cross-index verification resolves venue disputes but leaves preprints and schema gaps contested

[minor] measurement30 papers were marked status=contested after wave-9/10 venue and hardware sweeps. An audit across arXiv API metadata, PMLR, Crossref, and conference indexes resolved only two records to verified (deepspeed-moe at ICML 2022, whisper at ICML 2023), while 24 records remain contested…gotchas/w11-contested-paper-audit-resolutions-and-unresolved-gaps

verifiedhigh · 0.95

context_scaling_method accepts 'pi' and 'self-extend' and no model record uses either - verified absence, not a gap to be filled

[minor] configA reader querying the model records for context_scaling_method = 'pi' or 'self-extend' gets zero rows out of 155, while the schema's enum lists both as valid values. Read quickly, that looks like two unfilled fields - the same shape as the four records whose context_scaling_metho…gotchas/mdl-gotcha-pi-and-self-extend-are-valid-schema-values-with-zero

verifiedhigh · 0.90

context_scaling_method accepts 'pi' and 'self-extend' and no model record uses either - verified absence, not a gap to be filled

verifiedhigh · 0.90

Continuous batching makes the engine look permanently busy, so queue time disappears into per-request TTFT while throughput stays high — the queue is not empty, it has been absorbed into the batch

[major] operationsUtilisation and throughput look healthy — GPU busy, batch full, output tokens/second at its best — while a large and growing number of requests wait. Nothing in the resource metrics rises. Time-to-first-token for the affected requests grows in proportion to how deep the wait is, …gotchas/ops-continuous-batching-hides-queue-time

verifiedhigh · 0.85

Converting an NVFP4 checkpoint to GGUF silently drops most per-tensor scales, and the model then emits one repeated token for every prompt

[major] toolchainRedHatAI/Muse-Glimmer-30B-NVFP4 converted with convert_hf_to_gguf.py --outtype q8_0 produces a GGUF that does not load: 'check_tensor_dims: tensor blk.0.attn_q_norm.weight not found' / 'llama_model_load_from_file_impl: failed to load model'. The reporter fixed items 1 and 2 local…gotchas/nvfp4-gguf-conversion-drops-block-scales

verifiedhigh · 0.85

Cost attribution per model/tenant is a known gap - no standard solution exists

[major] measurementThe finance team asks how much each model/tenant costs to serve, and there is no answer. GPU utilization is known, but per-request cost is not.gotchas/serve-cost-attribution-gap

verifiedmedium · 0.80

Counter-consensus worth keeping: on Apple Silicon the corpus does not treat llama.cpp as the default, and nominates MLX instead - so the strongest engine consensus in this slice is vendor-scoped

[minor] toolchainThe 'llama.cpp for single user' consensus is stated for CUDA and AMD hardware in every practitioner thread. On Apple Silicon it is contested, from two directions. In the r/LocalLLaMA engine thread a Mac-only user objects to the framing outright: 'Depends on your hardware! For Mac…gotchas/sent-llamacpp-not-the-apple-fast-path

verifiedmedium · 0.70

CPU- and GPU-utilisation-based autoscaling cannot see LLM serving load: an HPA on CPU never fires, and an HPA on GPU utilisation fires on a lightly loaded server

[blocker] operationsA CPU-based HPA on a GPU inference Deployment sits pinned at its minimum replica count while requests queue and latency climbs, because the serving process is not CPU-bound — the frontend is idle waiting on the accelerator. Switching to DCGM GPU-utilisation as the HPA metric look…gotchas/ops-cpu-hpa-blind-to-llm-load

verifiedhigh · 0.92

CPU-based HPA autoscaling fails for LLM serving because load is not visible in request count

[blocker] configHPA scales on CPU utilization, but LLM inference is GPU-bound and CPU utilization stays low even when the GPU is saturated. A single request can hold a GPU for minutes during decode, so request count does not correlate with GPU load. The HPA never scales up, and the fleet is unde…gotchas/serve-hpa-cpu-autoscaling-fails-llm

verifiedhigh · 0.90

CPU/EPYC inference collapses unless every memory channel is populated

[major] configA Genoa box with ONE 64GB DIMM measured 1.5 tok/s on 34B Q4_0, jumping to 4 tok/s with a second stick - one DIMM wastes ~92% of the bandwidth you paid for.gotchas/gotcha-cpu-inference-must-populate-all-memory-channels

verifiedmedium · 0.80

CUDA enumerates at most one Compute Instance per GPU Instance, at most 64 MIG devices total, and needs ulimit -n raised - so a MIG layout reports fewer usable devices than the profile table implies

[major] configA MIG layout created to expose many instances does not present that many usable CUDA devices. Driver >= R570 enumerates a single Compute Instance per GPU Instance, and CUDA 'supports at most 64 MIG instances across all GPUs'. The guide's instruction is explicit: 'Increase the ope…gotchas/mgpu-mig-driver-enumeration-limits-cut-ci-count

verifiedhigh · 0.85

CUDA MPS on top of MIG caps the 48-client limit proportionally to Compute Instance size, and gives up the error isolation that justified MIG

[minor] hardwareStacking MPS over MIG produces a lower client ceiling than the headline 48: 'CUDA MPS is supported on top of MIG. The only limitation is that the maximum number of clients (48) is lowered proportionally to the Compute Instance size.' Separately, MPS never provided error isolation…gotchas/mgpu-mps-max-clients-scale-down-on-mig

verifiedmedium · 0.75

CUDA MPS provides no error isolation - one client crash kills all clients on the GPU

[major] hardwareOne inference engine crashes, and all other engines on the same GPU crash with it. The failure is not contained.gotchas/serve-mps-no-error-isolation

verifiedhigh · 0.85

Custom all-reduce works over PCIe at TP=2 but is silently disabled above TP=2 without full one-hop NVLink, so a PCIe box quietly falls back to the slower NCCL path

[major] configNo error. The observable is a throughput cliff with no diagnostic. The eligibility rule, stated explicitly in upstream: 'At TP=2, custom all-reduce can operate over PCIe when functional GPU P2P is available; NVLink is not required. At TP>2, the existing constructor requires full …gotchas/mgpu-custom-allreduce-needs-nvlink-above-tp2

verifiedhigh · 0.88

CUTLASS grouped FP8 MoE has its own CUDA floor (12.3 on Hopper, 12.8 on Blackwell) that is higher than per-tensor FP8 (12.0 / 12.4) on the same GPU, and no vendor table expresses it

[major] toolchainAn FP8 MoE deployment fails its native-GEMM path on a Hopper GPU that passes every FP8 row in the vendor support matrix, while a dense FP8 model on the same GPU and the same CUDA build runs natively. Or, on Blackwell, both FP8 block-scaled and FP8 MoE fall to a weight-only kernel…gotchas/cutlass-fp8-moe-needs-a-higher-cuda-than-dense-fp8-on-the-same-h

verifiedhigh · 0.90

CUTLASS grouped FP8 MoE has its own CUDA floor (12.3 on Hopper, 12.8 on Blackwell) that is higher than per-tensor FP8 (12.0 / 12.4) on the same GPU, and no vendor table expresses it

verifiedhigh · 0.90

CXL memory expanders are not a consumer path to more inference RAM - host root-port support is server-only

[major] hardwareA tempting path for '70B fits but decode is slow' builds is to bolt on CXL memory. It cannot be done: all shipping CXL expanders (Marvell Structera X 2404/2504/3406/3506, Astera Leo 2 / Leo X, Samsung CMM, XCENA MX1) are E3.S/E3.C PCIe-CEM modules sold to server OEMs, not retail;…gotchas/gotcha-cxl-memory-expanders-not-viable-for-consumers

verifiedmedium · 0.80

CXL memory expanders are not viable for consumer low-budget LLM inference

[minor] hardwareA builder chasing 'more memory for a 70B+ model' looks at CXL expander announcements (Marvell Structera X 2404/2504/3406/3506, Astera Leo 2/Leo X, Samsung CMM, XCENA MX1) and expects to plug one into a desktop board to add cheap capacity. No retail channel sells them, and even if…gotchas/gotcha-cxl-expanders-not-viable-consumers

verifiedhigh · 0.85

CXL memory expanders are server-only - no consumer path in 2026

[blocker] hardwareShipping CXL expanders (Marvell Structera X 3406/3506 CXL 3.2, Astera Leo, Samsung CMM, XCENA MX1) are E3.S/E3.C modules for hyperscaler/server platforms; a card in a consumer board enumerates as nothing.gotchas/gotcha-cxl-expanders-not-consumer-viable

verifiedmedium · 0.80

CXL memory expanders are server-only - no retail channel, no consumer root-complex support

[major] hardwareA hobbyist who wants "more memory for a 70B model" finds CXL expander cards (Marvell Structera X, Astera Leo, Samsung CMM) and assumes they can drop one into a desktop. They cannot: every shipping CXL memory module is an E3.S/E3.C or PCIe-CEM card aimed at hyperscaler/server plat…gotchas/gotcha-cxl-memory-expander-not-viable-for-consumers

verifiedmedium · 0.80

Decode throughput is flat from 400-600 W per GPU and then falls off a cliff at 300 W - a power cap below ~300 W is a hard throughput cliff, not a gradual efficiency gain

[major] measurementFor a memory-bandwidth-bound reasoning workload, per-GPU power caps of 400 W and 600 W leave throughput UNCHANGED, while a 300 W cap causes throughput to drop sharply. So the region between 400 and 600 W is free - you can halve or triple the power envelope with no throughput cost…gotchas/power-cap-cliff-at-300w-for-decode

verifiedmedium · 0.82

Disabling FlashInfer Q-quantization with an FP8 KV cache and speculative decoding silently corrupts sliding-window models on SM100

[blocker] kernel1x B200 (SM100), vLLM 0.23.1rc1.dev843, flashinfer-python 0.6.14, serving RedHatAI/gemma-4-31B-it-NVFP4 (50 of 60 layers SWA, window 1024) with --attention-backend FLASHINFER --kv-cache-dtype fp8_e4m3 --speculative-config MTP num_speculative_tokens=7 and --attention-config '{"dis…gotchas/flashinfer-bf16q-fp8kv-spec-decode-corrupts-swa

verifiedhigh · 0.90

Draft-model speculative decoding is not lossless at temperature 0 when the target model is quantized

[major] kernelUnder greedy sampling (temperature=0, top_k=1), llama-server with --spec-type draft-dspark or draft-mtp produces different text than the same server without -md, when the target is quantized. Example divergence on Qwen3-8B Q4_K_M: vanilla ends '...checks if it has a cached copy o…gotchas/spec-decode-greedy-diverges-on-quantized-target

verifiedhigh · 0.85

Dual-socket EPYC does not double 70B decode - NUMA domain splitting gives ~1.3-1.6x

[minor] configA 2P Genoa box (24 channels, ~920 GB/s aggregate) reads like it should double single-socket 70B decode. It does not: NUMA domain splitting means the model spans both sockets memory unevenly and the interconnect tax eats the gain, so measured uplift is ~1.3-1.6x a single socket.gotchas/gotcha-dual-socket-numa-does-not-double-decode

verifiedmedium · 0.80

Each data-parallel worker picks free ports independently, so concurrent engines collide and the API server dies at startup with a ZMQ bind error

[major] framework2P 4D (pipeline 2, data 4) DeepSeek-V3 on vllm 0.11.0: the API server dies during EngineCoreClient construction. The exact failure, at the bottom of the traceback: 'File ".../vllm/utils/__init__.py", line 2983, in make_zmq_socket / socket.bind(path) ... zmq.error.ZMQError: Addres…gotchas/mgpu-dp-workers-race-for-the-same-zmq-ports

verifiedhigh · 0.85

EAGLE/MTP speculative decoding costs 6-10 points of prefix-cache hit rate, and hybrid layouts make the recompute 100x larger

[major] configMulti-round and repeated-prompt workloads lose ~6-10 points of prefix-cache hit rate versus an identical no-spec boot (97.5% -> 86.9-91.7%). On one hybrid Qwen3.8 GDN layout each hit recomputes a 1,648-token scheduler-aligned unit, costing ~30-40% batch throughput on prefix-reusi…gotchas/eagle-prefix-cache-last-block-drop

verifiedhigh · 0.85

eBay multi-option listings advertise a headline price that buys a cheaper variant

[major] operationsP40 24GB appears at ~$105 on eBay but the '$104.98' headline is a multi-option listing ('P4 8GB / P40 24GB / K80 / M40') - that price buys the P4 variant. Actual P40 selections inside those listings run ~$273-310 and dedicated P40 listings cluster $273-325.gotchas/gotcha-ebay-multilist-headline-price-is-cheaper-variant

verifiedmedium · 0.80

eBay multi-option listings put a bait headline price on the cheapest SKU, not the card you searched for

[minor] operationsA used-card search returns a ~$105 Tesla P40 listing, but that price buys the P4 8GB variant option, not the P40. The actual P40 selection inside the same listing runs ~$273-310. Any scraper or buyer reading the headline price under-prices the real card by ~3x.gotchas/gotcha-ebay-multioption-listing-headline-price

verifiedhigh · 0.85

Eight MLPerf v5.1 Llama 3.1 8B Interactive results are published with unit 'Queries/s' while the underlying harness log prints tokens/s - a silent 129x unit error in the official database

[major] measurementA Llama 3.1 8B Interactive result is read out of the MLCommons v5.1 results database and the Performance_Units column says 'Queries/s'. It is treated as a queries-per-second number, or as a token number, or quietly dropped because the unit does not match the rest of the table. Th…gotchas/bench-mlperf-interactive-unit-mislabelled-queries-per-second

verifiedhigh · 0.95

Enabling MTP speculative decoding together with prefix caching silently costs ~20% accuracy

[blocker] configOn a finetuned Qwen3.6 35B-A3B (MoE) served with MTP (num_speculative_tokens=2) plus --enable-prefix-caching, internal classification accuracy drops ~20% versus the identical setup without prefix caching. Nothing errors; the server is simply faster and wrong.gotchas/mtp-plus-prefix-caching-accuracy-loss

verifiedhigh · 0.85

Enabling speculative decoding with FlashInfer silently downgrades CUDA graphs to PIECEWISE, costing ~16% decode

[major] configA single logger.warning announces that cudagraph_mode was downgraded; decode throughput is 47.5 tok/s on FLASHINFER vs 55.2 tok/s on FLASH_ATTN (+16%), changing only --attention-backend. The warning names neither the cost nor the alternative, so it reads as an informational line.gotchas/spec-decode-silently-downgrades-cudagraphs

verifiedhigh · 0.85

Enabling the V1 engine shrinks usable context several-fold on the same hardware and model

[major] frameworkQwen2.5-Coder-32B-Instruct-GPTQ-Int4 ran at 12K context on 4x RTX 3070 (32 GB) under vLLM 0.6.x. The same model with the same flags under 0.7.0 + VLLM_USE_V1=1 cannot exceed ~3K context and hits CUDA OOM.gotchas/vllm-v1-memory-footprint-growth

verifiedmedium · 0.80

Enabling vLLM's batch-invariance flag does not make the engine deterministic: it silently does not hold for fused-MoE experts, for AWQ on sm8x, and TF32 breaks the matmul invariant - all with no warning

[major] kernelThe failure mode is the dangerous one: not an error, not a crash, and no warning. With VLLM_BATCH_INVARIANT=1 set, vLLM documents that output is 'deterministic and independent of the batch size or the order of requests in a batch' - and on several real configurations it is not. O…gotchas/nrg2-batch-invariance-is-not-universal-across-kernel-paths

verifiedhigh · 0.85

Engine crash loses all in-flight requests - no graceful degradation

[major] frameworkAn inference engine crashes (OOM, CUDA error, segfault), and all requests being processed are lost. Clients receive connection errors and must retry.gotchas/serve-engine-crash-in-flight-requests-lost

verifiedhigh · 0.85

EPYC CPU inference wastes ~92% of bandwidth if every memory channel is not populated

[major] configA Genoa/EPYC box bought for CPU LLM inference delivers a tiny fraction of its expected tok/s when the DIMM slots are not fully populated. llama.cpp discussion #3167 measured a Genoa box with ONE 64GB DIMM at 1.5 tok/s on 34B Q4_0, jumping to 4 tok/s with just a second stick.gotchas/gotcha-epyc-must-populate-all-memory-channels

verifiedmedium · 0.82

Every MLPerf Inference invalidation on record has the same cause - preview results that were never converted to 'available' - so an invalidated number is an unfinished submission, not a rule violation

[major] configA vendor number is quoted from an MLPerf round, and later the MLCommons results change log shows that submission was invalidated. The reflexive conclusion is that the submitter cheated or mismeasured. Reading the actual change log, that conclusion is wrong every time. The complet…gotchas/bench-mlperf-invalidation-is-always-preview-not-fraud

verifiedhigh · 0.95

Every serious engine has THREE different build steps with three different durations, and conflating them is the reason 'how long does it take to build' has no single answer

[major] buildA reader asks how long it takes to build TensorRT-LLM, or vLLM, or FlashInfer, and gets an answer that is wrong for their situation because three distinct costs share the name 'the build'. COST 1, the C++/CUDA compile of the engine itself: vLLM >=3 hours on an A100 host for a sin…gotchas/ev-build-cost-is-three-separate-builds

verifiedhigh · 0.85

Expert-parallel size that does not divide the physical expert count fails with a bare AssertionError naming num_redundant_experts

[major] configOn a 3x A100-SXM4-80GB with --ep-size 3 and a MoE model with 256 physical experts: 'AssertionError: n_physical_experts=256 must be divisible by ep_size=3. Adjust num_redundant_experts.' The engine aborts during model construction. The message's suggested remedy (num_redundant_exp…gotchas/mgpu-ep-size-must-divide-physical-experts

verifiedhigh · 0.85

FlashInfer's b12x MoE backend silently defaults quant_mode to W4A4, so a weight-only NVFP4 checkpoint is computed at the wrong activation precision

[major] configQwen3.6-35B-A3B-NVFP4 (quant_algo W4A16_NVFP4 for mlp.experts) served with --moe-backend flashinfer_b12x runs the FP4xFP4 micro-kernel (MoEMicroKernel in blackwell_sm12x.moe_micro_kernel, BF16 activations quantized to FP4 in-kernel) instead of the W4A16 kernel in moe_w4a16_kernel…gotchas/flashinfer-b12x-moe-runs-w4a16-as-w4a4

verifiedhigh · 0.85

For almost every engine in this repo the honest answer to 'does it run in production, and at what scale' is NO EVIDENCE FOUND - and that negative is worth more than an inferred answer

[minor] measurementAsked to record real-world deployment evidence for each engine, the actual yield on 2026-10-04 was TWO engines with first-party-adjacent numeric field reports from named production sites: [[sglang]] (issue #40877, DeepSeek-V4.1-Flash on 8x RTX PRO 6000, 'field data from running .…gotchas/ev-production-evidence-is-scarce-and-nearly-always-single-site

verifiedhigh · 0.90

For Cloud TPU, 'scale-up' is within a POD over ICI and 'scale-out' is between slices/pods over the DCN - Google documents this as ICI vs DCN and never uses the words scale-up/scale-out, so a reader who maps scale-up to 'chips on one host' or scale-out to 'between hosts in a pod' inverts the boundary

[major] hardwareTwo distinct misstatements, and the first is the one this gotcha is named for. (a) HOST/POD INVERSION: treating scale-up as 'chips attached to one host' and scale-out as 'between hosts'. On TPU this is wrong in both directions at once, because the scale-up domain is the whole POD…gotchas/w7c-gotcha-tpu-scale-up-scale-out-is-ici-vs-dcn-not-host-vs-pod

verifiedhigh · 0.85

For FP4 Llama 2 70B a full 72-GPU NVL72 rack is measurably worse per GPU than an 8-GPU node: scale-up fabric buys nothing at this model size

[major] hardwareA whole NVL72 rack is specified for a throughput-bound Llama 2 70B serving workload, on the reasoning that a 72-GPU NVLink domain will serve it faster than a node will. The audited MLPerf numbers say the opposite, per GPU. NVIDIA's own 8-GPU slice of a GB300 NVL72 (2 nodes, 4 acc…gotchas/bench-mlperf-rack-scale-is-per-gpu-negative-for-70b

verifiedhigh · 0.90

Forcing the FlashInfer attention backend fails on head_dim 256/512 models

[major] kernelEngine init aborts with: ValueError: Selected backend AttentionBackendEnum.FLASHINFER is not valid for this configuration. Reason: ['head_size not supported']. Reproduces for the bf16 base model, an FP8 checkpoint and an NVFP4 checkpoint of google/gemma-4-31b-it on an RTX PRO 600…gotchas/flashinfer-rejects-large-head-dim

verifiedhigh · 0.90

Forking after HPU driver initialization causes hangs on exit; vLLM auto-overrides to spawn

[major] configOn HPU, if vLLM's worker start method is set to 'fork' (the upstream default), forking after HPU driver initialization leaves driver state in child processes and can cause hangs on exit. vLLM logs a warning when it auto-overrides to 'spawn'.gotchas/xpu-gaudi-fork-after-driver-init

verifiedhigh · 0.90

FP8 KV cache silently loses long-context accuracy above ~100K tokens on Hopper, with no error and no latency change

[blocker] kernelWith --kv-cache-dtype fp8 on Hopper, a 128k needle-in-a-haystack task scored 13 percent against a 91 percent BF16 baseline on the same model and hardware. There is no exception, no warning, and no change in the latency histogram - the request is fast, the answer is fluent, and it…gotchas/ctx-gotcha-fp8-kv-accumulation-cliff-at-100k

verifiedhigh · 0.85

FP8 on RDNA3 blows the 600 s engine-ready timeout on first start even though weights load in seconds

[major] configvLLM serve of an FP8 checkpoint on a gfx1100 Radeon aborts during startup with an engine-ready timeout after waiting VLLM_ENGINE_READY_TIMEOUT_S (default 600 s); measured first init was 1474.26 s. The message blames 'slow weight loading' while the weights actually loaded in 16.33…gotchas/vllm-rocm-fp8-cold-start-timeout

verifiedhigh · 0.95

Gaudi 2 HL-225 is OAM-only - no PCIe SKU, needs $11,995 chassis + fabric

[blocker] hardwareUsed Gaudi 2 modules list at $1,500-2,700 and look like a bargain for 96GB HBM2e, but there is no PCIe SKU and the module cannot power on outside a Gaudi UBB chassis.gotchas/gotcha-gaudi2-oam-only-no-pcie-sku-chassis-required

verifiedmedium · 0.80

Gaudi 2 rumoured $700-900 module price is a stale 2025 liquidation echo - not on eBay Oct-2026

[major] hardwarePlanning docs and community threads carry a '$700-900 used Gaudi 2 module' figure that does not appear in any Oct-2026 eBay listing - actual asks are $1,499-2,700.gotchas/gotcha-gaudi2-rumoured-700-price-not-on-market

verifiedmedium · 0.75

GDPR Article 30 obliges a PROCESSOR to keep a record of processing including third-country transfers and safeguards - and an inference deployment is a processor with both

[major] complianceAn organisation treats GDPR compliance as a web-form exercise done once by legal. The engineering reality is that every place a prompt can land - logs, traces, object storage, caches, the inference vendor - is a processing activity that has to appear in a record, and a new infere…gotchas/sec-gdpr-art30-record-of-processing-is-a-per-deployment-obligati

verifiedhigh · 0.90

GDPR Article 30 obliges a PROCESSOR to keep a record of processing including third-country transfers and safeguards - and an inference deployment is a processor with both

verifiedhigh · 0.90

Generic container metrics (CPU, memory, request count) are actively misleading for LLM serving

[major] measurementDashboard shows healthy CPU/memory/request metrics while users experience high latency. The monitoring stack reports green while the LLM service is degraded.gotchas/serve-observability-generic-container-metrics-misleading

verifiedhigh · 0.88

Genuinely divided, recorded as a split: on whether the Strix Halo is a bad buy, the thread splits into owners who are delighted and buyers who are not, and the split is not resolvable by benchmarks

[minor] hardwareA post titled 'Why the Strix Halo is a poor purchase for most people' (103 upvotes, 394 comments) produced a thread where the top-voted response is a single line of mockery at 182 upvotes - 'This just in, more expensive computers are faster than less expensive computers. More at …gotchas/sent-community-split-strix-halo-owners-versus-buyers

verifiedmedium · 0.70

Genuinely divided, recorded as a split: whether to buy inference hardware at all is settled only by usage pattern, and price-per-token arguments are explicitly rejected by the local-hardware camp

[minor] hardwareA thread asking whether to buy 4x Ascend GX10 (39 upvotes, 169 comments) surfaces the whole buy-versus-rent dispute cleanly. Rent side, 46 upvotes: 'This is such a bad financial decision it's almost funny. There's some real Tulip Mania nonsense going on in this community... you a…gotchas/sent-community-split-buy-local-versus-rent

verifiedmedium · 0.70

gfx906 has no MFMA matrix-core instructions: every fp8/int8 matrix path that assumes matrix cores is unavailable or must be emulated, and it is the architectural reason gfx906 decode lags newer AMD parts

[major] hardwareNot a crash - a ceiling. Anything requiring matrix cores simply cannot be used. Three concrete community reports: (1) a proposed fp8 kernel 'is not really compatible with gfx906 as it used the power of matrix cores (tensor engines) not supported by gfx906', forcing a fallback tha…gotchas/comm-gfx906-mfma-absent-no-matrix-cores

verifiedmedium · 0.75

gfx906 has no upstream vLLM or Triton support: MI50/MI60 inference requires a community fork of vLLM, a fork of Triton, or a fork of llama.cpp

[blocker] frameworkStock vLLM on ROCm does not bring up an LLM on MI50/MI60. Observed and reported failure modes, each by different users: (a) vLLM V1 engine aborts at init with 'HIP error: invalid argument at torch.cuda.mem_get_info()' in spawned workers because HSA_OVERRIDE_GFX_VERSION does not p…gotchas/comm-gfx906-no-upstream-support-forks-required

verifiedhigh · 0.90

gfx906 is officially 'deprecated' in ROCm's support matrix yet still works in practice - the gap between the two states is where every workaround in this slice lives, and AMD has been asked to reverse it

[minor] configThe upstream issue that established the state of play states it plainly: 'However, Radeon VII gfx906 still listed as deprecated instead of unsupported', alongside the rocBLAS break - i.e. the documentation promise and the packaging do not agree. In the other direction, a communit…gotchas/comm-gfx906-spec-deprecation-vs-working

verifiedmedium · 0.80

gfx906 PCIe peer-to-peer links work but are non-coherent and carry NO 32-bit or 64-bit atomics, which is the root cause behind gfx906 multi-GPU hangs and RCCL/VLLM_SKIP_P2P_CHECK workarounds

[major] driverA user dumping KFD p2p_links on a 3-GPU gfx906 box with 32 GB BAR each reports every direction as: ROCm0 -> ROCm1 | 15 (0b1111) | ENABLED | NON_COHERENT | NO_ATOMICS_32_BIT | NO_ATOMICS_64_BIT and the same for ROCm0->ROCm2 and ROCm1->ROCm2. So P2P is nominally enabled and bandw…gotchas/comm-gfx906-p2p-no-atomics

verifiedmedium · 0.70

GitHub's repos/{owner}/{repo} open_issues_count INCLUDES open pull requests, so quoting it as an issue count overstates unresolved bug reports - and the repos API is undocumented on this point

[minor] measurementAn engine record says 'N open issues' and a reader concludes there are N unresolved problems. The GitHub REST API's repository object returns a field named open_issues_count whose value includes OPEN PULL REQUESTS, and the endpoint's own documentation does not say so. The effect …gotchas/ev-github-open-issues-count-includes-pull-requests

verifiedmedium · 0.80

Google Coral Edge TPU is INT8 vision-only - no LLM path

[blocker] hardwareNo LLM serving path, no transformer-decode support, no int4/GGUF story exists for the Edge TPU.gotchas/gotcha-coral-edge-tpu-vision-only-int8

verifiedhigh · 0.90

Google does not use one spec-table column convention across TPU generations: the same row name means per-chip on v3, per-pod on v5e/v5p, and per-chip-summed-over-chiplets on Ironwood

[major] hardwareEvery Google per-generation spec table uses the SAME ROW LABELS - 'Peak compute per chip', 'HBM capacity per chip', 'HBM bandwidth per chip', 'TPU Pod size', 'ICI bandwidth per chip' - while the SCOPE behind each label differs per page and per generation, in three separate ways. …gotchas/w7x-gotcha-tpu-compute-table-columns-change-scope-generation-to

verifiedhigh · 0.85

Google does not use one spec-table column convention across TPU generations: the same row name means per-chip on v3, per-pod on v5e/v5p, and per-chip-summed-over-chiplets on Ironwood

verifiedhigh · 0.85

GPU board power is 55-59% of system power: CPU, DRAM, NICs, PSU losses and cooling add 41-45% on top, so a joules-per-token figure without its measurement boundary is meaningless

[major] measurementA joules-per-token figure is quoted in a paper or vendor deck without stating whether it is measured at the GPU package, the accelerator board, the node, or the rack. The number is then compared against another figure measured at a different boundary, and the comparison is wrong …gotchas/board-power-vs-system-power-measurement-boundary

verifiedhigh · 0.88

GPU power and link state are node-global, so you cannot set or change them from inside a shared multi-tenant allocation - and a node someone else down-clocked will silently cap your throughput

[minor] configA benchmark or production tuning session that works perfectly on a dedicated node behaves differently on a shared cluster, with no error from the tooling. Attempts to fix it fail confusingly: the clock lock does not take, or applies to some devices and not others, or the command …gotchas/node-global-power-state-unsettable-in-shared-allocation

verifiedmedium · 0.82

GPU utilization is a poor proxy for LLM load - KV cache utilization is the metric that matters

[major] measurementGPU utilization is high but throughput is low, or GPU utilization is low but the service is degraded. The metric does not correlate with user experience.gotchas/serve-kv-cache-utilization-not-gpu-utilization

verifiedhigh · 0.88

Grid carbon intensity varies 10-50x by region, so a token generated in Quebec (~20 gCO2/kWh) is not the same as one generated in West Virginia (~900 gCO2/kWh) — and almost nobody accounts for it

[major] measurementAn emissions claim for an inference deployment is computed as joules-per-token x average grid carbon intensity, producing a single number that is wrong by an order of magnitude depending on where the tokens are actually generated. The claim 'inference emits X gCO2 per token' is m…gotchas/grid-carbon-intensity-by-region-inference-emissions

verifiedhigh · 0.85

HabanaAI/vllm-fork reached EOL at v1.23.0; vllm-gaudi plugin is the only supported path

[major] frameworkThe HabanaAI/vllm-fork repository reached end-of-life (EOL) at v1.23.0 and is deprecated in v1.24.0, remaining functional only for legacy use cases until then. The vllm-gaudi plugin is production-ready in v1.23.0 and becomes the default by v1.24.0.gotchas/xpu-gaudi-fork-eol

verifiedhigh · 0.95

HIPAA requires auditability AND minimisation, and inference logging collides with both - 164.312(b) wants the records, 164.316 keeps them 6 years

[major] complianceA team tries to satisfy the HIPAA Security Rule on an inference deployment and gets stuck in a contradiction: turn on detailed request logging to satisfy the audit-control standard, and you have just built a six-year PII repository; turn it off, and you cannot evidence who access…gotchas/sec-hipaa-audit-controls-conflict-with-prompt-log-retention

verifiedhigh · 0.90

Huawei Ascend 310P / Atlas 300I requires CANN+MindIE - hobbyist trap

[major] toolchainReal serving hardware that can run LLMs, but requires heavyweight enterprise tooling: driver/toolkit version pinning, a model-conversion pipeline, and primarily Chinese-language docs.gotchas/gotcha-ascend-cann-mindie-stack-pain

verifiedmedium · 0.75

Hugging Face TGI was retired, not merged into vLLM - the codebase was never absorbed and no TGI code is in vLLM

[minor] frameworkPeople describe TGI as having been 'merged into vLLM' or 'absorbed into vLLM' when Hugging Face retired it. Acting on that belief produces two wrong conclusions: that vLLM contains TGI code (it does not), and that TGI was kept alive by vLLM (it was not). The repository is ARCHIVE…gotchas/tgi-retired-not-merged-into-vllm

verifiedhigh · 0.92

HW Power Brake fires on external power-supply assertion and cuts core clocks by a factor of 2 or more - a PSU or power-delivery limit presents identically to thermal throttling

[minor] hardwareThroughput drops sharply and then recovers, intermittently, with no thermal explanation available - and the symptom looks exactly like thermal throttling, which sends people to check fans and inlet temperature and find nothing wrong. The correlated signature is that the events cl…gotchas/psu-power-brake-throttles-under-dvfs-transients

verifiedmedium · 0.72

HW Power Brake means the PSU asserted: core clocks drop by 2x or more, and the symptom is an unexplained throughput collapse rather than a PSU error

[major] hardwarenvidia-smi reports 'HW Power Brake' active, meaning an external power brake assertion has been triggered by the system power supply. Per NVIDIA's definition this engages HW Slowdown, which is 'reducing the core clocks by a factor of 2 or more'. The observable is a sudden, large, …gotchas/hw-power-brake-psu-instability-crash

verifiedhigh · 0.90

HW Thermal Slowdown cuts core clocks by a factor of 2 or more, and nvidia-smi reports it as a microsecond counter rather than a visible failure

[major] hardwareSustained-token-rate collapses mid-run with no error, no crash and no assertion. A batch that measured X tokens/s over a short run measures roughly X/2 over a long one. The harness reports the run as VALID, the output stays correct, and nothing anywhere in the application's logs …gotchas/hw-thermal-slowdown-halves-clocks-silently

verifiedhigh · 0.88

Hybrid Gated DeltaNet models keep fast prompt processing but collapse ~25x in decode past ~80K context

[major] kernelWith the model fully resident on GPU (zero 'assigned to device CPU' lines in the log), decode falls from ~33 t/s at 68K KV position to 1.4 t/s (714 ms/token) at 91K position, or times out, while prompt processing stays ~1300 t/s at the same positions. Zero-context decode is norma…gotchas/gated-deltanet-decode-collapse-long-context

verifiedhigh · 0.85

Hybrid sparse+dense retrieval takes gradient-guided corpus poisoning from 38% to 0% - until the attacker optimises against both, then back to 20-44%

[major] securityThe RAG hardening measure that works in the paper does not work in production, because production attackers optimise against whatever defence is deployed. A defence that reduces poisoning to zero against a naive attack is not a defence.gotchas/sec-rag-corpus-poisoning-not-caught-by-hybrid-retrieval-alone

verifiedhigh · 0.88

If one NCCL communicator-init thread errors (for example an OOM), the remaining init threads block forever on a bootstrap socket that has no timeout

[major] driverObservable is a hang, not an OOM error. NCCL's group.cc init path spawns one thread per rank and then joins them: it spins on pthread_tryjoin_np for each ASYNC_FUNC_INIT thread, propagating any thread's error into ret. The deadlock is that when one thread errors out, the others a…gotchas/mgpu-nccl-init-thread-error-deadlocks-others

verifiedmedium · 0.80

In disaggregated serving a dead prefill worker is not detected: the decode side attempts the KV pull anyway, and with the default failure policy the client gets a 500 while the stranded KV is only reclaimed by a lease timer

[major] operationsA prefill pod crashes. Every request whose KV pull was in flight against it fails, and there is no liveness check to notice the pod is gone first. Separately, the prefill instance keeps the dead requests' KV blocks reserved — the decode side has not pulled them, so they are stran…gotchas/ops-pd-stranded-kv-and-silent-recompute

verifiedhigh · 0.90

In llama.cpp, --tensor-split does not move MoE expert tensors, so a GPU split that looks configured is actually leaving expert compute on one device

[minor] configObservable is a persistent asymmetry, not a crash or corruption: 'Even where --tensor-split doesn't cause degeneration, it barely rebalances actual per-GPU compute load: MoE experts stay almost entirely on one device regardless of the split ratio tried; only non-MoE tensors seem …gotchas/mgpu-llamacpp-tensor-split-does-not-balance-moe-experts

verifiedmedium · 0.72

In-decode filters must handle batch-granularity rows, not requests: rows reorder every step and one request can own several rows under speculative decoding

[major] securityA logits processor written against per-request semantics produces output attributed to the wrong request, or applies the wrong transform, once speculative decoding is enabled or once continuous batching reorders the batch. The failure is silent and looks like occasional mis-filte…gotchas/sec-decode-loop-filter-row-is-not-request

verifiedhigh · 0.85

InfiniBand costs 2-5x more than RoCE per port but offers hardware congestion control and SHARP

[major] hardwareInfiniBand hardware costs significantly more than equivalent Ethernet/RoCE hardware, but provides hardware-based congestion control, SHARP in-network reduction, and better tail latency under load.gotchas/nic-infiniband-vs-roce-cost

verifiedmedium · 0.75

InfiniBand NDR/XDR naming refers to whole-port rate, not per-lane rate

[major] measurementNDR is described as '400 Gb/s' and XDR as '800 Gb/s', but these are whole-port figures. The per-lane rates are 100 Gb/s and 200 Gb/s respectively. Confusing per-lane and per-port rates leads to 4x errors in bandwidth calculations.gotchas/nic-ndr-xdr-naming-confusion

verifiedhigh · 0.95

Injection resistance cannot be self-evaluated - it needs a benchmark with a fixed threat model, fixed system prompt and a defined attack suite

[major] securityA team believes its injection defence works because a set of hand-written attacks is refused. Every published attack they later read breaks it. The problem is that the evaluation was never comparable to anything.gotchas/sec-prompt-injection-eval-needs-a-benchmark-not-a-handwritten-su

verifiedhigh · 0.90

Injection resistance cannot be self-evaluated - it needs a benchmark with a fixed threat model, fixed system prompt and a defined attack suite

verifiedhigh · 0.90

Input guardrails on the user turn do not cover the retrieved turn, because indirect injection arrives as data the app itself fetched

[blocker] securityYou put a classifier on the user's prompt. The tests pass against hand-written direct jailbreaks. Then the application retrieves a document the attacker controls - a web page, a wiki page, an email, an issue comment, a support ticket - and the injection succeeds through a turn no…gotchas/sec-guardrail-on-user-turn-misses-the-retrieved-turn

verifiedhigh · 0.90

Intel Arc GPUs achieve only ~32% of theoretical memory bandwidth in llama.cpp due to unoptimized SYCL and Vulkan quantization kernels

[major] kernelOn Intel Arc GPUs (such as Arc B580 Battlemage), llama.cpp token generation throughput via both SYCL and Vulkan backends severely underperforms hardware bandwidth. On an Arc B580 with 456 GB/s memory bandwidth, generating tokens on a 3.56 GB model achieves only 41.7 tok/s (~148 G…gotchas/intel-arc-sycl-vulkan-underutilization

verifiedmedium · 0.80

Intel Extension for PyTorch (IPEX) is archived and unsupported as of March 2026

[major] frameworkThe intel-extension-for-pytorch GitHub repository was archived by the owner on Mar 30, 2026. It is now read-only. Intel will not provide or guarantee development, support, maintenance, bug fixes, new releases or updates. Patches are no longer accepted. The project has known secur…gotchas/xpu-ipex-archived-no-support

verifiedhigh · 0.95

Intel Gaudi 2 HL-225 is OAM-only with no PCIe SKU - needs a UBB chassis + 24x 100GbE fabric to light up

[blocker] hardwareA bare Gaudi 2 HL-225 module (~$1,500-2,700 on eBay) cannot be installed in a workstation - it has no PCIe edge connector and no standalone power/cooling path. It requires the Gaudi UBB baseboard (8-module tray ~$11,995) plus a 24x 100GbE fabric just to boot.gotchas/gotcha-gaudi2-hl225-oam-only-chassis-required

verifiedhigh · 0.85

Inter-node all-reduce runs at roughly half speed under SR-IOV InfiniBand VF pass-through into VMs, and NCCL needs a hand-written topology XML to even see the fabric

[major] hardwareTwo physical A100 SXM4*8 nodes with two HDR InfiniBand cards each, with SR-IOV VF pass-through into a VM per node (host NUMA and PCIe topology mirrored, hugepages and vCPU pinning applied): inter-node all_reduce_perf runs at about 56% of expected. To get NCCL to recognise the top…gotchas/mgpu-vm-passthrough-halves-inter-node-allreduce

verifiedmedium · 0.75

Inter-token latency and per-request time-per-output-token have different denominators and different weights; swapping them silently changes your SLO

[major] measurementAn alert tuned on one metric does not fire when the other degrades, or fires constantly with no user complaint. A dashboard shows p99 decode latency of 80 ms while individual users report multi-second stalls, or the reverse: a comfortable-looking p99 with visible stutter. Two eng…gotchas/obs-itl-vs-request-tpot-are-different-metrics

verifiedhigh · 0.92

Intra-node NVLink latency is ~10x lower than inter-node InfiniBand latency, dictating TP vs PP placement

[major] hardwareTensor-parallel sharding across a node boundary (using InfiniBand instead of NVLink) incurs ~10x higher latency per collective operation, which can dominate inference time for small-batch or low-latency workloads.gotchas/nic-nvlink-vs-infiniband-latency-gap

verifiedhigh · 0.85

Ironwood and Trillium are two different TPU generations, and Google's own naming hides which is which: a query for 'Ironwood' or for 'Trillium' returns half the answer

[major] hardwareGoogle gave the two most recent TPU generations plant codenames and kept the marketing name DETACHED from the generation number in the docs: v6e is 'Trillium' and TPU7x is 'Ironwood'. The mechanical version number IS printed on every page ('TPU v6e (Trillium)', 'TPU7x (Ironwood)'…gotchas/w7x-gotcha-ironwood-is-tpu7x-not-trillium

verifiedhigh · 0.85

Joules per token across the corpus spans 4.11 to 20.53 on comparable batch-1 rows, but three rows had to be excluded - an unsourced figure, three multi-GPU runs charging one card's TDP, and a batch-2048 row whose real batch is 16x larger than batch 1

[major] measurementThe corpus has both TDP for 116 accelerators and decode_tok_s for 32 benchmark records, so joules per token looks like a straightforward join. The naive join produces absurd figures: the GB200 NVL72 row at 0.16 J/tok, which would make a 27kW rack-scale system the most energy-effi…gotchas/value-joules-per-token-and-why-the-cheapest-energy-is-not-the-ch

verifiedhigh · 0.90

Kernel compilation cost dominates for short-running or frequently-changing workloads

[major] toolchainFor workloads with short execution times or frequently-changing shapes, the cost of JIT compilation (Triton: 0.5-5s per kernel, torch.compile: 30s+ for complex models, CUTLASS: minutes for template instantiation) can exceed the actual inference time. A model with 50 unique kernel…gotchas/kernel-compilation-cost-dominates-for-short-running-or-frequentl

verifiedhigh · 0.85

KGW watermarking: hash the previous token, split the vocabulary in half, promote green tokens - and the decode-time cost is dominated by the vocabulary permutation

[minor] securityServing a watermarked model either 'costs nothing' or is unbearably slow, and the difference is the vocabulary permutation, not the logit bias. A delta of a few hundredths on quality is easy; a randperm over a 150k-token vocabulary on every decode step is not.gotchas/sec-watermark-kgw-green-list-mechanism-and-cost

verifiedhigh · 0.85

Kinara first-party datasheets and domain inaccessible (HTTP 404)

[major] hardwareHTTP 404 Not Found returned when fetching kinara.ai, www.kinara.ai, and specific product paths (ara-1, ara-2).gotchas/ainv-gotcha-kinara-datasheets-inaccessible

verifiedhigh · 0.85

kv_cache_bytes_per_token = 0 is three different answers wearing one number, and a capacity plan built on it is wrong in two of the three cases

[major] measurementA reader sums kv_cache_bytes_per_token across a fleet to size KV memory and gets a plausible-looking number, because the field is populated on every model record and is a number everywhere. The field carries THREE semantically different answers behind the single value 0, and the …gotchas/kv-cache-bytes-zero-is-three-different-answers

verifiedhigh · 0.85

kv_cache_bytes_per_token = 0 is used for three unrelated situations in this repo, and a reader querying on it gets all three unless the notes are read

[minor] configThirty-two records carry kv_cache_bytes_per_token = 0 and they do not mean the same thing. Three groups sit behind the single value: (1) genuinely cache-free architectures - bidirectional encoders (bge-*, e5-*, gte-*, snowflake-arctic-embed-l, modernbert-base, xlm-roberta based r…gotchas/mdl-gotcha-kv-zero-means-three-different-things

verifiedhigh · 0.90

llama.cpp -sm tensor on multiple gfx906 cards produces pure garbage text, and scaling past 2 GPUs over PCIe collapses decode tok/s by ~2.5x per doubling even though prefill keeps rising

[blocker] configTwo separate failure shapes. (1) CORRECTNESS: --split-mode tensor on 2 MI50s emitted continuous word-salvage: 'l own same aCB l’aC-// own-B l’CCS-al ly’-s’B l own same own sameS ...' - no error, no crash, just destroyed text. (2) SCALING: with gpt-oss-120b MXFP4 MoE, -b 2048 -ub …gotchas/comm-gfx906-tensor-split-garbage-and-pcie-degradation

verifiedmedium · 0.70

llama.cpp backend performance varies dramatically: CUDA > Metal > Vulkan > CPU

[major] frameworkThe same GGUF model on the same hardware can run at wildly different speeds depending on the llama.cpp backend. CUDA is typically 2-5x faster than Metal, which is 2-5x faster than Vulkan, which is 2-5x faster than CPU.gotchas/local-llamacpp-backend-reality

verifiedmedium · 0.80

llama.cpp has no paper, and no arXiv ID may be invented for it - the absence is a verified fact, not a gap

[minor] toolchainA reader auditing this repo for inference coverage will notice that llama.cpp - by most measures the most widely deployed LLM inference engine in the world - has engine, kernel and gotcha records but NO record in data/papers/. The tempting response is to go and find an arXiv ID f…gotchas/meta-llama-cpp-has-no-paper-do-not-invent-one

verifiedhigh · 0.90

llama.cpp HIP on gfx1151 returns wrong-but-plausible logits for any prompt longer than n_ubatch, and the default n_ubatch makes it almost universal

[major] kernelAny prompt longer than n_ubatch produces badly wrong logits on HIP/ROCm on gfx1151 - no error, no warning, no crash, and the numbers look plausible. Default n_ubatch is 512, so in practice almost every real prompt is affected. WikiText-2 perplexity, same 2-item set, HIP vs Vulkan…gotchas/hip-wrong-logits-when-prompt-exceeds-ubatch

verifiedhigh · 0.85

llama.cpp is MIT: no patent grant, no copyleft, only requires preserving copyright notices

[minor] configYou want to use llama.cpp in a commercial product and need to understand the license terms. MIT is simpler than Apache 2.0 but has no patent grant.gotchas/meth-llamacpp-mit-no-patent-grant

verifiedhigh · 0.90

llama.cpp SYCL backend removed Nvidia and AMD GPU support in 2026.02

[major] frameworkAs of the 2026.02 release, the llama.cpp SYCL backend no longer supports Nvidia or AMD GPUs. Previously, the SYCL backend could target any GPU via the oneAPI plugin for Nvidia and AMD, but the download/installation channels for that plugin are no longer available.gotchas/xpu-sycl-removed-nvidia-amd-support

verifiedhigh · 0.90

llama.cpp upstream regressions on gfx906 ship undetected because AMD maintainers hold NVIDIA hardware - the community has to run its own pre/post-merge A/B on every merge

[major] frameworkA concrete instance: the fork maintainer reports that upstream PR ggml-org/llama.cpp#20793 'got merged... and by my testing the original PR broke pipeline parallelism at least on my MI50 system.' Models affected in his test: gemma-4-26B-A4B q8, gpt-oss-120B MXFP4 MoE, gemma-4-31B…gotchas/comm-gfx906-upstream-regression-risk-no-amd-review

verifiedmedium · 0.70

llama.cpp's --tensor-split can produce intermittent output degeneration on long-context MoE inference, and barely rebalances MoE experts regardless of the ratio given

[major] frameworkllama-server 0.3.0-dev (build 10731) with '--model Qwen3-Coder-Next-UD-Q4_K_M.gguf --ctx-size 65536 --gpu-layers 999 --split-mode layer --tensor-split 4,1 --n-cpu-moe 34 --flash-attn on --parallel 1' on sessions with growing multi-turn context: the model intermittently degenerate…gotchas/mgpu-llamacpp-tensor-split-breaks-nondivisible-multi-gpu-shardin

verifiedmedium · 0.80

llama.cpp's --tensor-split can produce intermittent output degeneration on long-context MoE inference, and barely rebalances MoE experts regardless of the ratio given

verifiedmedium · 0.80

llama.cpp's native FP4 MMQ kernel config is gated at CC 1200-1299, so an MXFP4 or NVFP4 GGUF on a B200/B300 runs the Ampere integer MMQ path while the same quant type on an RTX 5090 runs the Blackwell tensor-core path

[major] kernelAn MXFP4 or NVFP4 GGUF shows no Blackwell-class throughput on a B200 or B300, and does on an RTX 5090 — the inverse of every other engine in this repo. Read as 'llama.cpp does not have a native FP4 path on datacenter Blackwell', which is what the constant values say.gotchas/llamacpp-native-fp4-mmq-is-sm120-only-not-sm100

verifiedhigh · 0.90

Long context is not servable on older hardware - the attention kernel is gated by SM generation, and one family hits two head_dim-256 walls

[major] kernelTwo failures that look like anything but what they are. (1) A context length that works on one node fails on another with the same models, because the attention backend is chosen by SM generation: SM80 (A100, A10, L40S, 4090) has FlashAttention-1/2 only and no validated fp8 atten…gotchas/ctx-gotcha-long-context-kernel-gate-by-sm-generation

verifiedmedium · 0.82

Maintenance status is four dated facts, and the one that predicts everything else is the last commit date - measured across all 100 engine repos with resolvable GitHub metadata

[minor] measurementA reader picks between engines on the axis 'is this alive' and gets impression, star count or an undated adjective. The four facts that answer it are: last commit date, last release date, contributor count, archived flag - all four pullable from the GitHub API on a stated date, a…gotchas/ev-maintenance-status-is-four-dated-facts

verifiedhigh · 0.95

Major cloud providers do not publish internal GPU fabric bandwidth specifications

[minor] measurementCloud providers (AWS, GCP, Azure) do not publish the internal bandwidth of their GPU cluster fabrics, making it impossible to compare cloud vs on-premises performance directly.gotchas/nic-cloud-fabric-bandwidth-not-public

verifiedhigh · 0.90

Matryoshka embedding models support dimension truncation with minimal quality loss

[minor] configEmbedding models trained with Matryoshka Representation Learning (MRL) can be truncated to fewer dimensions at inference time with little to no loss in retrieval quality, turning the vector-store storage/compute tradeoff into a simple configuration choice.gotchas/emb-matryoshka-dimension-truncation

verifiedhigh · 0.90

Methodology: upvote and point counts in this sentiment layer are a popularity proxy observed at a date, never a measurement of quality or of popularity in general

[minor] measurementEvery quantitative claim about community sentiment in this slice carries an upvote or Hacker News point count, and the risk is that a reader treats those numbers as results. They are not. A 182-upvote one-line reply on a 103-upvote post is not 'the community prefers X by a factor…gotchas/sent-vote-counts-are-a-popularity-proxy-not-a-measurement

verifiedmedium · 0.70

MI100 multi-GPU segfaults on bare-metal ROCm; needs VM passthrough + HSA_AMD_SVM

[blocker] frameworkMore than one MI100 segfaults llama.cpp on bare-metal Arch/ROCm; single card works.gotchas/gotcha-amd-mi100-multigpu-segfaults-without-vm-passthrough

verifiedmedium · 0.80

MI50 will hang or crash under sustained inference load at default clock settings; lowering the SCLK level or capping power avoids the failure and drops temperature from 100C to 82C, so cards that 'mysteriously throttle' are usually thermally or power-supply limited

[major] hardwareSustained-load failures: 'when i send requests to vllm, sometimes pc hangs. Lowering power level helps with the hang issue but reveals another problem: when you do two simultaneous requests, vllm crashes.' The fix that also survives: `rocm-smi --setsclk 4` drops the core frequenc…gotchas/comm-mi50-conditional-slowdown-under-load

verifiedmedium · 0.70

MIG isolates SMs, memory capacity, L2 and bandwidth per slice, but NVLink and the interconnect path remain shared physical links - so a MIG partition is not a bandwidth-isolated device

[major] hardwareUnder-appreciated rather than loud: there is no error and no warning. Two MIG instances on one A100/H100 both reach high SM utilization while neither can reach its nominal memory bandwidth or full NVLink line rate, because the NVLink interfaces and their bandwidth are properties …gotchas/mgpu-mig-slice-nvlink-and-memory-bandwidth-are-shared-not-isolat

verifiedhigh · 0.85

MIG isolates SMs, memory capacity, L2 and bandwidth per slice, but NVLink and the interconnect path remain shared physical links - so a MIG partition is not a bandwidth-isolated device

verifiedhigh · 0.85

MIG slices cannot participate in NCCL tensor-parallel groups

[blocker] hardwareA MIG slice cannot join a tensor-parallel group. NCCL initialization fails or hangs.gotchas/serve-mig-blocks-nccl-tensor-parallel

verifiedhigh · 0.90

MindIE 3.0.0 '300I Duo' container image on a single Atlas 300I Pro (310P3): service start hard-faults the NPU (aclnnTril fail → HDC disconnect, unrecoverable until reboot)

[major] configStarting mindieservice_daemon from the mindie:3.0.0-300I-Duo-py311-openeuler24.03-lts image on a single-card Atlas 300I Pro (Ascend 310P3, 24GB) fails with aclnnTril error 107000, then cascades into halMemAlloc failures and 'internal host device communication (HDC) session is dis…gotchas/gotcha-mindie300-duo-image-on-300i-pro-npu-fault

verifiedmedium · 0.70

Missing EOS or unhandled special token syntax triggers runaway generation and unified memory exhaustion crash on Apple Silicon

[minor] toolchainDuring inference on Apple Silicon, a request fails to terminate upon reaching the intended conclusion, continuing to emit repetitive prose, reasoning traces, or empty tokens. As generation proceeds into tens of thousands of tokens, the KV cache and working buffers expand until th…gotchas/gotcha-special-token-runaway-unified-memory-exhaustion

verifiedhigh · 0.85

Missing SYCL reorder optimization makes Q8_0 and IQ4_NL decode ~4x slower than Q4_0 on Battlemage

[major] kernelOn Intel Arc Pro B70, Q8_0 and IQ4_NL decode at a fraction of the speed of Q4-family quants despite moving comparable or less data. Full sweep on Qwen3.5-27B, one GPU, 608 GB/s of theoretical bandwidth: Q4_0 23.67 t/s (57% of peak), Q4_K_S 23.05 (56%), Q4_K_M 20.56 (53%), IQ4_XS …gotchas/sycl-quant-reorder-gap-makes-some-quants-4x-slower

verifiedhigh · 0.90

MLPerf 'MaxQ' is not a power cap - on H100-SXM it sets no watt limit at all and can disable NVLink, so MaxQ-vs-MaxP throughput gaps are not efficiency curves

[major] configThe same benchmark, same node, reports about 19% lower throughput in the power-measured submission than in the ordinary one - 25262.1 versus 31302.7 tokens/s for llama2-70b-99.9 Offline on an 8xH200 SXM node (19.3% lower), and a consistent 17.0% for mixtral-8x7b (48987.9 versus 5…gotchas/mlperf-maxq-is-not-a-watt-cap

verifiedhigh · 0.92

MLPerf Inference LLM numbers are measured on fixed-shape, closed-loop, single-model traffic with a submitter-chosen scheduler, so an MLPerf lead is not a production lead

[major] measurementA vendor wins MLPerf Inference on a model and a deployment blog quotes the win as evidence that its platform will serve a production traffic mix faster. The inference does not hold, and the gap is structural rather than a matter of degree. Concretely, what MLPerf fixes and what i…gotchas/bench-mlperf-fixed-shape-is-not-production-traffic

verifiedhigh · 0.93

MLPerf Inference v6.1 publishes zero watts and zero declared power limits for every benchmark: has_power is false on all 492 result rows and design_power_watts is null on all 492

[major] measurementAnyone who wants an MLPerf figure they can divide by watts to get cost per token discovers that in the current round there is nothing to divide by. Reading every row of the v6.1 official summary.csv: the has_power column is False for all 492 rows and the design_power_watts column…gotchas/bench-mlperf-v6-1-publishes-no-power-data-at-all

verifiedhigh · 0.97

MLPerf Interactive results are stored and published as Interactive but the harness log labels them 'Scenario : Server', so raw-log auditing silently misclassifies them

[major] measurementAn auditor reads every raw harness log in a submission, as the most defensible way to check MLPerf numbers, and concludes that every Interactive result is really a Server result. The evidence is in the logs themselves: for AMD's 8xMI355X Llama 2 70B Interactive run the log at clo…gotchas/bench-mlperf-interactive-scenario-labels-itself-server

verifiedhigh · 0.95

MLPerf LLM 'tokens per second' is generation-length times samples per second, and generation length varies ~9% between submitters at the same accuracy target, so tok/s figures are not directly comparable even within one round and scenario

[major] measurementTwo MLPerf results for the same model, same scenario, same round and even the same GPU are compared directly and the comparison is quietly wrong by several percent. The reason is that MLPerf's Tokens-per-second for LLM benchmarks is not an instrumented decode rate - it is samples…gotchas/bench-mlperf-token-count-is-not-fixed-across-submitters

verifiedhigh · 0.92

mlx-lm --kv-bits passes its startup check, then crashes on the first request for hybrid sliding-window models

[major] configmlx_lm.server starts cleanly and /health returns 200, then the first chat completion raises 'NotImplementedError: RotatingKVCache Quantization NYI'. Affects models whose attention schedule is mixed - Gemma 4 uses sliding-window attention on 35 of 42 layers in the 26B-A4B configur…gotchas/mlx-kv-bits-crashes-on-first-request-for-hybrid-attention

verifiedhigh · 0.90

MLX-LM KV cache quantization disables batching: concurrency vs memory tradeoff

[major] frameworkEnabling KV cache quantization (4-bit or 8-bit) in MLX-LM disables batching entirely, reducing the server to single-request inference. This is a hard tradeoff: save memory or serve concurrent users, not both.gotchas/local-mlx-kv-bits-disables-batching

verifiedmedium · 0.80

MoE-Lightning's ASPLOS 2025 venue survived a four-corpus sweep - a negative across ML conference indexes is not a negative

[major] toolchainVerifying MoE-Lightning (arXiv 2411.11217) on 2026-10-04 required checking four corpora: the OpenReview note index, PMLR v267 (ICML 2025), the MLSys 2025 and 2026 indexes, and the NeurIPS 2024 index. All four were negative, the arXiv comment field was empty, and the record was fi…gotchas/w6p-acl-or-crossref-index-misses-venue-a-negative-across-four-co

verifiedhigh · 0.93

MoE-Lightning's ASPLOS 2025 venue survived a four-corpus sweep - a negative across ML conference indexes is not a negative

verifiedhigh · 0.93

Mooncake's TCP fallback fails the first transfer after an idle gap, and the decode side hangs until timeout

[major] frameworkWith the TCP transport, the first KV transfer between two engine instances succeeds, then a transfer submitted after a short idle gap (~5 s) fails: on the producer every slice of the batch is rejected by the TCP lane queue (queue-full rejection count climbing 1 -> 2 -> 4 -> ... -…gotchas/pd-tcp-fallback-queue-full-hang

verifiedmedium · 0.80

Most engine histograms are per-request observations but Triton reports per-model aggregates, and prefix-cache counters are token counts not request counts - the same metric name shape hides different denominators

[minor] measurementA p99 computed from an engine histogram looks nothing like the p99 a client measured, and nobody can say which is right. A prefix-cache 'hit rate' of 0.7 turns out to mean 70% of tokens, not 70% of requests, so a workload with one huge cached prompt and many tiny misses reads as …gotchas/obs-metric-names-that-look-per-request-are-not

verifiedhigh · 0.85

Multi-GPU HIP on Windows 11 produces fluent gibberish at full speed, and benchmarks do not detect it

[blocker] driverOn Windows 11 with a gfx1100 + gfx1201 pair, the HIP backend splits a model across both cards and returns fluent-but-nonsensical word salad instead of an answer, with no error, no warning and healthy-looking throughput (llama-bench: pp ~197, tg ~26). Corrupted samples begin plaus…gotchas/windows-hip-multi-gpu-silent-output-corruption

verifiedhigh · 0.90

Multi-GPU Intel Arc under SYCL loses P2P, then crashes or hangs in device-to-device copies

[major] hardwareMulti-card Intel Arc dGPU configurations fail in the peer-copy path in three distinct ways. (1) The OpenCL runtime has no P2P at all: ur_die with 'Experimental P2P feature is not implemented for OpenCL adapter'. (2) Under --split-mode tensor on a dual Arc Pro B70 with P2P working…gotchas/sycl-multi-gpu-arc-loses-p2p-and-crashes

verifiedhigh · 0.85

Narrow but consistent: ONNX Runtime and OpenVINO are chosen for constrained and non-CUDA deployment, but the corpus treats them as frameworks you build against rather than engines you adopt

[minor] toolchainONNX is recommended in the situational map with an explicit downgrade: 'ONNX is also technically an option if you're willing to setup an inference script for a specific model, and you need to support weird hardware (like a Rockchip NPU), but that's not really an LLM engine so muc…gotchas/sent-onnx-and-openvino-for-constrained-deployment

verifiedmedium · 0.70

Native NVFP4 compute exists only on sm100/103 and sm120/121 with CUDA 12.8+: every other NVIDIA part, including Hopper, has no NVFP4 GEMM at all and the engine raises NOT_IMPLEMENTED

[major] hardwareAn NVFP4 checkpoint on an H100, an H200, an A100, an L40S or a 4090 either (a) loads and runs with no tensor-core FP4 benefit, the FP4 weights dequantized and the FP4 activations upcast, so the memory saving appears and the compute speedup does not, or (b) fails with STD_TORCH_CH…gotchas/w4q-nvfp4-native-only-sm100-sm120-cuda128

verifiedhigh · 0.93

NCCL bus bandwidth is typically 2x achieved bandwidth for inter-node collectives

[major] measurementNCCL all-reduce performance over InfiniBand achieves only ~50-60% of the theoretical bus bandwidth, and this gap is rarely stated explicitly.gotchas/nic-nccl-bus-bandwidth-vs-achieved

verifiedmedium · 0.80

NCCL is not supported with MIG, so a MIG slice cannot participate in a tensor-parallel group - MIG and TP are mutually exclusive as shipped

[blocker] hardwarePlanning error rather than a runtime crash: an operator who partitions an H100 into MIG instances and then tries to run TP=2 inside them gets NCCL unsupported-with-MIG rather than a working deployment. The NVIDIA MIG User Guide lists it flatly under Application Considerations: 'N…gotchas/mgpu-mig-blocks-nccl-tensor-parallel

verifiedhigh · 0.90

NCCL's own mismatch guards convert a detectable multi-node configuration error into an unbounded hang, because suppressing them is the documented escape hatch

[blocker] driver2 nodes x 8x H20 over RoCE/eRDMA, NCCL 2.30.7. Node A exposes erdma_0 and erdma_1; node B has erdma_1 present in the container but not openable (ibv_open_device fails - the device plugin admits only one uverbs device). With NCCL_IGNORE_NET_MISMATCH=1 set to silence the mismatch w…gotchas/mgpu-partial-node-failure-deadlocks-instead-of-erroring

verifiedhigh · 0.88

Newegg refurbished vs eBay/peer used price spread on consumer GPUs

[major] operationsRepo holds two very different 'used' prices for the same card: e.g. RTX 3090 ~$950 on eBay (ebay-used-rtx3090-oct2026) vs ~$2,399 on Newegg refurb (used-rtx-3090-market); RX 7900 XTX ~$875 peer vs $1,405 Newegg.gotchas/gotcha-newegg-refurb-vs-ebay-used-2x-price-spread

verifiedmedium · 0.80

Newer NVIDIA consumer architectures are not supersets: SM100 supports FP8 block-scaled GEMM and SM120 does not, because SM90/100/120 have entirely different FP8 instruction sets

[major] hardwareA user with 8x RTX 6000 PRO (768GB VRAM total) trying to serve original FP8 weights gets the flat refusal `Unsupported SM version for FP8 block scaling GEMM`. The FP4 checkpoint does load, but at 27 tok/s for batch 1 with ~37B active parameters - the reporter expected an order of…gotchas/comm-sm120-fp8-arch-split-expert-account

verifiedhigh · 0.85

No vendor publishes joules-per-token for inference: the only way to get it is to measure it yourself or estimate from TDP and throughput, and the estimation error is 30-50% because TDP is a thermal ceiling not a power measurement and throughput depends on batch size, sequence length and model

[major] measurementA joules-per-token figure is needed for a cost model, a carbon calculation, or a capacity plan. The vendor datasheet does not contain it. The only available figures are TDP (a thermal design power ceiling, not a measured power draw) and peak throughput (a best-case number at maxi…gotchas/vendors-dont-publish-joules-per-token-estimation-error

verifiedhigh · 0.85

NVFP4 fused-MoE loading detects mismatched gate/up global scales, warns once, and then dequantizes the up half with the gate half's scale anyway

[major] formatModelOptNvFp4FusedMoE.process_weights_after_loading fuses gate_proj and up_proj into w13 and keeps only the gate per-tensor global scale. When a checkpoint ships different scale_2 for the two, the up half of w13 is dequantized with the gate half's global scale - a silently mis-sc…gotchas/nvfp4-fused-moe-w13-global-scale-mismatch

verifiedhigh · 0.90

NVFP4 models with --dtype bfloat16 produce garbled output on any GPU without native FP4

[blocker] formatvllm serve of any NVFP4 model with --dtype bfloat16 on SM < 100 (e.g. RTX 4090 SM89, V100 SM70) loads, starts and serves, but every response is garbled. Measured on GLM-4.7-Flash-NVFP4: 54.7% of output values are zero and cos_sim is 0.22 against an FP32 reference; GSM8K 5-shot is…gotchas/nvfp4-marlin-bf16-garbled-output

verifiedhigh · 0.95

NVIDIA publishes a per-LINK NVLink bandwidth figure for exactly one generation (NVLink 1) and none for the other five, so the per-link column in this family is arithmetic on two other numbers for Gen2-6 and looks equally well-sourced in every row

[major] hardwareSix records each carrying a bandwidth_basis that opens with a confident 'PER LINK' assertion, and the reader cannot tell which of them NVIDIA printed and which were computed - because all six are formatted identically. nvlink-1 states '20' as a transcription ('NOW FIRST-PARTY AND…gotchas/w7n-nvlink-per-link-rate-is-derived-not-published-except-gen1

verifiedhigh · 0.88

NVIDIA RTX PRO 6000 Blackwell Max-Q cited in benchmarks lacks first-party datasheet for dedicated record

[minor] hardwareField benchmarks and production reports for DeepSeek-V4.1-Flash (such as data/benchmarks/ev-sglang-dspark-spec-decoding-2x-decode-ab.json, ev-sglang-dsv41-production-128k-prefill-tps.json, ev-sglang-dsv41-production-decode-tok-s-per-user.json, and ev-sglang-dsv41-prefix-cache-hit…gotchas/gap-nvidia-rtx-pro-6000-blackwell-max-q

verifiedhigh · 0.95

NVIDIA's published NVLink figure is a bidirectional per-GPU aggregate, so every per-link or per-direction number in this family is a DERIVATION - reading the vendor number as per-direction doubles it and reading it as per-link multiplies it by the link count

[major] hardwareTWO independent axes of error on the SAME family of numbers, differing by exactly 2x and by exactly link_count respectively, and the vendor page supports a correct reading and each of two wrong readings equally well. AXIS 1 (direction): NVIDIA's spec table row is 'NVLink Bandwidt…gotchas/w7n-nvlink-per-gpu-aggregate-vs-per-link-rate

verifiedhigh · 0.90

Offline is NOT reliably faster than Server: on identical MLPerf LLM hardware the ordering reverses sign across scale, from -6% on one node to +1.9% on a 72-GPU rack

[major] measurementThe rule 'the Offline scenario produces higher throughput than Server, so quote Offline' is repeated constantly in vendor comparison posts and it is false as a general statement. Measured Offline-minus-Server on identical hardware, identical model, identical quantization, within …gotchas/bench-mlperf-offline-is-not-always-faster-than-server

verifiedhigh · 0.95

On a streaming endpoint an output filter that inspects completed text cannot un-send the tokens it already streamed

[major] securityYou add a moderation check on the assistant turn. The compliance test passes because the test client is non-streaming. In production the first sentence of the offending content is already delivered before the check has completed, and there is no way to take it back over the strea…gotchas/sec-streaming-output-scan-cannot-unretract

verifiedmedium · 0.80

On AMD Strix Halo gfx1151, dense 72B models collapse to 4 tok/s due to 256 GB/s LPDDR5X bandwidth while 120B MoE achieves 49 tok/s

[minor] hardwareRunning large dense models on AMD Strix Halo (Radeon 8060S, 128 GB unified memory) yields low token generation speed despite full GPU offload: Qwen 2.5 72B Q5_K_M runs at only 4.1 tok/s (Q6_K at 3.7 tok/s). However, running GPT-OSS 120B (an MoE model with MXFP4 quantization) on t…gotchas/amd-strix-halo-dense-vs-moe-throughput-gap

verifiedhigh · 0.90

On consumer Blackwell (RTX 50-series) vLLM had no prebuilt wheel, and the community recipe is build-from-source against the NGC PyTorch container - with FlashAttention 3 disabled and FP8 unavailable

[major] buildThe community answer to 'how do I run vLLM on my 5090' was not a flag. It was: clone vLLM at or above commit ed6ea06577ec06f0b3a9ac921b55ef254f19d923, start nvcr.io/nvidia/pytorch:25.02-py3, run `python use_existing_torch.py`, `pip install -r requirements/build.txt`, then `python…gotchas/comm-blackwell-consumer-vllm-no-prebuilt-wheel

verifiedhigh · 0.90

On dual consumer RX 7900 XT, tensor parallelism yields only 1.11x-1.23x decode scaling due to KV-head starvation while achieving 1.56x-1.87x prefill scaling

[major] frameworkAdding a second consumer RX 7900 GPU under vLLM TP=2 yields only 1.11x to 1.23x speedup on batch-1 token generation across context lengths from 500 to 32K tokens on dense and MoE models (e.g. 1.13x-1.23x on gemma-4-12B, 1.11x-1.12x on gemma-4-26B-A4B). In contrast, prompt process…gotchas/amd-rdna3-tensor-parallel-decode-scaling-bottleneck

verifiedhigh · 0.85

On gfx1151 the llama.cpp HIP backend produces corrupt output where Vulkan is correct with byte-identical flags, so cross-backend comparison is the only clean detector

[major] kernelOn a gfx1151 Strix Halo APU (ROCm 7.2.4), same machine, same build, byte-identical server flags (verified with diff), the HIP backend is wrong and Vulkan is right. Two severities: Llama-3.1-8B-Instruct Q4_K_M (llama, dense) is broken outright at every depth tested - 0/16 at 30,51…gotchas/hip-rocm-corrupt-output-vulkan-is-reference

verifiedmedium · 0.80

On gfx12 (RDNA4) consumer silicon RCCL selects the LL protocol for small collectives even though the LL primitives were never given the gfx12 memory-ordering fixes, deadlocking multi-GPU inference

[blocker] kernelvLLM TP=2 inference on dual Radeon R9700 (gfx1201) deadlocks outright. In vLLM terms (issue 40980): 'Tensor Parallel = 2 across two AMD Radeon R9700 (gfx1201) GPUs results in a complete deadlock. The server starts, routes are registered, but both GPUs immediately sit at 100% util…gotchas/mgpu-rccl-ll-protocol-deadlocks-gfx12

verifiedhigh · 0.88

On gfx1201 the Vulkan backend loses 4.7-6.7x decode throughput to HIP at hidden_size >= 4096, and effective bandwidth collapses to ~70-100 GB/s on a 640 GB/s card

[major] kernelllama-bench on an RX 9070 XT 16 GB (gfx1201, RDNA4, Windows 11, AMD proprietary driver 25.20.42.14) at -ngl 99: Qwen3-4B Q4_K_M (hidden 2560) gives tg 183.0 t/s (~424 GB/s effective); Qwen3-8B Q4_K_M (hidden 4096) gives tg 21.0 (~98 GB/s); Qwen3-8B Q8_0 (hidden 4096) tg 9.5 (~77 …gotchas/vulkan-decode-cliff-hidden-size-4096

verifiedhigh · 0.85

On gfx906 vLLM, AWQ quants fail under tensor parallelism while GPTQ works, and the cause is structural: AWQ packs group scales in a way that cannot be split across TP ranks

[major] frameworkEarly on the nzly fork, 'AWQ quants dont seem to work with -tp 4. I think it has to do with how the model is split across cards, and quants groups value together for compression, so they are less flexible when it comes to splitting up the underlying matrix.' The same member later…gotchas/comm-gfx906-awq-tensor-parallel-breaks-gptq-works

verifiedmedium · 0.65

On gfx906, vLLM containers die with a bare 'Bus error (core dumped)' unless --shm-size is set large, and a corrupted local Docker BASE layer can make every ROCm 6.3.3 image fail identically - neither failure is a hardware fault

[blocker] config'mixa3607/vllm-gfx906:0.10.2-rocm-6.3.3 always dies instantly without --shm-size with Bus error; with --shm-size tries to start'. Running `vllm` inside a correctly-started container yields 'Bus error (core dumped)' with no further diagnostics. The same member chased this across i…gotchas/comm-gfx906-docker-shm-size-and-base-image

verifiedmedium · 0.70

On GPUs in NVIDIA Confidential Computing mode, vLLM's V2 model runner serves silent garbage

[blocker] frameworkOn an H100 NVL in CC mode inside a TDX guest, vLLM 0.29.0 with the default V2 GPU model runner starts and serves, but every response is garbage - '!!!!!!!!' (token id 0 repeated) or '[][][][]'. No error or warning anywhere. The same model is correct under plain transformers on th…gotchas/cc-mode-uva-view-garbage-output

verifiedhigh · 0.90

On Intel Arc dGPUs, llama.cpp agentic workloads stall and enabling FA can cut prompt processing ~15x

[major] hardwareOn a Ryzen 5600g with Intel Arc Pro B50 (Battlemage), agentic coding with Devstral 2 small 24b or GPT-OSS 120b stalls in the first few agent steps; prompt processing drops to single digits and effectively stops once context reaches 4k-8k tokens. Measured without FA: ~300 tok/s pp…gotchas/intel-dgpu-vulkan-or-sycl-agentic-stall

verifiedmedium · 0.75

On MI50 the PCIe slot topology dominates: cards split across two NUMA nodes get 0 in the P2P bandwidth matrix, and PCIe 3.0 x8 links cost 50-100% of pipeline-parallel throughput versus PCIe 4.0 x16

[major] configA member with 2 cards on cpu0 and 2 on cpu1 on an X99 board reports rocm-bandwidth-test Inter-Device Access with 'some 0s in the matrix' - and explicitly that neither a 32 GB BAR under the V420 BIOS nor the full mitigation kernel line ('no iommu OR iommu=pt mitigations=off pcie_a…gotchas/comm-gfx906-numa-topology-and-pcie-lane-count

verifiedmedium · 0.65

On PCIe-only GPUs that sit across NUMA nodes, TP=2 produces garbled model output through the custom all-reduce path, and the corruption is silent

[major] framework2x NVIDIA L20 (compute capability 8.9, no NVLink) where nvidia-smi topo -m reports SYS between GPU0 and GPU1 (connection traversing PCIe plus the SMP interconnect between NUMA nodes): with '--tp 2', 'The model outputs garbled text. After I add --disable-custom-all-reduce, it outp…gotchas/mgpu-pcie-cross-numa-tp-garbles-output

verifiedmedium · 0.80

On RDNA, models with head_dim != 128 silently fall back to Triton paged attention and decode collapses at long context

[major] kernelQwen3.6-27B on gfx1100 decodes at 12.1 tok/s at 518 tokens of context but 4.2 tok/s at 32K. Linear-attention layers keep constant cost; only the 16 full-attention layers degrade. Profiling puts essentially all the growth in kernel_paged_attention_2d, which grows 28.3x from 356.66…gotchas/rdna-triton-paged-attn-decode-cliff

verifiedhigh · 0.90

On ROCm, any KV cache type other than q8_0/q8_0 or q4_0/q4_0 explodes the graph into 34 splits

[major] kernelWith flash attention on and a q5_0 KV cache (either K or V), the ROCm backend fragments the compute graph from 2 splits into 34 and pushes about 550 MiB of compute work onto the CPU host buffer. Measured on an RX 7900 XTX with Qwen3.8-27B Q4_K_XL: pp512 falls from 957.11 t/s to 3…gotchas/rocm-flash-attention-kv-type-graph-split-explosion

verifiedhigh · 0.85

On single-node PCIe boxes NCCL's cuMem UDS file-descriptor exchange loses a message and the engine hangs at startup in a spin-wait, intermittently, with no error

[major] driverOn a single-node 8x RTX 3090 box (no NVLink, P2P over PCIe at PXB level, NCCL 2.29.7+cuda13.2) serving a large MoE under vLLM with TP2xPP4 plus expert parallel - a layout that creates on the order of fifty communicators in quick succession at engine start - roughly one boot in se…gotchas/mgpu-nccl-cumem-uds-fd-exchange-hangs-at-init

verifiedhigh · 0.88

On some AMD boards Vulkan dies with ErrorDeviceLost only when SAM/ReBAR is enabled

[blocker] driverThe opposite failure of the previous record, and it is a hard crash rather than a slowdown: with SAM/ReBAR enabled, every decode on an AMD Radeon AI PRO R9700 fails with vk::Queue::submit: ErrorDeviceLost, llama-server returns HTTP 500 and exits. Disabling ReBAR in BIOS makes the…gotchas/vulkan-rebar-on-causes-amd-device-lost

verifiedmedium · 0.80

On some gfx1201 systems the ROCm backend allocates no VRAM and streams a 30 GB model from system RAM

[major] driverOn an 8x RADEON AI Pro 9700 XT (gfx1201) host, llama.cpp's ROCm backend reports the full VRAM size in --list-devices, shows both GPUs at normal compute activity, and then allocates nothing on the GPU: nvtop shows zero VRAM in use and a 30 GB model at 256k context consumes over 25…gotchas/rocm-backend-silently-serves-weights-from-host-ram-on-gfx1201

verifiedmedium · 0.80

On Vulkan, enabling Flash Attention can cut prompt processing by ~60% and decode by ~48%

[major] kernelOn Strix Halo (RADV, warp size 64) llama-bench shows FA on costing more than it saves: llama-2-7b.Q4_0 pp512 drops 1252.63 -> 508.36 tok/s and tg128 drops 49.91 -> 25.80 tok/s. Qwen3.5-122B-A10B-GGUF Q4_K_XL-MTP drops pp512 325.58 -> 142.33 tok/s and tg128 22.42 -> 7.x tok/s.gotchas/vulkan-flash-attn-slower

verifiedmedium · 0.70

On-wafer/on-chip SRAM filed in vram_gb and memory_bandwidth_gbps is a false friend, and a prose warning does not fix it

[major] hardwareA query, join or sort over the accelerator corpus returns a number for a non-GPU part that is arithmetically right and semantically wrong. Concretely, three records in this repo held on-chip SRAM figures in the two fields that every GPU record uses to mean off-package memory capa…gotchas/w7z-onchip-sram-filed-as-vram-and-bandwidth

verifiedhigh · 0.90

One dead rank does not stop new KV loads to its siblings: per-port peer identity means a 30 s timeout is the only terminator

[major] frameworkOn a disaggregated P/D deployment with multi-rank prefill pods, a source pod became unavailable mid-transfer. The consumer marked the source's rank-0 control peer down within a second - and then submitted a NEW load to a sibling rank of the same dead pod, which ran the full load …gotchas/pd-p2p-peer-liveness-per-rank-not-per-pod

verifiedhigh · 0.90

One GPU, one provider, one page, several prices - and the record keeps the dearest one

[major] measurementA supply record carries a per-GPU-hour figure that is arithmetically correct for its own product and nevertheless reads as 'what this GPU costs here'. Together AI's pricing page publishes THREE tables with three different rates for the same NVIDIA H100 GPU-hour on the same URL, a…gotchas/w5s-one-page-one-product-many-tiers

verifiedhigh · 0.90

One tenant's long context evicts every other tenant's KV cache: a single request can consume more KV than hundreds of short ones, so a quota on request count does not protect anyone

[blocker] operationsA small number of users sending long-context requests degrades the service for everyone: their p50 latency is fine while other tenants' p50 and p99 climb sharply, and the other tenants' requests are the ones being preempted and re-prefilled (see ops-kv-cache-full-preempts-and-rec…gotchas/ops-noisy-neighbor-long-context-evicts-kv

verifiedhigh · 0.90

OpenTelemetry support is real in four of eight engines and absent in the rest; a trace of one request is one span in vLLM and three spans in Triton, and neither is per-token

[minor] measurementYou enable OTel on four engines and get nothing on the other four, with no warning. Or you get a trace and expect a span per token or per prefill/decode step and find one span for the whole request, so the trace cannot answer 'which phase was slow'. Or you compare a vLLM trace wi…gotchas/obs-tracing-status-per-engine-honest-negatives

verifiedhigh · 0.90

Output filtering at serve time: logit masking, post-hoc scanning and a separate guard model are three different systems with three different bills

[major] securityAn engineer asks 'add content filtering' and gets three mutually exclusive answers depending on who is asked. Logit masking inside the decode loop costs throughput; scanning completed text costs nothing until you notice you already streamed it; a separate guard model costs a seco…gotchas/sec-output-filter-three-placement-cost

verifiedhigh · 0.85

Owned-hardware TCO is computable and electricity is only 8-20% of it over three years; rent-vs-own break-even is NOT computable, because no accelerator in this corpus has both a priced rental row and a local decode benchmark

[major] measurementAsked for a rent-vs-own break-even, the corpus appears to have everything needed: prices for owned hardware in data/supply/, TDP for almost every accelerator, a kWh rate, and 221 priced cloud rows. It cannot produce a break-even, because the two sides of the comparison never touc…gotchas/value-tco-and-why-rent-vs-own-cannot-be-computed-from-this-corpu

verifiedhigh · 0.90

Passive Tesla cards use EPS-12V CPU power, not PCIe 8-pin

[blocker] hardwareP40/M40/P100 passive cards have an EPS-12V 8-pin power connector; a standard PCIe 8-pin lead does not fit and adapters are easy to get wrong.gotchas/gotcha-pascal-tesla-eps12v-not-pcie-power

verifiedmedium · 0.80

Passive-OAM modules and rack-scale systems physically cannot run air-cooled: no heatsink means no air path, and the host must have a cold plate

[blocker] hardwareA server built for air cooling will not accept these parts at all, and the failure is a procurement/integration one rather than a runtime one: the chassis has no cold plates, no manifold inlets, no coolant fittings and no CDU, so there is nothing for a passive module to attach to…gotchas/passive-oam-and-rack-scale-parts-cannot-run-air-cooled

verifiedhigh · 0.93

PCI DSS requirement text was not retrieved and no PCI-specific requirement is recorded here - recorded as a refusal, not a gap

[minor] complianceA reader expects this slice to state which PCI DSS requirements apply to prompt and completion handling. It does not, because the standard text was not obtainable.gotchas/sec-pci-dss-requirement-text-not-retrieved

verifiedhigh · 0.95

PCIe GT/s rate is not proportional to usable bandwidth across the PAM4 boundary: Gen5->Gen6 payload per direction rises only 1.92x, not the 2x the GT/s doubling implies, so a GT/s-based comparison concludes PCIe 6.0 is slower than it is

[major] hardwareComparing two PCIe generations by their GT/s rate gives the wrong answer, in both directions depending on which pair is compared. Concretely, for this repo's seven generations, payload per direction at x16 is 4.0 / 8.0 / 15.75 / 31.5 / 63.0 / 121.0 / 242.0 GB/s while the GT/s lad…gotchas/w7p-gotcha-pcie-gt-s-hides-pam4-payload-step

verifiedhigh · 0.90

Per-query anomaly scoring does not catch extraction; distribution testing over a traffic window does, with published numbers

[minor] securityA fraud/extraction monitor scores each query against a benign distribution and raises almost nothing, while an attacker steadily distilling the model walks away with a clone. The single queries really do look benign - that is the premise of the attack, not a bug in the monitor.gotchas/sec-model-extraction-detectable-at-traffic-level-not-per-query

verifiedhigh · 0.88

Pi AI HAT+ 2 (Hailo-10H) cannot run 7B+ models despite 'GenAI' marketing and 8GB

[blocker] hardwareThe $200 AI HAT+ 2 is sold as a GenAI accelerator with 8GB onboard RAM, but the actual compiled-model set tops out at <=2B: Llama-3.2-1B 9.9 TPS, Qwen2.5-1.5B 7.35, Qwen3-1.7B 4.78.gotchas/gotcha-hailo-pi-ai-hat-2-capped-at-2b-models

verifiedmedium · 0.80

Preemption costs no visible throughput and only shows up in latency, so a throughput dashboard stays green through a KV-cache collapse

[major] measurementAggregate output tok/s and request-success rate look normal or even improve while users report stalling. Inter-token latency and time-to-first-token p99 climb. Nothing in the throughput panel moves, and on an engine without a preemption metric there is no signal at all. The givea…gotchas/obs-preemption-hides-latitude-from-throughput

verifiedhigh · 0.90

Prefetching more than one predicted expert, and quoting a full-layer-prefetch baseline, both misstate the MoE-CORE decode result

[major] measurementA hand-written expert-offload decode loop that prefetches the top three predicted experts of the next layer is slower than the same loop prefetching one, and a reported 30x decode speedup against 'vLLM prefetch' does not survive contact with a load-on-demand baseline at the same …gotchas/moe-core-prefetch-depth-and-full-layer-baseline

verifiedmedium · 0.80

Prefix cache blocks computed under one LoRA adapter are reused under another when the adapters share a name

[major] configPrefix cache corruption when two LoRA adapters share a name but differ in lora_init_id: blocks computed under one adapter are served for requests targeting the other, producing wrong output with no error. Confirmed as offline/batch mode only - maintainer jeejeelee states it 'shou…gotchas/pd-lora-adapter-swap-reuses-stale-prefix-blocks

verifiedhigh · 0.85

Prefix caching leaks other tenants' prompts: prefix reuse across tenants is a timing side channel distinguishable at AUC 0.99 from only 8 tokens of guessed prefix

[blocker] privacyAn attacker sharing a backend can determine whether another tenant's prompt prefix is resident in the KV cache purely by measuring time-to-first-token. The signal is strong enough to be usable as an oracle — the security advisory reports ROC AUC of 0.99 at prefix lengths of just …gotchas/ops-prefix-cache-is-a-cross-tenant-side-channel

verifiedhigh · 0.95

Prefix-aware routing scores the first request of a session on a stale QPS average, then latches that decision for the life of the session - so the busiest backend gets picked first and keeps every subsequent turn

[major] frameworkHalf the fleet goes idle while half the sessions queue, and nothing in the system says so. Two vLLM backends of equal capacity, prefixaware routing, eight concurrent agent sessions opened in a burst: the measured result was 4 running plus 4 queued on one backend while the other s…gotchas/dep-prefixaware-subthreshold-latches-a-bad-first-pick

verifiedhigh · 0.90

Priced consumer accelerators with no decode benchmark: two were sourced on 2026-10-11 (RX 9070 XT, Arc B580) and five remain unsourced, with the exact sources checked

[major] measurementValue research found sixteen accelerators that carry a purchase price but no local decode benchmark, which makes them unknowns rather than picks. For the consumer cards a buyer can actually acquire (RTX 3090, 3090 Ti, 4090, 4080, 5070 Ti, RX 9070 XT, Arc B580), the corpus could n…gotchas/local-decode-benchmarks-for-the-priced-consumer-cards-what-exist

verifiedmedium · 0.75

Priced consumer accelerators with no decode benchmark: two were sourced on 2026-10-11 (RX 9070 XT, Arc B580) and five remain unsourced, with the exact sources checked

verifiedmedium · 0.75

Production sparse attention (NSA, MoBA) is strictly gated on native model architectures and lacks 128K serving throughput benchmarks

[major] frameworkOperators expecting 5x-10x throughput gains on arbitrary dense 128K-context models by enabling sparse attention find that production engines (vLLM, SGLang) cannot enable NSA or MoBA on standard dense checkpoints (e.g. Llama-3.1, Qwen2.5), and that neither paper publishes multi-te…gotchas/sparse-attention-128k-production-gating

verifiedhigh · 0.90

Prompt-injection detectors are not robust to adaptive attackers - a universal optimised suffix fools all layer-wise detectors at once

[major] securityAn injection detector works in evaluation and fails in the wild, and the failure mode is worse than an occasional miss: the attacker optimises against the detector specifically and defeats every layer simultaneously while keeping the injection fully effective.gotchas/sec-injection-detectors-fail-to-adaptive-attacker

verifiedhigh · 0.85

Putting more than one rank on the same GPU is unsupported and produces either 'Duplicate GPU detected' with invalid usage, or a hang

[blocker] configTwo ranks bound to the same CUDA device inside one communicator produce one of two signatures. With NCCL_CHECKS_DISABLE=1 the observed output is 'Failed, NCCL error a.cpp:106 \'invalid usage\'' from ncclCommInitRank. With checks enabled the warning appears first: 'NCCL WARN Dupli…gotchas/mgpu-multiple-ranks-per-gpu-requires-mps

verifiedhigh · 0.85

Q: 'Is it worth upgrading my GPU?' - A: the corpus answers a different question - buy VRAM capacity, not compute - and declines to give a generational verdict

[minor] hardwareNobody in this corpus answers 'upgrade or not' with a generational recommendation. What they answer instead is a capacity test. A 2024 thread predicting post-5090 price moves states the capacity framing in the clearest form available, 54 upvotes: 'For LLMs I don't see any point w…gotchas/sent-qa-is-it-worth-upgrading-a-card

verifiedmedium · 0.70

Q: 'Should I move off Ollama to vLLM/SGLang?' - A: the corpus gives a numeric trigger of about five concurrent users, and names the two situations where you should not move at all

[minor] toolchainThe 'Best LLM Inference engine for today?' thread is literally this question - a user migrating off Ollama for a personal assistant - and the corpus answer is engine-by-situation rather than migrate. The concrete trigger appears in a real deployment writeup: 'Ollama is good enoug…gotchas/sent-qa-ollama-vs-real-engine-migration

verifiedmedium · 0.70

Q: 'Which engine for structured output / agentic / multi-turn workloads?' - A: SGLang and vLLM, with the reason stated as prefix reuse, and llama.cpp kept for the local-model reference path

[minor] toolchainFor prefix-heavy and structured work the corpus names vLLM's prefix caching as the specific asset: 'Prefix caching is amazing for best-of-n style generative tasks.' For large-context single-box work on current NVIDIA hardware the advice is to leave a simple local stack for a serv…gotchas/sent-qa-which-engine-for-structured-and-reasoning-workloads

verifiedmedium · 0.70

Q: 'Which GPU should I buy for local LLMs?' - A: stack used RTX 3090s, the consensus answer, on VRAM per dollar rather than on benchmark rank

[minor] hardwareAsked directly in a thread about getting above 200B parameters on a $10k budget (79 upvotes, 185 comments), the highest-scoring answer in the entire corpus (85 upvotes) is a used-3090 recommendation with the reasoning attached: 'Stack 3090s, 4090s, however cheap you can get them.…gotchas/sent-qa-which-gpu-should-i-buy-local

verifiedmedium · 0.70

Qualcomm Cloud AI 100 runs LLMs, but only via ahead-of-time compile - not load-a-GGUF

[major] toolchainA used Qualcomm Cloud AI 100 PCIe card (~$200-600, 32GB LPDDR4x) looks like cheap 8B-capable silicon, and it technically is - but only through the QPC compile-and-serve pipeline: quic/efficient-transformers converts an HF checkpoint to a Qualcomm Package (QPC) with AWQ-INT4/mxfp6…gotchas/gotcha-qualcomm-ai100-compile-ahead-not-gguf

verifiedmedium · 0.80

Quantization bit-width and format family names cited as format records

[minor] formatFour records in data/benchmarks/ cite format_id '2bit' or 'mlx-quantized-4bit', producing 4 dangling reference warnings under tools/validate.py.gotchas/gap-quantization-family-format-identifiers

verifiedhigh · 0.95

Quoting a per-token API price without naming the tier and the context band is meaningless - the same model and the same tokens differ by up to 8x on one provider's own page

[major] measurementYou build a cost comparison of hosted inference APIs from per-million-token prices and get a stable-looking ranking. It is an artefact of which page section you scraped. On OpenAI's page the SAME model gpt-6.1-sol short-context input reads $2.00 Standard, $1.00 Batch, $1.00 Flex,…gotchas/rent-token-rate-without-a-tier-is-meaningless

verifiedhigh · 0.90

Rack power limits (32kW air-cooled, 48kW liquid-cooled) determine whether a big die can be deployed at all: GB200 NVL72 draws ~120kW and GB300 NVL72 up to 140kW, requiring liquid cooling and a facility retrofit with 12-24 month lead times

[blocker] hardwareA procurement plan assumes a new GPU rack can be dropped into an existing colocation facility. The plan fails at the facility integration stage: the rack draws 120-140 kW, the facility delivers 8-32 kW per rack, the chassis has no cold plates, and the building has no coolant dist…gotchas/liquid-cooling-retrofit-lead-time-and-rack-power-limits

verifiedhigh · 0.90

Raspberry Pi AI HAT+ 2 (Hailo-10H) tops out at <=2B models despite the 'GenAI' badge and 8GB

[major] hardwareA Pi AI HAT+ 2 bought to run a 7B-class model cannot - the compiled-model set for the Hailo-10H tops out at <=2B parameters (Llama-3.2-1B at 9.89 TPS is the best-supported row). The 'GenAI' branding and 8GB onboard RAM suggest more headroom than exists.gotchas/gotcha-pi-ai-hat2-cannot-run-7b

verifiedhigh · 0.85

RDNA4 gfx1201 (RX 9070 XT) requires ROCm 7.x and is unsupported on ROCm 6.x stacks, requiring Vulkan backend for stable inference

[major] toolchainAttempting to run LLM inference engines (llama.cpp, Ollama, vLLM) on AMD Radeon RX 9070 XT (gfx1201, Navi 48) fails on standard Linux distributions and ROCm 6.x packages with missing rocBLAS libraries ('No TensileLibrary for GPU arch: gfx1201') or device initialization errors. St…gotchas/amd-rdna4-gfx1201-rocm-version-barrier

verifiedhigh · 0.85

Reproducibility: most published inference performance numbers lack the methodology needed to reproduce them

[major] measurementA paper or blog post reports 'X tokens/sec' but doesn't state batch size, input/output length, warmup procedure, clock settings, or kernel version. The number is impossible to reproduce or compare.gotchas/meth-reproducibility-missing-methodology

verifiedmedium · 0.80

Request-count rate limiting is the wrong unit for LLM serving: one 8192-token prompt is 16x the prefill work of a 512-token one, and a request counter rates them identically

[blocker] operationsA per-tenant RPS quota that looks correctly sized lets one tenant saturate the fleet with a modest request rate of very long prompts, while a tenant sending many short requests never approaches its quota despite consuming comparable compute. Autoscaling driven by request counts u…gotchas/ops-ratelimit-token-vs-request-count

verifiedhigh · 0.90

Reseller and procurement channels publish availability and a 'Get a Quote' form instead of a price - the reseller gap in the supply layer is structural, and no amount of re-checking a live page will close it

[minor] measurementTen supply/supply-side source records sit at confidence 0.5-0.6, and every one of them has already been fetched and read. sup-used-itcreations-gpu, sup-used-compucycle and sup-used-equipnet-auctions each carry a full 2026-10-03 description of what the channel publishes - IT Creat…gotchas/w10-supply-channels-publish-availability-not-price-and-the-schem

verifiedhigh · 0.90

Returning logprobs turns your endpoint into a training-data oracle: top-k logits from under 10,000 queries are enough to reconstruct the output projection and distil a clone

[blocker] securityAn open-weights model is served with logprobs enabled because a product feature needs them, behind an API key with rate limits. The rate limit bounds requests per minute. It does not bound what is learned across the requests that are allowed, and the weights behind the key were p…gotchas/sec-logprobs-on-your-api-are-an-extraction-surface

verifiedhigh · 0.90

RK3588 RKLLM is a trap for 7B+ models — ChatGLM3-6B already down to ~5 tok/s

[major] hardwareAn RK3588 SBC ($80-200) bought to run a 7B-class open model technically loads one but is unusable: the vendor's own RKLLM w8a8 benchmark table shows ChatGLM3-6B — the largest model they even benchmark — already down to 4.98 tok/s decode, so a real 7B/8B lands ~3-4 tok/s.gotchas/gotcha-rk3588-rkllm-trap-for-7b

verifiedmedium · 0.80

RoCEv2 PFC-based lossless Ethernet suffers incast collapse at GPU scale

[blocker] hardwareRoCEv2 networks using PFC (Priority-based Flow Control) for lossless operation experience throughput collapse and latency spikes when GPU count exceeds a threshold, typically around 1000-2000 GPUs in a single network domain.gotchas/nic-roce-pfc-incast-collapse

verifiedhigh · 0.85

ROCm 6.4.0+ ships no rocBLAS TensileLibrary data file for gfx906, so every BLAS GEMM fails with 'Illegal seek for GPU arch' until a 6.4 rocBLAS file is copied in by hand

[blocker] buildAfter installing ROCm 6.4.x or 7.x on gfx906 hardware (MI50/MI60/Radeon VII/Pro VII), the first BLAS call raises: rocBLAS error: Cannot read /opt/rocm-6.4.0/lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx906 List of available TensileLibrar…gotchas/comm-gfx906-rocblas-tensile-missing-from-6-4

verifiedhigh · 0.90

ROCm LLVM gfx instruction set targets cited as hardware accelerators

[minor] hardwareSix records in data/flops/ (amd-cdna3-mfma-precision-support, amd-cdna4-infinity-fabric-change, amd-cdna4-mfma-precision-support, amd-fp4-mxfp4-support, amd-mi300x-infinity-fabric-topology, amd-rocm-failure-modes) cite affected_by_hardware 'gfx942', 'gfx950', or 'gfx1151', genera…gotchas/gap-rocm-llvm-gfx-target-architectures

verifiedhigh · 0.95

ROCr runtime refuses kernel dispatches declaring hostcall buffers when PCIe root port lacks AtomicOps, causing all RCCL collectives to fail

[blocker] driverWhen initializing multi-GPU inference with RCCL/vLLM on consumer motherboards or virtualized setups (VFIO/KVM), all RCCL collective operations fail at kernel dispatch. The ROCr runtime refuses any kernel execution that declares a hostcall buffer.gotchas/rocm-rccl-pcie-atomicops-requirement

verifiedhigh · 0.85

Scale-down is bounded by pod warmup, not by demand: a loaded model takes minutes to become ready, so the load signal stays high after traffic is gone and the fleet cannot shrink

[blocker] operationsTraffic drops to near zero, the demand signal falls, and the replica count does not come down — or comes down and immediately goes back up, and the newly added replicas are not serving for minutes. The autoscaler's own view says the load is gone while the fleet is stuck at its hi…gotchas/ops-scale-down-never-reaches-idle

verifiedhigh · 0.90

Scale-to-zero on an LLM is a bill-shaped decision, not a latency-shaped one: keeping a model warm is the only way to be interactive, so the economics only work when utilization is very low or traffic is genuinely spiky

[major] configServerless GPU looks cheap until you notice what the cold-start penalty forces you to do about idle capacity, and the fix for that penalty is to keep the container warm - which is the thing you were trying to avoid. The platforms state the trade-off themselves: 'It is not always …gotchas/dep-serverless-scale-to-zero-only-pays-for-spiky-or-mostly-idle

verifiedmedium · 0.82

Searching arXiv for DeepGEMM by title returns a same-name paper by unrelated CPU-inference authors - reject it on author grounds

[major] toolchainThe natural way to fill the DeepGEMM citation is a title search on the arXiv API. That search DOES return exactly one hit, it is titled 'DeepGEMM', and it looks authoritative - so it is easy to record it as NVIDIA's DeepGEMM and move on. It is a different project by entirely diff…gotchas/meta-deepgemm-title-search-returns-the-wrong-deepgemm

verifiedhigh · 0.95

Setting VLLM_MARLIN_INPUT_DTYPE=fp8 enables Marlin W4A8-FP8 on sm_121a where it silently produces garbage

[blocker] kernelOn NVIDIA GB10 (DGX Spark, sm_121a) serving a WNA16 INT4 MoE (Qwen3.5-122B-A10B, AutoRound/GPTQ-style, group_size 128) with VLLM_MARLIN_INPUT_DTYPE=fp8, the server starts normally, no crash and no NaN assert, and at temperature=0 every reasoning prompt degenerates into a repeated…gotchas/marlin-w4a8-fp8-silent-corrupt-sm121a

verifiedhigh · 0.85

sglang:utilization and sglang:max_running_requests_under_SLO are documented but permanently zero on current main - do not build an autoscaling rule on them

[minor] measurementYou configure SGLang with an SLO target, expect sglang:max_running_requests_under_SLO to report the headroom the engine computed, and find it permanently 0. Or sglang:utilization is flat at 0 on a decode worker that is plainly saturated. On a prefill-role worker the same metric r…gotchas/obs-sglang-utilization-metric-is-dead

verifiedhigh · 0.90

SGLang's is_blackwell / is_sm100_supported / is_sm120_supported predicates all include a CUDA >= 12.8 requirement, so 'is Blackwell' in SGLang dispatch means Blackwell AND a new enough runtime

[minor] configThe same Blackwell GPU takes a different dispatch branch under CUDA 12.6 than under CUDA 12.8. A native FP4 or MXFP4 path silently degrades, or a hard error appears, on hardware that a support table marks as supported — and the only way to tell which is which is to know the CUDA …gotchas/sglang-blackwell-predicate-is-a-cuda-version-test

verifiedhigh · 0.90

SHARP in-network reduction only helps all-reduce, not all-to-all or all-gather

[major] hardwareSHARP in-network reduction is often described as accelerating 'collective communication', but it only accelerates all-reduce operations. All-to-all (used in MoE expert-parallel) and all-gather do not benefit from SHARP.gotchas/nic-sharp-in-network-reduction-limits

verifiedmedium · 0.80

Silicon Box is an advanced chiplet packaging foundry, not an AI inference accelerator

[minor] hardwareSearching for Silicon Box AI inference accelerators or silicon compute specifications yields zero accelerator parts or TOPS numbers.gotchas/ainv-gotcha-silicon-box-packaging-not-accelerator

verifiedhigh · 0.95

Sixteen of the eighteen venue='other' papers name a real venue with a publisher deposit - the gap was a missing enum member, not missing information

[major] toolchainAuditing all 18 papers with venue='other' in this repo, on the expectation that most were genuinely unverifiable, produced the opposite result: every one of them names a real venue, and 12 of the 15 in-scope records have a PUBLISHER deposit in hand that was already sitting in dat…gotchas/w10-b-venue-other-is-a-provenance-claim-not-a-vocabulary-claim

verifiedhigh · 0.92

Sizing a rack by summing GPU TDP understates it badly: a 72-GPU GB200 NVL72 rack draws ~120 kW against a 1,200 W per-GPU ceiling, and its installed PSU capacity is 264 kW

[major] configA power budget built by multiplying published per-GPU Max TDP by GPU count is wrong by roughly 40% on facility draw and by more than 2x on installed electrical capacity. For GB200 NVL72: 72 x 1,200 W = 86.4 kW of GPU TDP, versus the vendor-stated 'The rack power consumption is ap…gotchas/power-floor-scaling-rack-is-not-sum-of-gpu-tdp

verifiedhigh · 0.90

SM120 FlashInfer sparse-MLA only instantiates page_block_size=64, so mixed-compress-ratio DeepSeek checkpoints cannot start

[blocker] kernelOn 8x RTX PRO 6000 Blackwell (SM120) with TP8, deepseek-ai/DeepSeek-V4.1-Flash fails to start with: ValueError: SM120 sparse-MLA has no decode kernel for this shape: num_tokens=8, num_heads=8, topk=128, d_qk=512, page_block_size=32, model_type=dsv4, extra_topk=0.gotchas/sm120-sparse-mla-block-size

verifiedhigh · 0.90

Speculative decoding that works standalone crashes under PD disaggregation, on a per-algorithm basis with no warning

[major] frameworkDFLASH speculative decoding crashes at startup under PD disaggregation: the first update_running_batch on the decode side dies with AttributeError: 'NoneType' object has no attribute 'prepare_for_decode' at sglang/srt/speculative/spec_utils.py:1010. Once that is locally patched, …gotchas/pd-speculative-decoding-unsupported-per-algorithm-under-pd

verifiedhigh · 0.85

Stacking a KV-tier connector under the PD transfer connector silently corrupts output, while either alone is clean

[blocker] frameworkIntermittent silent output corruption / garbled generation in a two-node 1P1D disaggregated deployment: long runs of nonsensical mixed-language and repeated tokens (for example repeated 'Effective/effective' mixed with unrelated English/Chinese fragments), some streams eventually…gotchas/pd-multi-connector-stack-silent-corruption

verifiedmedium · 0.80

Standard load balancers cannot see KV cache state - routing decisions are blind to cache locality

[major] configA standard load balancer (round-robin, least-connections) sends requests to replicas that do not have the prefix cached, causing redundant prefill computation and higher TTFT.gotchas/serve-routing-cache-state-not-visible-to-load-balancer

verifiedhigh · 0.85

Streaming ASR quality is materially worse than offline ASR

[major] configStreaming ASR systems (whisper-streaming, WhisperLive, etc.) produce transcription quality that is noticeably worse than offline Whisper on the same audio. Word error rates are higher, especially at segment boundaries.gotchas/aud-gotcha-streaming-quality-gap

verifiedhigh · 0.85

Sustained clocks are lower than burst clocks: power and thermal limits in compact cases

[major] hardwareA GPU's sustained inference performance is significantly lower than its burst performance. A card rated at 575W TGP (RTX 5090) may sustain only 400-450W under continuous load, reducing clocks by 10-20%. A laptop SoC may reduce power by 30-50% under sustained load to manage therma…gotchas/local-power-thermal-sustained-clocks

verifiedmedium · 0.80

Sustained load drives SW Power Cap and thermal throttling, which move the tail of the latency distribution rather than its mean - so a benchmark that reports only p50 hides the regression entirely

[major] measurementMean and median latency look fine while the tail degrades. A deployment passes an SLO defined on p50 and breaks for users, and the failures cluster in the long runs - after the node has been at full load for a while, or on the hottest day, or on the busiest GPU in the node. Bench…gotchas/dvfs-power-cap-moves-tail-latency-under-sustained-load

verifiedhigh · 0.85

SXM2/SXM4 accelerator modules are bare - they need a carrier/baseboard + heatsink and have no PCIe edge

[major] hardwareA used SXM2 or SXM4 module bought bare (V100 $349-800, A100 40GB $2,499) arrives with no cooler and no PCIe edge connector - it cannot be installed in a normal workstation slot. The listing photos look like a GPU but the part is electrically and mechanically a daughterboard for a…gotchas/gotcha-sxm-module-needs-carrier-baseboard

verifiedhigh · 0.88

SYCL crashes or emits gibberish on hybrid linear-attention architectures (qwen3next, qwen35) on Intel Arc Pro B60

[major] kernelOn 2x Intel Arc Pro B60 (24 GB each, no XeLink, Level Zero 1.15.38308+1, Ryzen 7 7700), models with a qwen3next or qwen35 architecture - specifically Qwen3-Coder-Next at Q3_K_XL (33.79 GiB) and Q4_K_M (48.18 GiB) - either crash inside ggml_sycl_op_mul_mat or return empty/gibberis…gotchas/sycl-hybrid-linear-attention-arch-crash-intel-arc

verifiedhigh · 0.85

SYCL_CACHE_PERSISTENT=1 causes crashes in llama.cpp; not recommended

[major] configSetting SYCL_CACHE_PERSISTENT=1 in the environment causes crashes when running llama.cpp with the SYCL backend. The crash occurs because the SYCL runtime caches and reuses JIT-compiled binaries, and when the binary file changes (e.g., due to a code update), the new and old codes …gotchas/xpu-sycl-cache-persistent-crash

verifiedhigh · 0.90

Sync Boost ties every GPU in a node to the slowest one's achievable clock, so one hot or throttled GPU silently caps the throughput of all eight

[major] hardwareA tensor-parallel job runs at a clock set by whichever GPU in the node is worst-conditioned, and the operator has no idea which one. Node throughput sits noticeably below the per-GPU figure, uniformly across all eight devices, and the deficit is stable rather than noisy. The fail…gotchas/sync-boost-slowest-gpu-caps-whole-node

verifiedhigh · 0.85

Temperature-0 greedy decode is not a correctness oracle: the same prompt at temperature 0 produced 80 distinct completions across 1000 runs, all fluent and plausible, with first divergence at token 103

[major] measurementThe cheap oracle fails open. Serving Qwen/Qwen3-235B-A22B-Instruct-2507 at temperature 0 - the configuration everyone reaches for when they want reproducibility - and sampling 1000 completions of 1000 tokens from the single prompt 'Tell me about Richard Feynman' yields 80 UNIQUE …gotchas/nrg2-temperature-zero-greedy-decode-is-not-a-determinism-oracle

verifiedhigh · 0.85

Ten speculative-decoding records store an acceptance LENGTH under metric=quality and state no draft length k, which is the one parameter without which the break-even arithmetic cannot be run at all

[major] measurementTen records - sd-eagle1-acceptance-vicuna13b-t0, sd-eagle2-acceptance-vicuna13b-t0 and -t1, sd-eagle2-acceptance-llama2-70b-t0, sd-eagle2-acceptance-llama3-8b-t0, sd-hydra-acceptance-vicuna13b-t0, sd-medusa-acceptance-vicuna13b-t0, sd-lookahead-acceptance-vicuna13b-t0, sd-pld-acc…gotchas/sd-acceptance-length-is-not-a-quality-score-and-makes-break-even

verifiedhigh · 0.85

Ten speculative-decoding records store an acceptance LENGTH under metric=quality and state no draft length k, which is the one parameter without which the break-even arithmetic cannot be run at all

verifiedhigh · 0.85

Tensor parallelism spanning nodes without a multi-node NVLink fabric hangs during communicator initialization, and the logs show partial success rather than failure

[blocker] configTwo distinct reports, same failure shape. (a) 2 nodes x 4 H200 NVL, TP=8: hangs in ncclCommInitRank during startup - and the diagnostic trap is that even local_rank ranks complete Init COMPLETE while the odd ranks block in the graph-connect phase, so a log that shows successful i…gotchas/mgpu-multinode-tp-across-fabrics-hangs-in-comm-init

verifiedhigh · 0.88

Tensor-parallel deployment fails NCCL init because the container's default 64 MiB /dev/shm is orders of magnitude too small

[blocker] configTP>1 across GPUs on the same node fails at NCCL initialization or shortly after, with shm-ENOSPC signatures. The clearest verbatim record is 4x Tesla T4 under AzureML MpiDistribution, where the failure set is: 'OSError: [Errno 28] No space left on device', 'RuntimeError: cuDNN er…gotchas/mgpu-tp-needs-dev-shm-larger-than-container-default

verifiedhigh · 0.92

TensorRT-LLM has no paper - the absence is verified by an empty title search, and its kernels are documented in this repo as records instead

[minor] toolchainA reader auditing this repo for inference coverage sees that TensorRT-LLM - one of the two or three production inference engines on NVIDIA hardware - has engine, observability and multiple FLOP/kernel records but NO entry in data/papers/. The temptation is to go looking for an ar…gotchas/meta-tensorrt-llm-has-no-paper-do-not-invent-one

verifiedhigh · 0.90

TensorRT-LLM ships a Marlin NVFP4 W4A16 kernel gated to SM89, SM90-99 and SM120/121 — the exact complement of its own native FP4 on sm100/103

[major] kernelAn NVFP4 checkpoint on an Ada, Hopper or RTX-50-series part can be served by TensorRT-LLM through a Marlin weight-only kernel rather than failing or dequantising, and the arch set is disjoint from the one where the native FP4 CUTLASS kernels exist.gotchas/trtllm-now-has-a-marlin-nvfp4-kernel-on-ada-hopper-and-sm120

verifiedhigh · 0.90

Tenstorrent Grayskull has no current LLM path in tt-metal

[blocker] toolchainmodels/demos/ in tt-metal has wormhole, blackhole, t3000, tg directories but NO grayskull dir; the vendor model matrix lists zero Grayskull rows.gotchas/gotcha-tenstorrent-grayskull-no-llm-path

verifiedhigh · 0.85

terminationGracePeriodSeconds defaults to 30s while an LLM drain window needs minutes — SIGKILL lands mid-drain, so the graceful path is configured but never used

[major] operationsA deployment has --shutdown-timeout set and still fails in-flight requests on rollout, with logs showing the drain starting and then stopping abruptly rather than completing. The pod sits Terminating for exactly the grace period and is then SIGKILLed. Because the abort is forced …gotchas/ops-termination-grace-period-kills-the-drain

verifiedhigh · 0.90

Tesla M40 (Maxwell) has no INT8 DP4A - slower than the P40 it barely undercuts

[major] hardwareM40 24GB at ~$189 looks like the cheapest 24GB card, but llama.cpp falls back to the FP32/CUDA-core path and it is SLOWER than a P40 despite similar VRAM.gotchas/gotcha-maxwell-m40-no-dp4a-slower-than-p40

verifiedmedium · 0.80

Tesla P40/M40 (Pascal/Maxwell) have no FP16 tensor cores - GGUF INT8 DP4A is the only usable LLM path

[blocker] hardwareAttempting to run an FP16 model or a non-DP4A quant on a Tesla P40 or M40 either fails outright or falls back to a path so slow it is unusable. On the P40, FP16 runs at 1/64 the FP32 rate. On the M40 (Maxwell CC5.2) there is not even an INT8 DP4A path, so llama.cpp drops to the F…gotchas/gotcha-pascal-p40-m40-no-fp16-tensor-dp4a-only

verifiedhigh · 0.85

TGI exposes queue time, inference time and time-per-token but no TTFT and no KV cache utilization - a TTFT SLO on TGI cannot be served by its metrics

[major] measurementYou stand up TGI, wire up /metrics, and find that the two latency numbers you need are missing. Queue time and inference time are there but the split you want is not; time-per-token is there but includes prefill so it is not comparable to anyone else's TPOT. There is no cache-occ…gotchas/obs-tgi-has-no-ttft-metric

verifiedhigh · 0.92

TGI's 429 'Model is overloaded' is a concurrency semaphore, not a queue-depth signal — so tgi_queue_size can grow while 429s are firing

[major] operationsUnder load, TGI returns 429 {"error":"Model is overloaded","error_type":"overloaded"} to some clients while the Prometheus gauge tgi_queue_size stays at or near zero, and tgi_batch_current_size shows the backend working normally. The two observations look contradictory: the serve…gotchas/ops-tgi-429-is-semaphore-not-queue

verifiedhigh · 0.85

The abbreviated GPU UUID form that NVIDIA documents for CUDA_VISIBLE_DEVICES crashes vLLM's device resolution before the model even loads

[major] configOn 8x H200 (driver 590.48.01, Rocky Linux 9.8), setting CUDA_VISIBLE_DEVICES to a short UUID - e.g. GPU-af7b61d8 where the driver reports GPU-af7b61d8-21af-baea-6a19-42a1f9f7c3cb - crashes during ModelConfig construction, before the model finishes loading: 'ValueError: invalid li…gotchas/mgpu-short-cuda-visible-devices-uuid-crashes-nvml-resolution

verifiedhigh · 0.85

The community-standard way to pool VRAM across consumer NVIDIA cards is layer sharding over llama.cpp's RPC backend, not NCCL tensor parallelism - because it sidesteps the P2P ban entirely

[major] configRun `ggml-rpc-server` on each machine (each exposing its own CUDA device over TCP), then point llama-server at the workers with `--rpc host:port[,host:port...]` plus `--split-mode layer -ngl 999 --tensor-split`. What you get is one model whose weights are spread across cards in d…gotchas/comm-community-standard-rpc-sharding-for-multi-gpu

verifiedmedium · 0.80

The corpus prices 221 cloud rows but only one is a per-user figure, so the two populations cannot be compared - and spot is 35-43% off while capacity-block is 49% MORE than on-demand

[major] measurementThe corpus holds 221 priced rows from python tools/cost_per_token.py across 18 providers, 6 accelerators and 17 models, and the cheapest row is $151.50 per million output tokens. Asked to rank them by value, the first trap is that 220 of the 221 rows are AGGREGATE throughput and …gotchas/value-rental-ranking-by-workload-and-the-aggregate-trap

verifiedhigh · 0.90

The dense FLOPS value is not always the first number printed and the governing footnote differs per row and per vendor, so NVIDIA alone uses three mutually inconsistent sparsity conventions

[major] hardwareA quoted peak FLOPS figure gets halved when it should not, or left unhalved when it should be, and the 2x error propagates into arithmetic intensity and every roofline, batching and cost conclusion built on it. The failure is silent because the wrong number is still a published N…gotchas/aa-gotcha-vendor-sparsity-footnote-conventions-differ-by-vendor

verifiedhigh · 0.90

The gateway rate limiter's counters are per-instance in-memory: N workers means N times the intended limit, and a restart resets every tenant's budget to full

[major] operationsTenant rate limits drift upward under load and reset to nothing on deploys. A tenant configured for 60 requests/minute sustains several hundred per minute when the gateway has several workers, and every rolling restart hands every tenant a fresh full budget, so a tenant that has …gotchas/ops-wasm-ratelimit-state-is-per-worker-and-lost-on-restart

verifiedhigh · 0.85

The gfx906 community's own supported-ROCm matrix, assembled from failures: 6.3.3 is the newest version that works unmodified, 5.7.1 is the last fully stable one, and 7.x works only after the rocBLAS workaround

[blocker] configThe community FAQ, written by its own members, states: 'Last official working rocm without modification is rocm 6.3.3. Last fully supported stable is rocm 5.7.1.' Specific version-by-version failures reported: ROCm 7.0.0 - cannot build any torch; ROCm 6.4.4 + torch 2.7.1 - text w…gotchas/comm-gfx906-rocm-version-support-matrix-from-users

verifiedmedium · 0.75

The gfx906 vLLM forks are architecturally single-node: MultiprocExecutor forces master_addr back to 127.0.0.1 and requires local_world_size == world_size, so a 16-node InfiniBand cluster deadlocks

[blocker] frameworkDocumented first-hand on a 16-node InfiniBand cluster with 1 MI50 per node, running branch gfx906/v0.19.1rc0.x of vllm-gfx906-mobydick via torchrun. EngineCore init prints: (EngineCore) INFO [multiproc_executor.py:137] DP group leader: master_addr=127.0.0.1, world_size=16, loca…gotchas/comm-gfx906-vllm-fork-locked-to-single-node-ipc

verifiedmedium · 0.70

The GreenBench Apple M4 Pro throughput set exceeds the batch-1 decode roof of the part it names, and the excess tracks exactly the two smallest models - which points at the bandwidth figure, not at the measurements

[minor] measurementA roofline sweep of the batch-1 decode records runs the GreenBench Apple set against the part the source actually names - a MacBook Pro 14-inch M4 Pro, 273 GB/s - and one of the five models exceeds its own hard roof. Llama 3.2 3B at Q4_K_M is 1.94 GB of weights, giving roof = 273…gotchas/bench-greenbench-apple-set-exceeds-its-own-part-roof

verifiedmedium · 0.70

The guard model is a second extraction target: an LLM-as-judge can be replicated from black-box queries, so moderation capability is intellectual property you have to defend

[major] securityThe moderation model is treated as plumbing - a small safety component in front of the real product. It is a trained model with measured capability, queried over an API, and it can be replicated.gotchas/sec-guard-and-moderation-models-are-themselves-extraction-target

verifiedhigh · 0.85

The guard model is a second extraction target: an LLM-as-judge can be replicated from black-box queries, so moderation capability is intellectual property you have to defend

verifiedhigh · 0.85

The injection defences with provable guarantees work by removing the LLM's control over control flow - and cost measured task completion

[blocker] securityEvery input filter and output filter fails to some published attack, and the literature's own conclusion is that filtering is not where the guarantee comes from. The designs that do provide a guarantee get it by changing the architecture, and they pay a measurable amount of utili…gotchas/sec-architectural-injection-defense-is-capability-scoping

verifiedhigh · 0.88

The KV-cache timing side channel largely collapses under realistic contention - do not extrapolate benchmark numbers to production

[major] privacyA security review quotes a KV-cache timing side channel with a large effect size, and the fix - disable prefix caching - is expensive. Then the channel turns out to be nearly unusable on a busy production server, and the fix bought almost nothing.gotchas/sec-kv-cache-timing-attack-collapses-under-load

verifiedhigh · 0.85

The llama.cpp no-paper negative still holds, but the count of papers merely MENTIONING llama.cpp grew from 5 to 70 - the gap grows faster than it gets filled

[minor] toolchainThe existing gotcha meta-llama-cpp-has-no-paper-do-not-invent-one was written against a search that returned 5 results for all:"llama.cpp". It now returns 70. That looks like the negative is decaying - as though a llama.cpp paper might now exist, or as though the original finding…gotchas/meta-llamacpp-negative-holds-but-mention-count-is-growing

verifiedhigh · 0.90

The migration resistance has a concrete number: Ollama is treated as finished at about five concurrent users, which is what pushes teams onto vLLM or SGLang

[minor] toolchainThe most operationally useful sentiment datum in the corpus is an explicit concurrency ceiling given by someone running a real organization on it. A single-firm deployment (Threadripper Pro 7965WX, 256GB DDR5-6000, one L40S, roughly EUR 14k, Docker on native Ubuntu, OpenWebUI for…gotchas/sent-migration-resistance-concurrency-ceiling-caps-ollama

verifiedmedium · 0.70

The migration resistance: people stay on old hardware and old stacks because the cost is toolchain churn, not because the new part is not faster

[major] toolchainThe blocker to upgrading is consistently described as version coupling across the inference stack, never as the performance delta. A practitioner who benchmarks vLLM on MI300X reports that configuration changes alone produced '2-3X differences in perf numbers', and that the same …gotchas/sent-migration-resistance-version-coupling-not-performance

verifiedmedium · 0.70

The practitioner answer to 'which inference engine is best' is a situation table, not a winner; asking for a ranking is answered by refusing the framing

[minor] toolchainA buyer or engineer asks which inference engine is best. The high-scoring replies do not name one. The most-cited framing is that the question is malformed: 'It's often very silly to ask "which is better?" between tools. Which is better: a screwdriver or a hammer?' (15 upvotes). …gotchas/sent-engine-choice-is-situational-not-a-ranking

verifiedmedium · 0.70

The provider's marketing page is static HTML but its actual rate table is JavaScript-rendered or JSON-only - the API is the only reproducible surface, and the table is the only one that goes stale silently

[major] measurementYou scrape a provider's pricing page and either get nothing or get a headline. openai.com/api/pricing/ renders the Business/Enterprise seat tiers, not the API rate card - the token prices are on platform.openai.com/docs/pricing. groq.com/pricing is a marketing page with no rate t…gotchas/rent-vendor-pricing-tables-are-js-but-their-apis-are-not

verifiedhigh · 0.90

The QMoE sub-1-bit expert kernel is a GEMV, and it loses as soon as two tokens share an expert

[major] kernelA sub-1-bit expert kernel that benchmarks faster than bfloat16 per layer comes out slower end to end, and the gap grows with the batch. On the published QMoE kernels the isolated compressed matrix-vector product beats PyTorch bfloat16 cuBLAS on every expert shape, by up to 35 per…gotchas/qmoe-sub-1-bit-gemv-only-pays-at-batch-1

verifiedmedium · 0.80

The queue in front of an LLM engine is the wrong place to queue — engine-local queues are invisible, ungovernable and lost on restart

[blocker] operationsRequests sit in a model server's local queue where the operator cannot see them, cannot prioritise them, and cannot move them. Priority inversion is invisible: a latency-critical interactive request waits behind large offline batches because, as the reference puts it, 'Model serv…gotchas/ops-queue-belongs-at-the-gateway

verifiedhigh · 0.90

The ROCm LLVM loop-unroller miscompiles gfx906 kernels - fixing it takes -mllvm --amdgpu-unroll-threshold-local=600, and the real fix belongs upstream in rocm-libraries

[major] buildTwo community members hit the same compiler defect independently and resolved it the same way. legitsplit: 'This looks like the same problem I had. Compiled with -DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600" flag fixes arch bug', with before/after llama-bench tab…gotchas/comm-gfx906-compiler-flag-unroll-threshold

verifiedlow · 0.60

The ROCm Triton w4a16 path asserts on GPTQ models with a non-zero-group qzeros layout

[major] formatServing GPTQ-Int4 quantized Qwen3.6-27B fails at triton_w4a16.py line 201 with an assertion on qzeros.shape == (K // group_size, N // 8). Unquantized models run fine, and two independent GPTQ-Int4 repos fail the same way (raydelossantos/Qwen3.6-27B-GPTQ-Int4, btbtyler09/Qwen3.6-2…gotchas/triton-w4a16-gptq-qzeros-assert

verifiedhigh · 0.85

The router is a stateful memory consumer, not a stateless proxy: its prefix tree grows without bound and takes the whole data plane down with it while every engine stays healthy

[major] frameworkThe router pod restarts periodically - every few minutes or hours, with no obvious trigger - and each restart drops in-flight requests, resets all prefix affinity, and forces a fresh scrape window before load-aware routing has data again. The vLLM engines behind it never restart …gotchas/dep-router-prefix-tree-ram-is-the-router-oom

verifiedhigh · 0.85

The router reads a worker's protocol once at registration and never re-reads it, so a worker that stops serving its registered protocol is not detected — the circuit breaker opens on live traffic instead

[minor] operationsAfter a rolling update or a network change, one router replica's throughput drops while its peers are fine, and its circuit breaker for that worker is open. The worker's health is red on the side that actually serves traffic but the registry entry was written at startup; nothing …gotchas/ops-router-circuit-breaker-blind-to-h2c-downgrade

verifiedhigh · 0.85

The seven feature axes a reader chooses an engine on were recorded for a minority of engines - a dated audit of which are stated, which are absent, and which are affirmatively NO

[minor] measurementA reader comparing engines on 'does it do speculative decoding, chunked prefill, prefix caching, disaggregated prefill/decode, structured output, multi-model, multimodal' finds these recorded for some engines and simply missing for others, with no way to tell 'lacks it' from 'nob…gotchas/ev-feature-axis-coverage-audit

verifiedhigh · 0.85

The SYCL backend aborts at model load when the GPU memory query fails, with no fallback

[blocker] toolchainllama-server and llama-cli abort at model load with GGML_ABORT in ggml_backend_sycl_device_get_memory: 'failed to get device memory size'. Confirmed on two very different parts - a Sparkle Arc Pro B60 24GB (Battlemage G21, 8086:e211) on a fresh Ubuntu 26.04 with the xe driver, Gu…gotchas/sycl-device-memory-query-abort-blocks-model-load

verifiedhigh · 0.90

The unifying finding of this slice: what practitioners optimise for is integration cost and ecosystem maturity, not measured throughput - the same driver explains every engine and hardware preference recorded here

[minor] frameworkEvery strong consensus in this sentiment layer resolves to the same underlying variable, and it is not performance. On engines: llama.cpp wins on dependency count and quant coverage, vLLM wins on Day-1 model support and contributor headcount, SGLang wins on multinode setup simpli…gotchas/sent-ecosystem-gap-is-the-real-preference-driver

verifiedmedium · 0.70

The used RTX 3090 is no longer ~$550-650 - it moved to ~$900-1,000 in Oct-2026 and escalated to a $1,430-2,220 asking band on the Dec-2026 refresh

[major] operationsAny cost model or buying decision predicated on a ~$550-650 used 3090 is now wrong by ~2-3x. The Oct-2026 realistic band was ~$900-1,000; the Dec-2026 refresh observed asking prices of $1,430-2,220 - a further +50-130% spike.gotchas/gotcha-used-3090-price-moved

verifiedhigh · 0.85

The widely-repeated 3.6 TB/s per-GPU figure for NVLink 6 has no first-party NVIDIA source and NVIDIA's own page says 3,000 GB/s - an unsourced secondary number is 20% high and reads as more authoritative than the vendor page it contradicts

[major] hardwareTwo first-party-adjacent numbers for the same quantity, differing by 20%, and the wrong one is the one that propagates. NVIDIA's NVLink product page spec table gives 'NVLink Bandwidth per GPU 3,000 GB/s' for Sixth Generation and 'NVLink Switch GPU-to-GPU Bandwidth 3,000 GB/s' wit…gotchas/w7n-nvlink-6-3600-gbps-has-no-first-party-source

verifiedhigh · 0.88

The XMX dequant-into-tile GEMM silently falls back to the library GEMM past its tile constraints

[major] kernelA hand-written XMX dequant-GEMM that is clearly faster than oneDNN on one shape is the same speed as oneDNN, or slower, on the next shape, with no error and no log line unless you read the kernel-selection trace. On the llama.cpp SYCL backend the fast path only runs for batches o…gotchas/sycl-xmx-dequant-gemm-falls-back-past-tile-constraints

verifiedmedium · 0.80

Tier A is 45.7% of source records but 87.8% of numeric figures - and the type that carries the figures is not the type that carries the sources

[None] measurementAsked how much of this corpus traces to a primary vendor document, the answer you get depends entirely on which denominator you use, and the two denominators disagree by a factor of two. By source-record count, Tier A (primary vendor document) is 895 of 1960 records = 45.7%. By t…gotchas/w7r-tier-a-by-record-count-is-not-tier-a-by-figure-weight

verifiedhigh · 0.92

Time-slicing provides no throughput isolation - one tenant's load silently degrades all others

[major] hardwareOne tenant runs a large batch job, and all other tenants on the same GPU experience latency degradation. There is no isolation.gotchas/serve-time-slicing-no-throughput-isolation

verifiedhigh · 0.85

torch.load with pickle is a remote code execution vector: loading a malicious model file can execute arbitrary code

[blocker] formatYou download a .pt or .bin model file from an untrusted source and load it with torch.load(). The file contains malicious pickle opcodes that execute arbitrary code on your machine.gotchas/meth-pickle-code-execution-vector

verifiedhigh · 0.90

TPU ICI is a 2D or 3D torus, so pod-wide bisection bandwidth is not a simple fraction of total bandwidth - Google's own all-reduce vs bisection figures differ by 32x to 53x, and a reader who assumes a uniform mesh mis-sizes the network by that factor

[major] hardwareA reader sizing a multi-host TPU job treats the pod as a uniform mesh or as an unordered set of equivalent chips, and computes usable fabric bandwidth by dividing a pod aggregate by a chip count or by assuming any-to-any behaviour. On a torus that is wrong by the gap between Goog…gotchas/w7c-gotcha-tpu-torus-bisection-is-not-a-share-of-total

verifiedhigh · 0.85

Triton JIT compilation on gfx906 makes first-token latency enormous and it recurs on every restart unless the compile cache is persisted; on MI50 it silently switches to the V0 engine

[minor] frameworkMulti-minute time-to-first-token attributed directly to Triton JIT compilation, recurring on every service restart and machine reboot. Compounding it, vLLM logs 'rocm is experimental on VLLM_USE_V1=1. Falling back to V0 Engine.' Another user reports large-context startups stuck f…gotchas/comm-gfx906-triton-jit-cold-start-ttft

verifiedlow · 0.60

trust_remote_code=True in HuggingFace transformers executes arbitrary Python code from the model repository

[blocker] frameworkYou call AutoModel.from_pretrained() with trust_remote_code=True on a malicious model repository. The repository's custom modeling code executes arbitrary Python code on your machine with your privileges.gotchas/meth-trust-remote-code-execution

verifiedhigh · 0.85

TT-Metal's decode attention kernel only implements multi-query attention

[major] kernelA decode attention call that is fast on a multi-query model does not accept a grouped-query model on the same kernel. The scaled-dot-product attention decode op states that its implementation is Flash-Decode and that it currently only supports MQA while decoding a single token. Q…gotchas/tt-metal-decode-attention-is-mqa-only

verifiedmedium · 0.80

Two models quantized at 'the same Q4_K_M level' can land at 4.75 and 4.30 bpw, so named-quant comparisons across models are not controlled experiments

[minor] measurementStandard llama.cpp quantization applies hardcoded rules ('use Q4_K_M, except bump some tensors up/down, except fall back if incompatible, except keep some tensors unquantized'), so two different models quantized at the same named level end up with different effective bits-per-wei…gotchas/same-named-quant-is-not-a-controlled-comparison

verifiedhigh · 0.85

Two NVFP4 kernels in vLLM's registry assert no architecture support at all, so 'the engine supports FP4' is not decidable from its own gate list

[major] kernelAn NVFP4 checkpoint serves on hardware that has no FP4 tensor-core path — a V100, an A100 — with no capability error at load time. The failure, if any, surfaces later and from inside FlashInfer's own dispatch rather than from vLLM's kernel gate.gotchas/nvfp4-registry-has-kernels-with-no-arch-gate

verifiedhigh · 0.90

Two parts of the same architecture shipping different 'Compute APIs' / 'Graphics APIs' rows on their datasheets is a vendor template-revision artifact, and reading it as a capability difference fabricates a hardware constraint

[None] hardwareAn accelerator record asserts a software-support ceiling - 'CUDA 11.6', 'Vulkan 1.33' - for one part, while a sibling part of the SAME architecture in the same product family and the same shipping wave reports 'CUDA 12.8' / 'Vulkan 1.43'. The records look like they disagree about…gotchas/aa-gotcha-datasheet-api-row-stale-not-hardware-difference

verifiedhigh · 0.90

Two source records cite a DOI that resolves to a DIFFERENT paper - a DOI is not self-verifying evidence just because it resolves

[major] toolchainWhen checking the venue of every paper filed venue='other', the Crossref deposit was read back for each DOI quoted in the source records. Two of them resolve cleanly to a DIFFERENT paper than the one they are attached to. pap-mm-colbert (source of papers/colbert, arXiv 2004.12832…gotchas/w10-a-source-doi-resolves-to-a-different-paper

verifiedhigh · 0.95

Typed fields cannot distinguish legal vs commercial vs disclosure vendor silence, nor definitional vs sourcing vs verified-negative null bandwidth without schema discriminators

[minor] measurementTwo independent audits discovered identical compression traps across disparate layers: in data/sources/, four distinct vendor silences (legal export control, commercial paywall, vendor disclosure practice with no spec table, and packaging lack of config.json) all collapsed into a…gotchas/w11-schema-discriminators-for-vendor-silence-and-null-bandwidth

verifiedhigh · 0.95

Unified memory is not VRAM: capacity, bandwidth, and sharing constraints

[major] hardwareA Mac with 128GB unified memory can serve a larger model than a discrete GPU with 24GB VRAM, but the advantage is capacity, not speed. The unified memory bandwidth is shared with the CPU and is not dedicated to the GPU.gotchas/local-unified-memory-not-vram

verifiedmedium · 0.80

Used datacenter passive cards need airflow/shroud mods for desktop use

[major] hardwarePassive dual-slot/fhfl cards run hot or throttle in a normal desktop; some need EPS-12V 8-pin (not PCIe 8-pin) and a ReBAR/Above-4G-capable BIOS.gotchas/gotcha-used-datacenter-passive-cooling-airflow

verifiedmedium · 0.80

Used MI50 cards accumulate correctable then uncorrectable HBM errors that pass ECC and silently corrupt the KV cache, so the model degenerates into repeating non-existent tokens with no error raised

[blocker] hardwareA member running four used MI50s reports one card accumulating correctable ECC errors that 'should be fine but they are not - sometimes errors slip past ECC and corrupt the models KV cache and break the model into repeating non existing tokens'. Same member, four cards total: one…gotchas/comm-mi50-uncorrectable-ecc-corrupts-kv-cache

verifiedmedium · 0.70

Verified negative: no vendor, no paper and no official results page publishes a tok/s figure for any of the six third-party field-report shapes in this slice - a single community post is the entire published record

[major] measurementA reader auditing data/benchmarks/ for provenance finds records whose only source is a GitHub issue, and the audit question is whether a better source exists that simply was not cited. For the six shapes below the answer is NO, and the answer was established by searching, not by …gotchas/w8-no-vendor-publishes-tok-s-for-these-six-shapes

verifiedhigh · 0.85

Video frame sampling rate is the primary cost knob

[major] configVideo inference cost is dominated by the frame sampling rate. A 1-minute video at 1 fps = 60 frames, but at 2 fps = 120 frames, doubling the cost.gotchas/video-frame-sampling-rate-is-the-primary-cost-knob

verifiedmedium · 0.80

Vision encoder is a separate model requiring separate memory

[minor] configThe vision encoder is a separate model that requires its own GPU memory, reducing the memory available for the LLM and KV cache. For a ViT-L/16 encoder (304M params, bf16), this is ~608 MB of additional memory.gotchas/vision-encoder-is-a-separate-model-requiring-separate-memory

verifiedhigh · 0.85

vLLM 0.30.0's FA4 kernel ignores num_splits on SM90, costing up to 48% decode throughput

[major] kernelDecode on H100 (SM90) is 48% slower per token than on v0.26.0 at batch 1 and 36% slower at batch 4 for google/gemma-4-26B-A4B-it at ~5.9k context, while TTFT is flat at 61 ms on both. Results stay correct, so nothing fails. Kernel sweep at Gemma-4 global-layer shapes: v0.26.0 goe…gotchas/fa4-num-splits-ignored-sm90

verifiedhigh · 0.95

vLLM auto-selects an FP8 linear kernel on RDNA4 that is slower than the one it picked before v0.28

[minor] frameworkOn RDNA4 (Radeon AI PRO R9700, gfx1201), serving FP8 compressed-tensors checkpoints with --kv-cache-dtype fp8 and TRITON_ATTN, decode throughput depends on a kernel selector rather than on the hardware. v0.27.1 picks ChannelWiseTorchFP8ScaledMMLinear Kernel and gets 35.45 t/s on …gotchas/vllm-rdna4-rowwise-fp8-kernel-auto-selected

verifiedhigh · 0.85

vLLM cannot transfer KV over PCIe P2P on Arc Pro B60 because the UCX ze_ipc backend is unmerged

[major] frameworkPrefill/decode disaggregation on Arc Pro B60 falls back off PCIe peer-to-peer for KV transfer. The blocker is upstream of vLLM: Intel's own engineer on the issue states that the ze_ipc support in UCX (openucx/ucx PR #11218) is not merged, so there is no supported P2P path for KV …gotchas/vllm-arc-b60-no-pcie-p2p-kv-transfer

verifiedhigh · 0.85

vLLM crashes on a MIG UUID in CUDA_VISIBLE_DEVICES, and its NVML path reports the parent GPU's memory for a MIG slice

[major] frameworkOn an A100 40GB split into 2x 3g.20gb MIG instances on bare metal, setting CUDA_VISIBLE_DEVICES to a MIG UUID crashes vLLM at startup with: 'ValueError: invalid literal for int() with base 10: \'MIG-fe59bfcd-3946-5d58-a00e-183ee397373b\''. There is also a silent second failure: e…gotchas/mgpu-mig-uuid-cannot-be-parsed-by-vllms-device-resolution

verifiedhigh · 0.85

vLLM derives prefix-cache block hashes from a FIXED default seed so nodes can share a cache - keeping the seed secret is not a security control

[minor] privacySomeone asks how to prevent an attacker forging prefix-cache blocks. The hash is sha256 and the seed is a documented constant, so any identical token sequence produces an identical block hash on every vLLM process anywhere - which is a deliberate feature and also the reason cache…gotchas/sec-kv-cache-hash-seed-is-fixed-by-default-for-cross-node-reuse

verifiedhigh · 0.85

vLLM has no plain 4-bit or sub-3-bit KV cache on NVIDIA: the only sub-8-bit cache dtypes are TurboQuant (research kernels) and int4_per_token_head, and there is no 2-bit KV dtype at all

[minor] formatA reader planning capacity for long-context serving asks for a 4-bit (or 2-bit) KV cache, finds that --kv-cache-dtype accepts only a fixed list of strings, and concludes the engine supports 8-bit and 4-bit in the same way. It does not. Every --kv-cache-dtype value vLLM accepts is…gotchas/w4q-vllm-no-sub-8bit-kv-cache-on-nvidia

verifiedhigh · 0.90

vLLM is Apache 2.0: the patent grant is the key differentiator from MIT, and it terminates on patent litigation

[minor] configYou want to use vLLM in a commercial product and assume it's 'open source' like MIT. But Apache 2.0 has a patent grant that MIT lacks, and it terminates if you sue for patent infringement.gotchas/meth-vllm-apache2-patent-grant

verifiedhigh · 0.90

vLLM MXFP4 is native on sm100/sm120 only if an optional module is installed, and is unconditional software emulation on sm90 and below: the model loads either way and only a logger.warning_once distinguishes them

[major] kernelNothing at all, at load time. An MXFP4 checkpoint (group 32, E8M0 scales) starts normally on an RTX 5090, an RTX PRO 6000 Blackwell, an H100 or a 4090, produces correct output, and reports a plausible tok/s. The only trace is a line that appears once in the startup log and is eas…gotchas/w4q-vllm-mxfp4-no-native-kernel-sm120

verifiedhigh · 0.90

vLLM ROCm custom paged attention gate requires gqa_ratio >= 3 on gfx1100 vs >= 1 on CDNA, forcing 1.8x-7.3x slower Triton fallback

[major] kernelOn AMD RDNA3 (gfx1100, RX 7900 series), vLLM's use_rocm_custom_paged_attention in vllm/platforms/rocm.py requires gqa_ratio >= 3, whereas on CDNA parts the same function requires gqa_ratio >= 1. For models with GQA ratio between 1 and 2, gfx1100 is silently excluded from the ROCm…gotchas/vllm-rocm-gfx1100-gqa-gate-excludes-rdna3

verifiedhigh · 0.85

vLLM silently downgrades FP8 activation scaling from per-token to per-tensor when CUTLASS FP8 is unavailable (Ampere, or CUDA older than 12.4 on Ada): the checkpoint says dynamic per-token, the engine computes per-tensor, nothing warns

[major] kernelAn FP8 checkpoint quantized with per-token (per-row) dynamic activation scales produces worse accuracy than the same checkpoint quantized with per-tensor scales would predict, with no error and no warning, and the served config still describes itself as dynamic per-token. The ope…gotchas/w4q-vllm-fp8-per-token-silently-downgrades-to-per-tensor

verifiedhigh · 0.92

vLLM silently substitutes per-tensor activation scaling for the requested per-token scaling whenever cutlass_fp8_supported() is false, losing accuracy headroom with no error

[minor] frameworkAn FP8 model configured for dynamic per-token activation quantization runs on hardware that cannot do it, produces correct output and the expected memory saving, and shows a throughput figure that does not match a per-token-quantised benchmark. No warning that a reader would noti…gotchas/fp8-per-token-silently-becomes-per-tensor-where-cutlass-is-unava

verifiedhigh · 0.90

vLLM silently substitutes per-tensor activation scaling for the requested per-token scaling whenever cutlass_fp8_supported() is false, losing accuracy headroom with no error

verifiedhigh · 0.90

vLLM W4A8 (INT4 weight + INT8 activation) is CPU-only, not Arm-only: it runs on x86 through two code paths the support table marks NO, and the NVIDIA NOs are real because loading fails loudly

[major] kernelTwo opposite surprises from one format. On the GPU side, an INT4-weight + INT8-activation checkpoint refuses to load at all with ValueError('Failed to find a kernel that can implement the WNA16 linear layer. Reasons: ...') followed by a per-kernel reason list in which every candi…gotchas/w4q-vllm-w4a8-is-cpu-only-on-cpu-not-arm-only

verifiedhigh · 0.92

vLLM's --api-key leaves /invocations and a dozen control endpoints unauthenticated, including /pause, /abort_requests and /update_weights

[blocker] operationsAn authenticated deployment is bypassable by anyone who can reach the HTTP server: an unauthenticated POST to /invocations runs inference at full capability, an unauthenticated POST to /pause takes the engine out of service, and /abort_requests drops in-flight work for every tena…gotchas/ops-api-key-covers-only-v1-prefixes

verifiedhigh · 0.95

vLLM's --gpu-memory-utilization is a fraction of TOTAL VRAM, so KV cache block allocation fails even with memory visibly free

[major] configEngine init fails with: ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine. Reported cases include llama-2 70B 4-bit AWQ on 4x A10G (worked on an older vLLM, fails from 0.2.5) and defog/sqlcoder2 at --gpu-mem…gotchas/vllm-kv-cache-block-budget

verifiedmedium · 0.80

vLLM's 'Your GPU does not have native support for FP4' warning fires on weight-only NVFP4 checkpoints, and gets quoted as evidence the silicon lacks FP4 kernels

[minor] measurementprepare_nvfp4_moe_layer_for_marlin() emits 'Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.' on a Jetson …gotchas/nvfp4-marlin-warning-blames-gpu-for-weight-only-checkpoint

verifiedhigh · 0.90

vLLM's AITER integration gate silently excludes RDNA3 (gfx1100)

[major] configOn a gfx1100 Radeon (Radeon PRO W7900 or 7900 XTX) vLLM logs 'AITER is not found or not supported on the current platform, QuarkOCP_MX will fall back to emulation' and then selects TritonInt8ScaledMMLinearKernel instead of AiterInt8ScaledMMLinearKernel. No error is raised; the IN…gotchas/aiter-gate-skips-rdna3-gfx1100

verifiedhigh · 0.95

vLLM's compiled extension fails to import against a source-built PyTorch with a different CXX11 ABI setting

[blocker] toolchainimport vllm raises ImportError: /workspace/vllm-abo/vllm/_C.abi3.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSsb. The .so file exists and vllm is installed; nothing about the version pins looks wrong.gotchas/torch-source-build-abi-mismatch

verifiedhigh · 0.90

vLLM's own overload signal is HTTP 503, not 429 — a naive 429-based retry policy never sees it

[major] operationsA load test or client with a 'retry on 429' rule reports 100% success while the server is shedding a large fraction of requests as HTTP 503 Service Unavailable. The 503s appear in the vLLM access log and as elevated 5xx, so an SLO built on 429 rate will read zero throttle events …gotchas/ops-vllm-admission-rejects-503-not-429

verifiedhigh · 0.94

vLLM's own security doc: all inter-node traffic is insecure by default - PyTorch Distributed has no authorization, sends unencrypted, and accepts connections from anywhere

[blocker] securityA multi-node tensor-parallel or KV-transfer deployment is network-reachable in ways the operator did not intend, and the traffic carrying prompts, KV cache contents and logits between nodes is neither authenticated nor encrypted.gotchas/sec-serving-open-weights-internode-network-is-unprotected-by-def

verifiedhigh · 0.92

vLLM's own security doc: all inter-node traffic is insecure by default - PyTorch Distributed has no authorization, sends unencrypted, and accepts connections from anywhere

verifiedhigh · 0.92

vLLM's query_marlin_supported_quant_types returns [] below compute capability 75 and SGLang's identical-named function returns [] below 80, so on Turing the two engines disagree about whether Marlin exists

[minor] frameworkThe same AWQ or GPTQ checkpoint on the same Turing GPU (T4, RTX 20-series) is served by a Marlin kernel in vLLM and by something else — Triton W4A16, or an error — in SGLang. Any statement of the form 'Marlin supports X' is ambiguous unless it names the engine.gotchas/vllm-and-sglang-disagree-on-where-marlin-starts

verifiedhigh · 0.90

vLLM's request logger writes full prompt bodies and token ids into the application log stream at DEBUG, with no field-level redaction anywhere in the engine

[blocker] privacyA deployment is found to have retained a year of prompt text and token ids in its log aggregator. Nobody enabled PII capture: --enable-log-requests plus a DEBUG log level does it, and the engine offers truncation (--max-log-len) but no redaction and no field selection.gotchas/sec-engine-request-logging-puts-full-prompts-in-logs-with-no-red

verifiedhigh · 0.85

vLLM's request logger writes full prompt bodies and token ids into the application log stream at DEBUG, with no field-level redaction anywhere in the engine

verifiedhigh · 0.85

vLLM's torch.compile cache defaults to a path inside the container, so on Kubernetes it is discarded on every pod recreation and you pay the full JIT compile on every cold start without ever seeing the benefit of caching

[major] configCold start does not improve no matter how many times you restart, even though the same model on the same hardware compiles much faster in a REPL. The intuition that 'the second start should be warm' is simply false on Kubernetes by default: the compile cache lives at /root/.cache…gotchas/dep-compile-cache-lives-in-container-and-is-wiped-every-pod

verifiedhigh · 0.94

Volta (V100, sm_70) is outside FlashAttention-2's target set - the cheapest HBM2 parts in the used market run the FA1/older attention path, not the modern FA2 kernels the llama.cpp/vLLM decode numbers assume

[major] kernelAn 8x V100 SXM2 box (~256 GB HBM2, ~900 GB/s per GPU, ~$3,000-6,000 used for a DGX-1-class chassis) looks like the fastest tok/s-per-dollar answer to a 70B Q4 dense model, and on aggregate bandwidth it is - vLLM/TGI tensor-parallel spread across the NVLink plane is roughly 30-50 …gotchas/gotcha-volta-v100-no-fa2-flashattention-2-does-not-target-sm70

verifiedmedium · 0.70

VRAM capacity is the hard wall for local inference: models either fit or they don't

[blocker] hardwareA model that fits in VRAM runs at full speed. A model that doesn't fit either fails to load or falls back to CPU inference, which is 10-100x slower. There is no middle ground.gotchas/local-vram-capacity-wall

verifiedmedium · 0.80

Vulkan decode collapses ~78% at long context from suballocation fragmentation; llama-bench does not see it

[major] measurementVulkan decode on an RX 7900 XTX falls off a cliff between 98K and 131K context: Qwen3.8-27B Q4_K_XL with q8_0 KV and FA on measures 40.4 t/s at ctx 65536, 40.8 t/s at 98304, and 8.9 t/s at 131072. Raising GGML_VK_SUBALLOCATION_BLOCK_SIZE to 4294967296 returns 131072 to 40.3 t/s. …gotchas/vulkan-suballoc-fragmentation-decode-cliff

verifiedhigh · 0.85

Vulkan decode on an AMD dGPU collapses 2-6x when Resizable BAR is disabled in BIOS

[major] driverWith ReBAR/SAM off in BIOS, llama.cpp's Vulkan backend runs decode at a fraction of the card's bandwidth. On an RX 7900 XTX with Qwen3.8-27B Q4_K_XL: 13.89 t/s for 128-token generation and 25.28 t/s at depth 4096, against 37.09 t/s and 812 t/s prompt processing when GGML_VK_DISAB…gotchas/vulkan-rebar-off-collapses-amd-decode

verifiedhigh · 0.85

Watermark detection fails exactly where the output matters most: low-entropy text, and about 25% of assistant-style responses

[major] securityA watermark works on essays and fails on the deployment's actual traffic. Mark My Words reports KGW watermarking Llama-2-7B-chat and Mistral-7B-Instruct with no perceivable quality loss and detection in fewer than 100 tokens on natural-language tasks, while explicitly struggling …gotchas/sec-watermark-low-entropy-output-is-where-detection-dies

verifiedhigh · 0.85

Watermark schemes are reverse-engineerable from a few black-box generations, so a watermark key is not a secret once the endpoint is live

[major] securityAn operator treats the watermark key as a durable secret. A user with ordinary API access and some patience recovers the scheme - which green-list rule applies where - and then knows exactly which substitutions remove the signal.gotchas/sec-watermark-stealable-from-black-box-generations

verifiedhigh · 0.90

Watermarking is a provenance signal, not a safety control - it does not moderate, refuse, or prevent anything

[minor] securityA deployment adds a watermark and reports the content-safety problem as solved. The watermark changes which tokens get sampled; it has no ability to refuse, to filter, or to detect harm at generation time. Harmful content is watermarked exactly as faithfully as benign content.gotchas/sec-watermark-provenance-is-not-a-compliance-control

verifiedhigh · 0.90

WebGPU in-browser inference is 2-5x slower than native GPU inference

[major] frameworkThe same model that runs at 30 tokens/sec on a native GPU may run at 6-15 tokens/sec in the browser via WebGPU, or 1-5 tokens/sec via WASM.gotchas/local-webgpu-performance-gap

verifiedmedium · 0.80

Weight loading is the one cold-start phase that gets worse the more replicas you add: N pods booting at once race each other for the same object store, and load times go from minutes to potentially hours

[major] configCold start is fine at low replica counts and collapses at high ones. One replica pulling weights from S3 is unremarkable; fifty or a hundred replicas pulling the same model at the same instant is a traffic jam, and the symptom is load times spiking 'from minutes to potentially ho…gotchas/dep-weight-pull-thundering-herd-is-superlinear-in-burst-size

verifiedhigh · 0.88

What happens when a queue fills is not one behaviour but three, and which one you get is a deployment choice: shed with a status code, block until the client times out, or OOM the box

[blocker] operationsThree different failure signatures for the same underlying condition of excess demand. (a) Explicit shed: immediate 429 or 503, visible in the client's error rate. (b) Silent unbounded growth: no error, latency grows without limit, connections pile up, and the client eventually g…gotchas/ops-queue-fill-modes-are-engine-specific

verifiedhigh · 0.90

What Intel hardware genuinely cannot do well relative to NVIDIA for LLM inference

[major] hardwareIntel GPUs and Gaudi accelerators have several fundamental limitations relative to NVIDIA for LLM inference: (1) No equivalent to CUDA's mature ecosystem - CUDA has 15+ years of optimization, while Intel's oneAPI/SYCL ecosystem is less mature. (2) No equivalent to TensorRT-LLM - …gotchas/xpu-intel-gpu-limitations-vs-nvidia

verifiedhigh · 0.85

What people regret buying: NVIDIA's own sm121 silicon variant, because the thing you paid the CUDA tax for is not available on it

[blocker] hardwareThe clearest regret record in the corpus is about a $3,000+ NVIDIA-branded box where the complaint is not performance and not price but that the advertised software platform is absent. u/goldcakes spent a week with a DGX Spark and was returning it: 'Just be careful when buying a …gotchas/sent-regret-dgx-spark-sm121-software-fragmentation

verifiedmedium · 0.70

What people regret buying: over-committing capex before validating demand - the recurring answer to 'should I buy this' is rent first

[minor] hardwareThe strongest cross-cutting consensus in the whole slice is not about any particular GPU. It is that the buy decision should be preceded by renting, and the most-voted piece of practical advice in a 'talk me out of buying an RTX Pro 6000' thread (141 upvotes, 333 comments) is 97 …gotchas/sent-regret-buying-without-validating-the-workload

verifiedmedium · 0.70

What people regret buying: parts whose inference engines are years behind the vendor's marketing, and the regret is measured in missing features not in tokens per second

[major] frameworkThe most valuable sentiment record here is a 242-upvote benchmark post whose conclusion is not about throughput at all. Asked on 4x RTX PRO 6000 (SM120), 397B NVFP4, the reporter's practical verdict is: reaching 50.5 tok/s - 'which should be 2-3x faster' - required '12 patches ac…gotchas/sent-regret-non-blackwell-engine-support-lags

verifiedmedium · 0.70

When the endpoint picker dies, failOpen routes requests straight to the model servers — and those servers have no queue, no priority and no fairness, so losing the control plane silently downgrades the serving policy

[major] operationsDuring an endpoint-picker election or restart, latency degrades and the fairness guarantees vanish for the duration: per-tenant limits stop applying, priority inversion returns, and requests that would have been queued centrally arrive directly at the model servers. Nothing error…gotchas/ops-failopen-router-failure-opens-a-bypass

verifiedhigh · 0.85

When the KV cache fills, engines preempt running requests and re-prefill them from token zero — the client sees a latency cliff, not an error

[major] operationsLatency climbs sharply for a subset of requests — typically the ones with long inputs and short outputs — while error rate stays at zero and the request is still 'served'. vLLM's log line shows a rising 'Preemptions: N' count; SGLang logs retractions. The affected requests have a…gotchas/ops-kv-cache-full-preempts-and-recomputes

verifiedhigh · 0.94

Whisper large-v3-turbo is pruned, not scaled

[major] configAssuming Whisper large-v3-turbo is simply a smaller version of large-v3 with proportional speedup across all components.gotchas/aud-gotcha-turbo-pruned-not-scaled

verifiedhigh · 0.90

Widely-cited benchmarks can measure the wrong thing: MT-Bench has low separation, LLM-as-a-judge has biases, and contaminated benchmarks measure memorization

[major] measurementYou cite a benchmark result (e.g., 'Model X scores 85 on MMLU') as evidence of capability, but the benchmark is contaminated, has low separation between models, or measures something other than what you care about.gotchas/meth-benchmarking-criticism-wrong-measurement

verifiedhigh · 0.85

With symmetric memory enabled, all TP schedulers can deadlock permanently and silently inside ncclCommWindowRegister, while /health still returns 200 and GPUs sit at 0%

[major] frameworkTL;DR from the reporter: with --enable-symm-mem on a multi-rank TP engine, every TP scheduler process deadlocks permanently and silently inside ncclCommWindowRegister. The trigger is routine - whenever the symmetric-memory mempool grows by a new segment at runtime, all ranks ente…gotchas/mgpu-symm-mem-window-registration-deadlocks-without-timeout

verifiedhigh · 0.85

Without locked clocks, cross-part and cross-run comparisons are meaningless - and short bursts measure the parked clock, not the machine

[major] measurementA fresh-process 8192x8192x256 bf16 matmul canary reads 25.0 TFLOPS in a quiet 0%-utilization window and 119.3 TFLOPS minutes later after a heavy workload - on the same machine, with pmset -g therm reporting clean and the host on AC high-power mode. A 2048^3 probe caps at ~19-30 T…gotchas/unlocked-clocks-make-cross-part-comparison-meaningless

verifiedhigh · 0.90

Most referenced

The records other records point at. Useful as an entry point: a heavily referenced accelerator or engine is one many decisions in this repo assume.

Record	Referenced by	Cites
NVIDIA's published NVLink figure is a bidirectional per-GPU aggregate, so every per-link or per-direction number in this family is a DERIVATION - reading the vendor number as per-direction doubles it and reading it as per-link multiplies it by the link count	2	20
A vendor 'PCIe peak bandwidth' number may be per-direction or bidirectional aggregate, and the two differ by exactly 2x, so a spec that does not state which one silently halves or doubles every host-bandwidth number	1	23
Average tokens/sec is not an SLO for LLM serving and will not detect the failure users actually report; the two that matter are TTFT and TPOT	1	27
The widely-repeated 3.6 TB/s per-GPU figure for NVLink 6 has no first-party NVIDIA source and NVIDIA's own page says 3,000 GB/s - an unsourced secondary number is 20% high and reads as more authoritative than the vendor page it contradicts	1	5
vLLM's --gpu-memory-utilization is a fraction of TOTAL VRAM, so KV cache block allocation fails even with memory visibly free	1	2