inference/research
verifiedmedium · 0.70minorupdated 2026-10-04

The practitioner answer to 'which inference engine is best' is a situation table, not a winner; asking for a ranking is answered by refusing the framing

[minor] toolchain

Fields

id
gotchas/sent-engine-choice-is-situational-not-a-ranking
Class
toolchain
Severity
minor
Concepts
engine-selectionconcurrent-usersgguftensor-parallelprefix-caching
Symptom
A buyer or engineer asks which inference engine is best. The high-scoring replies do not name one. The most-cited framing is that the question is malformed: 'It's often very silly to ask "which is better?" between tools. Which is better: a screwdriver or a hammer?' (15 upvotes). The most complete answer maps engines to situations and opens by saying 'It depends heavily on your situation.' The practical failure mode this record guards against is that a general-reader summary of 'the community prefers X' is simply false: in this corpus no single engine is preferred, and the preference is a function of user count, hardware vendor, quantization format, and whether the box is a laptop or a datacenter.
Root cause
The community has converged on a stable decision boundary that is cheaper to communicate than a ranking. Consensus boundary: llama.cpp for single user, consumer/edge, GGUF, CPU offload, mixed CPU+GPU, and AMD; vLLM or SGLang past roughly four concurrent users or any multi-GPU tensor parallel; TensorRT-LLM when you own the hardware and can absorb a build step. Two of the top answers in separate threads independently refuse the ranking framing, so this is not one contrarian.
Workaround
State the situation before naming an engine, and treat any 'X is the best engine' claim without a user-count and hardware qualifier as unsourced. See sent-llamacpp-single-user-consensus, sent-vllm-enterprise-default-consensus, sent-sglang-multinode-and-prefix-consensus for the individual cells.
notes
This is the governing record for the whole sentiment layer. Vote counts cited throughout these records are a popularity proxy observed at access time, never a measurement - see sent-vote-counts-are-a-popularity-proxy-not-a-measurement. Reader-facing consequence: 'what should I use' has no answer in this repo's corpus; only 'what should I use for a single user on a 3090 with a 70B GGUF' does.

Notes

This is the governing record for the whole sentiment layer. Vote counts cited throughout these records are a popularity proxy observed at access time, never a measurement - see sent-vote-counts-are-a-popularity-proxy-not-a-measurement. Reader-facing consequence: 'what should I use' has no answer in this repo's corpus; only 'what should I use for a single user on a 3090 with a 70B GGUF' does.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.