Papers (294)
Contributions with a mechanism and an adoption status. adoption is what separates a citation from an instruction, and speedup_reported always carries its baseline.
A Length-Extrapolatable Transformer
position-encoding · acl2022arXiv:2212.10554in-upstream-engineA transformer trained on short texts degrades sharply when evaluated on longer ones, and the usual framing (interpolate vs extrapolate positions) does not explain WHY: it lacks a quantity that predict…papers/ntk-aware-scaled-rope
contestedmedium · 0.78
Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LM)
other · iclr2025arXiv:2503.09573research-onlyDiffusion LMs permit parallel generation and controllability but lag in likelihood modelling and are locked to fixed-length generation, while autoregressive models cannot reuse compute across a parall…papers/block-diffusion-bd3-lm
contestedmedium · 0.65
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
speculative-decoding · icml2024arXiv:2402.02057in-upstream-engineSpeculative decoding requires an auxiliary draft MODEL or a data store, which is nontrivial to obtain, cannot generalise, and adds memory and engineering cost.papers/lookahead-parallel-decoding
contestedhigh · 0.85
Careless Whisper: Speech-to-Text Hallucination Harms
speech · other2024arXiv:2402.08021in-productionWhisper silently emits fluent text that was never in the audio, which is a safety and data-integrity problem in any unattended deployment.papers/careless-whisper
contestedlow · 0.60
Detecting Pretraining Data from Large Language Models
other · iclr2024arXiv:2310.16789research-onlyLLMs are trained on trillions of tokens, almost certainly including test data for widely reported benchmarks, but there's no way to know which data was included.papers/meth-detecting-pretraining-data
contestedmedium · 0.80
DINOv2: Learning Robust Visual Features without Supervision
multimodal · iclr2025arXiv:2304.07193in-productionSelf-supervised vision pretraining produced features that were brittle across distributions and needed finetuning per task.papers/dinov2
contestedhigh · 0.85
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
speculative-decoding · icml2024arXiv:2401.15077in-upstream-engineToken-level speculative drafting has irreducible uncertainty that caps acceptance rate; the draft model is forced to guess the next token in the same way the target does.papers/eagle-feature-level-speculative
contestedhigh · 0.90
Fast Inference from Transformers via Speculative Decoding
speculative-decoding · icml2023arXiv:2211.17192in-productionAutoregressive decoding is memory-bandwidth-bound: each new token requires a full pass over model weights, so decode FLOPs are wasted and accelerator compute sits idle.papers/leviathan-speculative-decoding
contestedmedium · 0.75
From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
other · icml2025arXiv:2406.11939in-productionManual curation of high-quality LLM benchmarks is expensive and doesn't scale.papers/meth-arena-hard-benchbuilder
contestedhigh · 0.85
High Fidelity Neural Audio Compression (EnCodec)
speech · iclr2024arXiv:2210.13438in-productionNeural codecs either matched speech-codec bitrates poorly or needed one model per bitrate, and training was slow and unstable.papers/encodec
contestedhigh · 0.85
Large Language Diffusion Models (LLaDA)
other · neurips2025arXiv:2502.09992in-upstream-engineLLM capability was treated as depending on autoregressive modelling, so the entire premise was untested at 8B scale.papers/llada-large-language-diffusion-models
contestedhigh · 0.88
Let's Verify Step by Step
inference-time-compute · iclr2024arXiv:2305.20050in-productionEven state-of-the-art models regularly produce logical mistakes, and it is unclear which of two supervision regimes - OUTCOME supervision (feedback on the final result only) or PROCESS supervision (fe…papers/process-supervision-prm800k
contestedmedium · 0.80
Linear Transformers Are Secretly Fast Weight Programmers
attention · icml2021arXiv:2102.11174in-upstream-engineLinearised softmax attention updates its state by a purely additive outer product k v^T, and the resulting fixed-size state has a memory capacity that cannot support precise recall.papers/linear-transformers-fast-weight-programmers
contestedhigh · 0.85
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding
speculative-decoding · iclr2025arXiv:2408.11049research-onlyConventional wisdom says speculative decoding only helps at SMALL batch sizes, so it is useless in the high-throughput serving regime that production actually runs in.papers/magicdec-long-context-specdec
contestedhigh · 0.85
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
speculative-decoding · icml2024arXiv:2401.10774in-upstream-engineSpeculative decoding needs a separate draft model, which is costly to acquire, train, and serve; maintaining a second model in memory also inflates memory footprint.papers/medusa-multiple-decoding-heads
contestedhigh · 0.85
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
moe · mlsys2023arXiv:2211.15841research-onlyMoE GPU stacks force a choice between dropping tokens that overflow an expert's capacity or computing over a padded batch, so realistic routing distributions pay a real quality or efficiency tax.papers/megablocks
contestedhigh · 0.85
MTEB: Massive Text Embedding Benchmark
embedding · other2023arXiv:2210.07316in-productionText embeddings were evaluated on a couple of STS datasets, so field progress could not be tracked and models did not generalise across task types.papers/mteb
contestedhigh · 0.85
OLMoE: Open Mixture-of-Experts Language Models
moe · iclr2025arXiv:2409.02060in-upstream-engineMoE research was closed: without released weights, data and training logs, nobody could audit whether a sparse model's routing actually specialises.papers/olmoe
contestedhigh · 0.88
SALMONN: Towards Generic Hearing Abilities for Large Language Models
speech · iclr2024arXiv:2310.13289research-onlyLLMs are text-only; cascaded ASR pipelines lose paralinguistic information and cannot handle music or audio events.papers/salmonn
contestedhigh · 0.85
Structured Denoising Diffusion Models in Discrete State-Spaces (SEDD)
other · neurips2021arXiv:2107.03006in-upstream-engineDiffusion models were defined in continuous Euclidean space, where adding Gaussian noise and denoising is well defined; text lives in a discrete state space, so naively applying diffusion to tokens is…papers/sedd-score-entropy-discrete-diffusion
contestedhigh · 0.90
The LLM Surgeon
pruning · iclr2024arXiv:2312.17244research-onlyData-driven compression of pretrained LLMs usually treats pruning and weight-update as separate problems, and unstructured pruning of a large model was thought to need full second-order computation.papers/llm-surgeon
contestedmedium · 0.82
Turning Whisper into a Real-Time Transcription System (Whisper-Streaming)
speech · other2023arXiv:2307.14743in-productionWhisper is a non-streaming 30s-window model, so naive streaming to it either blocks for 30s or transcribes unstable prefixes.papers/whisper-streaming
contestedmedium · 0.80
Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisions (PRISM)
serving-systems · arxiv2026arXiv:2609.35569research-onlyLLM serving has energy, carbon, water and biodiversity impacts, but these are evaluated in isolation, so it is unclear when they point at DIFFERENT serving decisions and silently conflict.papers/sustainability-dimensions-llm-serving
draftlow · 0.62
Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays
serving-systems · arxiv2026arXiv:2609.20874research-onlyNew serving replicas take 2-10 MINUTES to start because multi-gigabyte weights must load, which makes purely reactive autoscaling structurally late — no amount of reaction speed can beat the actuation…papers/k8s-predictive-autoscaling-llm
draftlow · 0.60
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
serving-systems · arxiv2024arXiv:2401.08671in-productionExisting frameworks force a choice between throughput and latency for LLM generation, and this hurts most with long prompts where the prefill phase dominates.papers/deepspeed-fastgen
draftmedium · 0.70
Fast Distributed Inference Serving for Large Language Models
scheduling · arxiv2023arXiv:2305.05920research-onlyRun-to-completion processing of LLM requests causes head-of-line blocking: one long request stalls everything behind it, so latency is high even when aggregate throughput is fine.papers/fastserve-llm
draftlow · 0.62
Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads (TetriInfer)
scheduling · arxiv2024arXiv:2401.11181research-onlyA production cluster mixes downstream tasks with very different prompt and output length distributions, and co-locating them means prefill-heavy and decode-heavy requests interfere destructively.papers/tetriinfer
draftlow · 0.60
Measured Joules, Learned Routes: Learning to Route for Energy-Efficient LLM Serving
routing · arxiv2026arXiv:2609.23085research-onlyMost queries do not need the largest model, and routing them to a high-capability model burns unnecessary compute AND energy — but energy is not what existing quality routers optimise.papers/energy-aware-routing
draftlow · 0.58
PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale
scheduling · arxiv2026arXiv:2609.33224unknownAgentic workloads reuse KV cache across long shared prefixes, so naive request consolidation onto fewer instances trades away cache locality and prefill/decode interference, wrecking both SLO attainme…papers/packserve-agentic-scheduling
draftmedium · 0.65
Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications
cluster · arxiv2019arXiv:1902.04610abandonedGPUs lack fine-grained sharing primitives, so time-sharing and preemption between DL applications are expensive, and when one application cannot fill a GPU the device is wasted for everyone else.papers/salus
draftlow · 0.60
Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
serving-systems · arxiv2026arXiv:2609.18112research-onlyExisting multi-tenant isolation equalises client THROUGHPUT over the long run (queueing/batching fairness) but gives no LATENCY guarantee, so a well-behaved client can still see its token-level latenc…papers/token-latency-fairness
draftlow · 0.62
A Simple and Effective Pruning Approach for Large Language Models
pruning · iclr2024arXiv:2306.11695research-onlyMagnitude pruning underperforms on LLMs, while second-order pruning methods need weight reconstruction that is expensive at billion-parameter scale.papers/wanda
verifiedhigh · 0.85
Accelerating Large Language Model Decoding with Speculative Sampling
speculative-decoding · arxiv2023arXiv:2302.01318in-productionSame memory-bandwidth bottleneck as Leviathan et al.; addressed with an alternative to sequence-level drafting.papers/chen-speculative-sampling-deepmind
verifiedhigh · 0.90
Accelerating Production LLMs with Combined Token/Embedding Speculators
speculative-decoding · arxiv2024arXiv:2404.19124in-productionAcademic speculative-decoding papers optimise draft quality in isolation, but a production system must accelerate an ALREADY highly optimised base-model implementation, where the easy headroom is gone…papers/production-token-embedding-speculators
verifiedmedium · 0.80
Accelerating Sparse Deep Neural Networks
sparsity · arxiv2021arXiv:2104.08378in-productionSparsity research mostly targets high sparsity levels that no shipping matrix-math unit supports, so it produces compression without speedup.papers/sparse-tensor-cores
verifiedhigh · 0.88
AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure
cluster · arxiv2025arXiv:2504.03648in-upstream-engineExisting cloud-native serving stacks are general-purpose and treat LLM inference as an ordinary workload, so every layer above the engine has to be retrofitted - which is why multi-tenant LLM deployme…papers/w4p-aibrix-multitenant-serving-control-plane
verifiedmedium · 0.70
AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
serving-systems · osdi2023arXiv:2302.11665research-onlyOne-model-per-GPU serving wastes each GPU's memory bandwidth whenever a single model cannot saturate it, and high-bandwidth memory designs are wasted on models too small to fill them.papers/alpaserve
verifiedhigh · 0.85
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
quantization · arxiv2026arXiv:2606.04980research-onlyMixed-precision MoE quantization allocates bit widths from a calibration set, but for frontier MoE models the training data is proprietary so the calibration set is an imperfect surrogate and mis-rank…papers/alphaq
verifiedmedium · 0.75
Apple Intelligence Foundation Language Models: Tech Report 2025 (PT-MoE server model)
moe · arxiv2025arXiv:2507.13575in-productionA private-cloud inference tier has different constraints from a public one: it must serve a fixed user population on fixed hardware, so cost per request matters as a first-class objective rather than …papers/w4p-apple-ptmoe-server-model
verifiedmedium · 0.80
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
quantization · mlsys2024arXiv:2310.19102research-onlyINT8 weight-activation schemes underuse modern GPUs (which have native 4-bit integer operators), and W4A4 breaks because of block outliers.papers/paper-atom
verifiedhigh · 0.85
Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
speech · icml2024arXiv:2402.01831research-onlyExisting audio LMs could not combine broad audio understanding with in-context learning and multi-turn dialogue.papers/audio-flamingo
verifiedhigh · 0.85
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
quantization · mlsys2024arXiv:2306.00978in-productionWeight-only quantization to 3-4 bits either degrades quality badly or requires expensive per-channel/group grid search at quantization time; naive round-to-nearest INT4 is much worse than RTN at INT8.papers/paper-awq
verifiedhigh · 0.85
BitNet: Scaling 1-bit Transformers for Large Language Models
quantization · arxiv2023arXiv:2310.11453research-only1-bit weights for LLMs had not been shown to scale with a normal power law or to be stable in training.papers/bitnet
verifiedhigh · 0.85
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
multimodal · icml2023arXiv:2301.12597in-productionEnd-to-end vision-language pretraining was prohibitively expensive because both towers had to be trained jointly.papers/blip2
verifiedhigh · 0.85
Blockwise Parallel Transformer for Large Context Models
attention · neurips2023arXiv:2305.19370research-onlySelf-attention memory and the feedforward layer's activation memory both limit the sequence length a single device can train on, so long-sequence tasks need awkward chunking or approximations.papers/blockwise-parallel-transformer
verifiedhigh · 0.85
Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
retrieval · sigmod2025arXiv:2502.15734research-onlyIn RAG the same retrieved chunk recurs across many queries, but engines cannot reuse its KV: a chunk appears at an ARBITRARY POSITION in the prompt with ARBITRARY PRECEDING CONTEXT, so there is no sha…papers/w4p-cachecraft-rag-chunk-cache
verifiedhigh · 0.90
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
kv-cache · arxiv2024arXiv:2405.16444in-upstream-enginePrecomputed KV caches can only be reused when the cached text is the input PREFIX. In RAG the retrieved chunks are scattered through the prompt and are not prefixes, so precomputed caches ignore each …papers/cacheblend-rag-kv-fusion
verifiedhigh · 0.85
CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving
kv-cache · sigcomm2024arXiv:2310.07240research-onlyReusing a cached KV context across requests saves prefill compute but the KV tensors are large, so fetching them over the network dominates context-loading delay.papers/paper-cachegen
verifiedhigh · 0.88
Carbon Emissions and Large Neural Network Training
other · arxiv2021arXiv:2104.10350in-productionML compute demand has grown fast with real energy and carbon costs, but energy use is hard to estimate without detailed information, so the cost is invisible and the optimisation target is undefined.papers/carbon-emissions-large-neural-training
verifiedhigh · 0.85
Cascade Speculative Drafting for Even Faster LLM Inference (CS Drafting)
speculative-decoding · neurips2024arXiv:2312.11462research-onlySpeculative decoding's drafting phase is itself slow autoregressive generation and spends equal time on every draft token regardless of how much that token matters to the target's acceptance.papers/w4p-cascade-speculative-drafting
verifiedhigh · 0.85
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
inference-time-compute · neurips2022arXiv:2201.11903in-productionLarge language models struggle with multi-step arithmetic, commonsense, and symbolic reasoning, and standard prompting gives the model no room to work through intermediate steps.papers/chain-of-thought-prompting
verifiedhigh · 0.90
Clipper: A Low-Latency Online Prediction Serving System
serving-systems · nsdi2017arXiv:1612.03079in-productionML frameworks handle training and not deployment, so every application reinvents a low-latency prediction serving layer over its own model, inconsistently and without batching or latency control.papers/clipper
verifiedhigh · 0.85
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
retrieval · sigir2020arXiv:2004.12832in-productionBERT-based cross-encoder rerankers must feed every query-document pair through a large network, which scales with the number of candidates.papers/colbert
verifiedhigh · 0.85
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
retrieval · naacl2022arXiv:2112.01488in-productionLate interaction's multi-vector representations inflate the model's space footprint by an order of magnitude, limiting index size.papers/colbertv2
verifiedhigh · 0.85
Conformer: Convolution-augmented Transformer for Speech Recognition
speech · interspeech2020arXiv:2005.08100in-productionTransformers model global context but miss the local patterns of audio; CNNs do the opposite.papers/conformer
verifiedhigh · 0.85
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
speech · icml2022arXiv:2202.03555in-upstream-engineEvery modality had its own bespoke self-supervised objective, so nothing transferred between speech, vision and text.papers/data2vec
verifiedhigh · 0.85
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
distillation · nature2025arXiv:2501.12948in-productionReasoning ability in LLMs has depended on large volumes of human-annotated chains of thought, which does not scale and caps the model's ceiling on hard problems.papers/paper-deepseek-r1
verifiedhigh · 0.85
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
kv-cache · arxiv2024arXiv:2405.04434in-upstream-engineFor autoregressive decode the KV cache is the memory bottleneck, and it gets worse with context length, batch size and every KV head - GQA reduces it only by an integer factor while the cache still sc…papers/deepseek-v2-mla
verifiedhigh · 0.90
DeepSeek-V3 Technical Report
moe · arxiv2024arXiv:2412.19437in-productionFrontier-scale MoE LLMs cost a great deal to train and to serve, and the two usual levers - more expert parallelism, more fine-grained routing - make training stability and inference communication har…papers/deepseek-v3-report
verifiedhigh · 0.88
DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (introducing DeepSeek Sparse Attention, DSA)
attention · arxiv2025arXiv:2512.02556in-upstream-engineLong-context attention cost grows with sequence length, and the gap between a model's ADVERTISED context and what it can actually attend to cheaply is the dominant constraint on deploying a frontier o…papers/w4p-deepseek-sparse-attention-dsa
verifiedmedium · 0.80
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
moe · acl2024arXiv:2401.06066in-productionGShard-style top-k-of-N MoE keeps experts overlapping and redundant, so DeepSeekMoE 16B merely matched a 2.9B GShard and still lost to a same-size dense model instead of beating it.papers/deepseekmoe
verifiedhigh · 0.90
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
serving-systems · sc222022arXiv:2207.00032in-productionInference at model scale needs kernels specialised per model family, and hand-writing them for every architecture does not scale as new transformer variants keep appearing.papers/deepspeed-inference
verifiedmedium · 0.82
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
moe · icml2022arXiv:2201.05596in-upstream-engineMoE's training cost saving did not carry over to serving, because inference keeps all experts resident and pays an all-to-all every layer, so a dense model can be cheaper per token to serve.papers/deepspeed-moe
verifiedhigh · 0.85
Dense Passage Retrieval for Open-Domain Question Answering
retrieval · emnlp2020arXiv:2004.04906in-productionOpen-domain QA retrieval used TF-IDF/BM25, which fails on semantic paraphrase and requires hand-tuned sparse statistics.papers/dpr
verifiedhigh · 0.85
Differential Transformer
attention · iclr2025arXiv:2410.05258research-onlySoftmax attention is a convex combination of value vectors, so noise or distracting context in the values is averaged into every output - the mechanism by which irrelevant context degrades key-informa…papers/diff-transformer
verifiedmedium · 0.84
Diffusion-LM Improves Controllable Text Generation
other · neurips2022arXiv:2205.14217research-onlyAutoregressive text generation is strictly left-to-right, so a language model cannot revise or control a span it has already emitted; controllability was limited to steering generation forward from a …papers/diffusion-lm
verifiedhigh · 0.85
DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training
attention · colm2024arXiv:2310.03294research-onlyTraining long-context LLMs needs the attention computation sharded across devices, and existing approaches pay for it either in sequence-length limits (attention must be exact per sequence) or in imba…papers/distflashattn
verifiedhigh · 0.86
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
distillation · acl2023arXiv:2305.02301research-onlyFine-tuning and distilling small models both need large amounts of training data to match a prompted LLM.papers/distilling-step-by-step
verifiedhigh · 0.85
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
serving-systems · osdi2024arXiv:2401.09670research-onlyColocating prefill and decode in one batch couples their resource-allocation and parallelism plans, so a serving system must trade TTFT against inter-token latency instead of meeting both SLOs.papers/paper-distserve
verifiedhigh · 0.92
Dream 7B: Diffusion Large Language Models
other · arxiv2025arXiv:2508.15487in-upstream-engineAutoregressive models generate strictly sequentially, so decoding cannot be parallelised and in-context infilling or arbitrary-order generation is awkward.papers/dream-7b
verifiedhigh · 0.85
DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
long-context · iclr2025arXiv:2410.10819research-onlyLong-context inference holds a large KV cache and does full attention over it, even though many heads in a long-context model are not retrieval heads and do not need the full cache.papers/duoattention
verifiedhigh · 0.88
DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurance
moe · arxiv2026arXiv:2509.07379research-onlyWith routed experts in host RAM, prefill and decode want opposite schedules. Prefill activates experts densely across many tokens, so a cache that tries to hold them blows peak GPU memory. Decode acti…papers/duoserve-moe-dual-phase-expert-prefetch
verifiedmedium · 0.80
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
multimodal · neurips2021arXiv:2106.02034in-upstream-engineVision transformers compute attention over every patch even though the final prediction depends on only a subset of informative tokens.papers/dynamicvit
verifiedhigh · 0.85
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
speculative-decoding · emnlp2024arXiv:2406.16858in-upstream-engineStatic draft trees (as in EAGLE-1, SpecInfer) implicitly assume a draft token's acceptance rate depends only on its POSITION in the tree, which is false.papers/eagle-2-dynamic-draft-trees
verifiedhigh · 0.90
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
speculative-decoding · neurips2025arXiv:2503.01840in-upstream-engineEAGLE and EAGLE-2 gain little from more draft-model training data, because feature prediction puts a ceiling on how much the draft model can learn.papers/eagle-3-training-time-test
verifiedhigh · 0.88
EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimization
quantization · arxiv2025arXiv:2506.13329research-onlyDense PTQ methods ignore that MoE routing is a DISCRETE decision: small quantization perturbations can change which experts are selected and thereby change the computation path.papers/eaquant
verifiedhigh · 0.85
Efficient Memory Management for Large Language Model Serving with PagedAttention
serving-systems · sosp2023arXiv:2309.06180in-productionExisting serving systems reserve a large, fixed, CONTIGUOUS maximum sequence-length block of KV memory per request, so most of that memory is wasted whenever the actual sequence is shorter - capping t…papers/pagedattention-vllm
verifiedhigh · 0.95
Efficient Streaming Language Models with Attention Sinks
long-context · iclr2024arXiv:2309.17453research-onlyStreaming applications (multi-round dialogue) need bounded KV memory, but simply keeping the most recent KVs - sliding-window attention - FAILS once the text exceeds the cache size, and popular LLMs c…papers/streamingllm
verifiedhigh · 0.87
Efficiently Modeling Long Sequences with Structured State Spaces
ssm · iclr2022arXiv:2111.00396research-onlyThe state space model x' = Ax + Bu, y = Cx + Du can represent long-range dependencies, but computing it naively is prohibitively expensive in both time and memory - so the theoretical strength of the …papers/s4
verifiedhigh · 0.90
Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
speech · icassp2021arXiv:2010.10759in-upstream-engineStreaming ASR with limited latency needs long-range context for accuracy but cannot afford full self-attention over unbounded history.papers/emformer
verifiedhigh · 0.85
ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp
serving-systems · arxiv2025arXiv:2512.10576research-onlyEven after DeepSeek Sparse Attention improves overall throughput, the DECODE stage of prefill-decode disaggregation remains a bottleneck: the Latent-Cache grows LINEARLY with sequence length while GPU…papers/w4p-ess-latent-cache-offload-deepseek
verifiedmedium · 0.75
Extending Context Window of Large Language Models via Positional Interpolation
position-encoding · arxiv2023arXiv:2306.15595in-upstream-engineExtending a transformer's context window by letting it see position indices beyond those it was trained on (extrapolation) produces catastrophically high attention scores that can destroy self-attenti…papers/position-interpolation
verifiedhigh · 0.85
Extreme Compression of Large Language Models via Additive Quantization
quantization · icml2024arXiv:2401.06118in-upstream-engine2-3 bit weight quantization of LLMs: existing methods degrade badly below 3 bits because uniform scalar grids cannot represent the weight distribution.papers/paper-aqlm
verifiedhigh · 0.88
Fast Matrix Multiplications for Lookup Table-Quantized LLMs
quantization · emnlp2024arXiv:2407.10960research-onlyNon-uniform lookup-table quantization at awkward bit widths (e.g. 3 bits) makes the dequant and matmul kernel slow, which can eat the entire bandwidth saving.papers/paper-flute
verifiedhigh · 0.85
Fast Transformer Decoding: One Write-Head is All You Need
kv-cache · arxiv2019arXiv:1911.02150in-upstream-engineAutoregressive decoding of a transformer is memory-bandwidth-bound, not compute-bound, because every step re-reads the large key and value tensors from HBM - and those tensors grow with the number of …papers/multi-query-attention
verifiedhigh · 0.88
FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
speech · icassp2021arXiv:2010.11148in-upstream-engineStreaming ASR emits words late because transducer training optimises full-sequence probability, not emission delay - and existing penalties need external alignments.papers/fastemit
verifiedhigh · 0.85
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
moe · iclr2025arXiv:2402.07033research-onlyMoE models are too large for a single GPU, and existing CPU-offload solutions either pay enormous CPU-to-GPU data movement overhead or ignore that CPU and GPU have genuinely different strengths.papers/w4p-fiddler-cpu-gpu-moe-orchestration
verifiedhigh · 0.90
Flamingo: a Visual Language Model for Few-Shot Learning
multimodal · neurips2022arXiv:2204.14198research-onlyFitting a new task from a handful of examples required fine-tuning on thousands of times more task data.papers/flamingo
verifiedhigh · 0.85
FlashAttention on Tenstorrent's Wormhole Architecture
attention · otherno arXivpapers/tt-metal-flashattention-wormhole
verifiedmedium · 0.80
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
attention · iclr2024arXiv:2307.08691in-upstream-engineFlashAttention reached only 25-40% of peak FLOP/s because its work partitioning across thread blocks and warps caused low occupancy and redundant shared-memory traffic.papers/flashattention-2
verifiedhigh · 0.93
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
attention · neurips2024arXiv:2407.08608in-upstream-engineOn Hopper, FlashAttention-2's synchronous schedule wastes the tensor cores: the softmax and the matmuls cannot overlap, so the FP8 tensor cores sit idle waiting on non-matmul work.papers/flashattention-3
verifiedhigh · 0.90
FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
attention · mlsys2026arXiv:2603.05451in-upstream-engineFlashAttention-3's design assumed symmetric hardware scaling, but Blackwell moved the bottleneck: tensor core throughput doubles from H100 to B200 while shared-memory bandwidth, the exponential unit a…papers/flashattention-4
verifiedhigh · 0.85
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
attention · neurips2022arXiv:2205.14135in-upstream-engineStandard attention is memory-bound: it materialises the full N x N attention matrix in HBM, so the cost of self-attention is dominated by HBM reads/writes rather than by FLOPs, and long sequences OOM.papers/flashattention
verifiedhigh · 0.95
FlashDecoding++: Faster Large Language Model Inference on GPUs
attention · mlsys2024arXiv:2311.01282in-upstream-engineAttention at decode time is memory-bandwidth-bound on the KV cache, and flat GEMMs of differing shapes hit different hardware bottlenecks - so a single schedule cannot be optimal across batch size, he…papers/flashdecoding
verifiedmedium · 0.82
FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems
serving-systems · mlsys2026arXiv:2601.00227research-onlyLLM agents can generate GPU kernels, but a kernel an agent writes cannot be used unless it is specified identically, benchmarked with correctness awareness, and actually deployed - so agent kernel gen…papers/flashinfer-bench
verifiedmedium · 0.75
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
serving-systems · mlsys2025arXiv:2501.01005in-upstream-engineServing attention is not one problem: paged KV layouts, variable batch, head counts and context lengths each stress a different part of the memory hierarchy, so a single hardcoded attention kernel lea…papers/flashinfer-paper
verifiedhigh · 0.92
FlexAttention: A Programming Model for Generating Optimized Attention Kernels
attention · mlsys2025arXiv:2412.05496in-upstream-engineAttention variants beyond softmax attention (Alibi, document masking, sliding window, PagedAttention) are hard to implement fast because each needs a bespoke fused kernel, so the field suffers a 'soft…papers/flexattention
verifiedhigh · 0.88
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
serving-systems · icml2023arXiv:2303.06865research-onlyServing a large model on a single commodity GPU requires offloading weights and KV cache to CPU, disk and even across GPUs, and that offloading is dominated by I/O and memory-transfer latency rather t…papers/flexgen-single-gpu-high-throughput
verifiedhigh · 0.90
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance
routing · tmlr2023arXiv:2305.05176research-onlyAPI pricing across strong models differs by orders of magnitude, so naively calling an expensive model for every query is economically indefensible at scale.papers/frugalgpt
verifiedmedium · 0.82
Gated Delta Networks: Improving Mamba2 with Delta Rule
attention · iclr2025arXiv:2412.06464in-upstream-engineLinear Transformers underperformed attention on retrieval and long-context tasks; prior work separately proposed gating for memory control and the delta rule for precise updates, but neither alone was…papers/gated-deltanet
verifiedhigh · 0.90
Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
embedding · acl2026arXiv:2608.23806in-productionHigh-quality embedders were too slow and expensive to serve at high throughput, forcing quality-versus-cost compromises.papers/giga-embeddings
verifiedhigh · 0.85
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
quantization · arxiv2022arXiv:2210.17323in-productionPost-training quantization of 100B+ parameter generative models: existing one-shot quantizers lose too much accuracy at 3-4 bits and cannot finish on a feasible compute budget.papers/paper-gptq
verifiedhigh · 0.85
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
kv-cache · emnlp2023arXiv:2305.13245in-upstream-engineMulti-query attention shrinks the decode-time KV cache to a single KV head but degrades quality, and it cannot be adopted after the fact because there is no cheap way to convert an already-trained mul…papers/grouped-query-attention
verifiedhigh · 0.93
Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
attention · arxiv2024arXiv:2402.19427research-onlyRNNs have fast inference and scale efficiently on long sequences but are hard to train and hard to scale, while Transformers train well but decode slowly with cost that grows with context.papers/hawk-griffin-gated-linear-recurrence
verifiedmedium · 0.80
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
moe · iclr2021arXiv:2006.16668in-upstream-engineScaling MoE to hundreds of billions of parameters required hand-written expert-parallel sharding code, so no team could practically adopt sparse MoE.papers/gshard
verifiedhigh · 0.85
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
kv-cache · neurips2023arXiv:2306.14048research-onlyFor long-content generation the KV cache stored in GPU memory scales linearly with sequence length and batch size, adding a large transient-memory cost on top of model parameters.papers/h2o
verifiedhigh · 0.90
Hardware-Aware FP4 FlashAttention-4
quantization · arxiv2026arXiv:2609.04105research-onlyBlackwell's native FP4 tensor cores do not translate into faster attention, because once the matrix products shrink the softmax conversion and the on-chip serial dependencies dominate the kernel.papers/flashattention-4-fp4
verifiedmedium · 0.72
HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
serving-systems · mlsys2024arXiv:2403.01164research-onlyOn a device whose DRAM cannot hold the model, offloading weights from storage is I/O-bound, so latency collapses; offloading to a CPU adds PCIe transfer that dominates the compute of the layers themse…papers/hetegen-heterogeneous-parallel-inference
verifiedhigh · 0.92
HiNa-MoE: High-Performance, Non-Intrusive MoE Inference on CPUs with Matrix Engines
moe · other2026arXiv:2610.05123research-onlyWhen routed experts live in host RAM and only a few tokens hit each one, copying an expert to the GPU costs more than computing it, so the expert FFN stays on the CPU and sits on the critical path. Th…papers/hina-moe-amx-general-layout
verifiedmedium · 0.74
HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
moe · arxiv2024arXiv:2411.01433research-onlyWith routed experts in host RAM or on SSD, a cache miss costs more than the layer's compute, so prefetch and LRU/LFU never hide it. On one Mixtral-8x7B layer the authors time expert loading at about 8…papers/hobbit-mixed-precision-expert-offload
verifiedmedium · 0.80
HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition
speech · interspeech2024arXiv:2204.06328research-onlySelf-supervised encoders are deep enough that a large share of utterances finish long before the last layer, and the remaining layers are wasted compute.papers/hubert-ee
verifiedhigh · 0.85
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
speech · taslp2021arXiv:2106.07447in-upstream-engineSpeech has no lexicon and no fixed segmentation, so masked prediction needs a target definition before the loss can be written.papers/hubert
verifiedhigh · 0.85
HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
moe · other2025arXiv:2504.05897research-onlyWith routed experts in host RAM and only a slice of them cached in VRAM, a cache miss is usually paid as a host-to-device copy. On a memory-bound decode step that copy costs more than computing the ex…papers/hybrimoe-hybrid-cpu-gpu-scheduling
verifiedmedium · 0.80
Hyena Hierarchy: Towards Larger Convolutional Language Models
attention · icml2023arXiv:2302.10866research-onlyExisting subquadratic attention replacements - low-rank and sparse approximations - have to be mixed with dense attention layers to match Transformer quality, so there is no single subquadratic operat…papers/hyena
verifiedhigh · 0.88
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
serving-systems · osdi2024arXiv:2406.19707research-onlyIn offloading-based serving, fetching the KV cache from host memory stalls the compute core; fetching all of it wastes bandwidth that only a few entries would have used.papers/infinigen
verifiedhigh · 0.85
InternVL-X: Advancing and Accelerating the InternVL Series with Efficient Visual Token Compression
multimodal · arxiv2025arXiv:2503.21307in-productionVisual tokens treated as a text prefix make LLM prefill and KV memory scale with image area, dominating VLM cost.papers/internvl-x
verifiedhigh · 0.85
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
multimodal · cvpr2024arXiv:2312.14238in-productionVision foundation models lagged language models in scale, capping multimodal capability.papers/internvl
verifiedhigh · 0.85
Introducing v0.5 of the AI Safety Benchmark from MLCommons
security · arxiv2024arXiv:2404.12241research-onlyAI safety benchmarks are needed to assess risks of chat-tuned language models, but existing benchmarks are ad hoc and don't cover the full risk surface.papers/meth-ai-safety-benchmark-v05
verifiedhigh · 0.85
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
ssm · arxiv2024arXiv:2408.12570in-upstream-engineScaling a hybrid Transformer-Mamba MoE to larger sizes exposed that naive hybrid and MoE choices do not survive at scale, and the models needed a serving-friendly expert quantisation to be affordable.papers/jamba-1-5
verifiedhigh · 0.85
Jamba: A Hybrid Transformer-Mamba Language Model
ssm · arxiv2024arXiv:2403.19887in-upstream-enginePure Mamba models lose the exact recall of attention; pure attention models pay a KV cache that grows with context and cannot fit long contexts in memory.papers/jamba-hybrid
verifiedhigh · 0.90
JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
structured-output · arxiv2025arXiv:2501.10868in-upstream-engineConstrained decoding has standardised on JSON Schema and grown in adoption, yet nobody has systematically evaluated whether the shipped frameworks actually WORK on real-world schemas, or what they cos…papers/jsonschemabench-structured-output-eval
verifiedhigh · 0.90
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
other · neurips2023arXiv:2306.05685in-productionEvaluating LLM chat assistants on open-ended questions is expensive and existing benchmarks don't measure human preferences well.papers/meth-llm-as-judge-mt-bench
verifiedhigh · 0.90
Kimi K2: Open Agentic Intelligence
moe · arxiv2025arXiv:2507.20534in-upstream-engineOpen-weight MoE models large enough to be useful for agentic software work are expensive to serve, and their serving cost is dominated by FFN weight traffic rather than by attention - so the MoE total…papers/w4p-kimi-k2-moe-agentic
verifiedmedium · 0.80
Kimi Linear: An Expressive, Efficient Attention Architecture
attention · arxiv2025arXiv:2510.26692in-upstream-engineLinear attention has been adopted mainly for efficiency at the cost of quality, and the claim that it can beat full attention under FAIR comparisons had not been established for short-context, long-co…papers/kimi-linear
verifiedhigh · 0.88
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
kv-cache · icml2024arXiv:2402.02750research-onlyKV cache quantization is applied per-token to both keys and values, which is exactly wrong for keys and wastes most of the value cache's headroom.papers/kivi
verifiedhigh · 0.85
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
kv-cache · neurips2024arXiv:2401.18079research-onlyKV cache activations become the dominant memory consumer at long context, and existing solutions fail to represent them accurately below 4 bits.papers/paper-kvquant
verifiedhigh · 0.85
Large Language Models Cannot Self-Correct Reasoning Yet
inference-time-compute · iclr2024arXiv:2310.01798in-productionSelf-correction - asking a model to check and fix its own output - is widely assumed to improve reasoning without any external feedback or retraining.papers/llms-cannot-self-correct-reasoning
verifiedhigh · 0.90
LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
speculative-decoding · acl2024arXiv:2404.16710research-onlySpeculative decoding needs a separate draft model, doubling model memory; and early-exit inference needs auxiliary exit heads, which cost accuracy and add modules.papers/layerskip-self-speculative
verifiedhigh · 0.90
Learning to Compress Prompts with Gist Tokens
long-context · neurips2023arXiv:2304.08467research-onlyTask-specific prompt optimisation normally means either re-prompting per task or retraining the model per task, so a reused system prompt is recomputed on every request and long prompts are expensive …papers/gist-tokens
verifiedmedium · 0.83
Learning Transferable Visual Models From Natural Language Supervision (CLIP)
multimodal · icml2021arXiv:2103.00020in-productionVision encoders trained on a fixed label set cannot represent any concept outside that set without new labelled data.papers/clip
verifiedhigh · 0.85
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
long-context · arxiv2024arXiv:2404.07143research-onlyAn LLM serving long or unbounded input either pays quadratic cost or must truncate, and the usual compressions (sliding window, sparse attention) lose the distant context that mattered.papers/infini-attention
verifiedhigh · 0.85
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
attention · arxiv2024arXiv:2401.04658abandonedLinear attention is theoretically linear-time, but the CUMULATIVE SUMMATION needed to compute it destroys the theoretical advantage in the causal (autoregressive) setting - so in practice linear atten…papers/lightning-attention-2
verifiedhigh · 0.85
Linformer: Self-Attention with Linear Complexity
attention · arxiv2020arXiv:2006.04768abandonedStandard self-attention uses O(n^2) time and space in sequence length, making long-sequence training and deployment prohibitively costly.papers/linformer
verifiedhigh · 0.85
Llama Guard 3-1B-INT4: Compact and Efficient Safeguard for Human-AI Conversations
security · arxiv2024arXiv:2411.17713in-productionLLM-based guardrails are a SECOND MODEL ON THE CRITICAL PATH of every conversation, so their size and latency are a direct tax on every interaction - and the 8B-class guard models are too heavy to dep…papers/llama-guard-3-1b-int4
verifiedhigh · 0.85
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
security · arxiv2023arXiv:2312.06674in-productionContent moderation at serving time requires classifying BOTH the user prompt and the model's response, and existing moderation tools were not purpose-built for the human-AI conversation setting or for…papers/llama-guard-safeguard-model
verifiedhigh · 0.85
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
multimodal · arxiv2024arXiv:2407.07895in-productionOpen LMMs were built for single images, so multi-image, video and 3D inputs could not be handled by one model.papers/llava-next
verifiedhigh · 0.85
LLaVA-OneVision: Easy Visual Task Transfer
multimodal · arxiv2024arXiv:2408.03326in-productionSingle-image, multi-image and video VLMs were separately built, so capabilities did not transfer between them.papers/llava-onevision
verifiedhigh · 0.85
LLM in a flash: Efficient Large Language Model Inference with Limited Memory
serving-systems · acl2024arXiv:2312.11514research-onlyAn LLM larger than available DRAM cannot run at all, and streaming weights from flash on demand serializes flash I/O with computation, so the transfer time dominates and throughput collapses.papers/llm-in-a-flash-limited-memory
verifiedhigh · 0.90
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
long-context · icml2024arXiv:2401.01325research-onlyLLMs do not generalise to contexts longer than their training length, and the usual fixes all require fine-tuning, which is expensive and needs data the model owner may not have.papers/selfextend
verifiedhigh · 0.85
LLM-Pruner: On the Structural Pruning of Large Language Models
pruning · neurips2023arXiv:2305.11627research-onlyCompressing an LLM usually means task-specific pruning, which destroys general capability, and recovery normally needs the original (enormous) training corpus.papers/llm-pruner
verifiedhigh · 0.85
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
quantization · acl2024arXiv:2305.17888research-onlyPost-training quantization works down to 8 bits and then breaks down; quantization-aware training needs the original training data, which is unavailable for most released LLMs.papers/paper-llm-qat
verifiedhigh · 0.85
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
quantization · neurips2022arXiv:2208.07339in-productionINT8 quantization of LLMs fails because a tiny set of emergent outlier feature dimensions dominates attention output and destroys the quantization error budget.papers/llmint8
verifiedhigh · 0.85
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
embedding · colm2024arXiv:2404.05961in-upstream-engineDecoder-only LLMs were not used for text embeddings despite being the best available pretrained weights.papers/llm2vec
verifiedhigh · 0.85
Llumnix: Dynamic Scheduling for Large Language Model Serving
scheduling · osdi2024arXiv:2406.03243research-onlyLLM requests are heterogeneous and unpredictable in both resource demand and latency requirement, so a static batching or partitioning policy produces severe queueing delay, poor tail latency and SLO …papers/llumnix-dynamic-scheduling
verifiedhigh · 0.88
LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
kv-cache · arxiv2025arXiv:2510.09665in-upstream-engineKV cache grows far past GPU memory in real deployments, and moving it to CPU/DRAM/SSD with existing ad-hoc paths is slow, so cache reuse across queries and across engines is impractical.papers/paper-lmcache
verifiedhigh · 0.86
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
position-encoding · icml2024arXiv:2402.13753in-upstream-engineExtending a 4K-context model to 2M tokens by uniform positional interpolation needs non-uniform treatment of RoPE frequencies, and naive uniform scaling wastes short-context quality.papers/longrope
verifiedhigh · 0.90
LongRoPE2: Near-Lossless LLM Context Window Scaling
position-encoding · icml2025arXiv:2502.20082in-upstream-engineExisting RoPE rescaling suffers persistent out-of-distribution behaviour at long context, and the paper's diagnosis is that the higher RoPE dimensions are UNDER-TRAINED because most training tokens ne…papers/longrope2
verifiedmedium · 0.84
LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism
serving-systems · sosp2024arXiv:2404.09526research-onlyResource usage swings enormously between requests AND between the phases of a single request when contexts are long, but STATIC parallelism strategies cannot adapt - the parallelism degree is fixed at…papers/w4p-loongserve-elastic-sequence-parallelism
verifiedhigh · 0.85
LQER: Low-Rank Quantization Error Reconstruction for LLMs
quantization · icml2024arXiv:2402.02446research-onlyLow-bit quantization error in LLMs is highly structured, but recovering it needs either expensive gradient-based optimization or irregular memory access patterns.papers/paper-lqer
verifiedhigh · 0.85
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings
embedding · acl2024arXiv:2402.03216in-productionRetrieval systems needed a separate model per retrieval style, language and input length.papers/paper-bge-m3
verifiedhigh · 0.85
MagicPIG: LSH Sampling for Efficient LLM Generation
kv-cache · iclr2025arXiv:2410.16179research-onlyTop-k sparse attention degrades quality on some tasks because attention is NOT always as sparse as the top-k assumption requires - when the mass is genuinely diffuse, dropping the tail is simply wrong…papers/w4p-magicpig-lsh-attention-sampling
verifiedhigh · 0.90
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
ssm · colm2024arXiv:2312.00752in-upstream-engineSubquadratic alternatives to attention - linear attention, gated convolution, recurrent models, structured state space models - had not matched attention on language, and the paper diagnoses why: they…papers/mamba
verifiedhigh · 0.93
MambaVision: A Hybrid Mamba-Transformer Vision Backbone
ssm · cvpr2025arXiv:2407.08083in-upstream-enginePure Mamba backbones for vision underperformed ViT at high resolution, because vision needs precise local spatial detail that a global fixed-state recurrence discards.papers/mambavision
verifiedhigh · 0.85
MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models
quantization · ppopp2024arXiv:2408.11743in-upstream-engine4-bit weight quantization speeds up single-stream decoding because it cuts memory traffic, but the speedup evaporates once you batch many concurrent clients, which is the regime real serving runs in -…papers/marlin-mixed-precision-autoregressive-inference
verifiedhigh · 0.85
Masked Autoencoders Are Scalable Vision Learners
multimodal · cvpr2022arXiv:2111.06377in-upstream-engineVision self-supervised learners had not scaled comparably to NLP because masking was applied in a way that wasted encoder compute.papers/mae
verifiedhigh · 0.85
Matryoshka Representation Learning
embedding · neurips2022arXiv:2205.13147in-productionA fixed-size embedding is either oversized for a cheap downstream task or too small for an accurate one, and the task is unknown at training time.papers/matryoshka
verifiedhigh · 0.85
Measuring the Carbon Intensity of AI in Cloud Instances
other · facct2022arXiv:2206.05229in-productionData scientists have no easy or reliable access to the greenhouse-gas impact of their own AI workloads, which precludes developing actionable tactics to reduce it.papers/carbon-intensity-ai-cloud-instances
verifiedhigh · 0.85
MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
moe · sigcomm2025arXiv:2504.02263research-onlyMoE's sparsity makes inference MEMORY-intensive rather than compute-intensive at decode, which drops GPU utilization and raises operating cost - and it means the attention and FFN halves of a layer wa…papers/w4p-megascale-infer-disaggregated-expert-parallelism
verifiedmedium · 0.80
MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
kv-cache · arxiv2024arXiv:2406.17565research-onlyServing has moved from stateless to STATEFUL - context caching and disaggregated inference both extend the lifetime and reach of the KV cache - which creates an architectural problem no single existin…papers/w4p-memserve-elastic-memory-pool
verifiedmedium · 0.75
Microscaling Data Formats for Deep Learning
quantization · arxiv2023arXiv:2310.10537in-productionNarrow formats force a choice between hardware efficiency and accuracy, and per-tensor scales are too coarse for weights and gradients that vary wildly element to element.papers/mx-formats
verifiedhigh · 0.85
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
long-context · neurips2024arXiv:2407.02490research-onlyPREFILL for long-context LLMs is quadratic in context length, and a single sparse-attention pattern cannot serve all long-context tasks because attention structure differs by head type and by task.papers/minference
verifiedhigh · 0.88
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
kv-cache · neurips2024arXiv:2405.14366research-onlyKV cache compression is almost always done along the TOKEN dimension - drop or quantize cache entries - which throws away attention resolution. The depth dimension (across layers) is a second, largely…papers/w4p-minicache-depth-kv-compression
verifiedhigh · 0.90
MiniLLM: On-Policy Distillation of Large Language Models
distillation · iclr2024arXiv:2306.08543research-onlyDistilling a WHITE-BOX LLM into a smaller LLM is under-explored - most KD work either distills classifiers or only imitates a black-box API - and forward KL lets the student waste capacity imitating l…papers/minillm
verifiedhigh · 0.88
MiniMax-01: Scaling Foundation Models with Lightning Attention
attention · arxiv2025arXiv:2501.08313in-productionLinear attention could not be scaled to hundreds of billions of parameters in practice, because long-context training of linear attention requires computation-communication overlap techniques nobody h…papers/minimax-01
verifiedhigh · 0.85
MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
attention · arxiv2025arXiv:2506.13585in-productionLong-context reasoning models chain many thousands of intermediate reasoning tokens, and paying full KV-cache attention cost for every one of those tokens does not scale, so test-time compute scaling …papers/paper-minimax-m1
verifiedhigh · 0.85
Mistral 7B
attention · arxiv2023arXiv:2310.06825in-productionA 7B model competitive with far larger models is wanted, but inference cost at that size is set by attention cost over long contexts and by KV-cache footprint.papers/mistral-7b-gqa-swa
verifiedhigh · 0.90
Mixtral of Experts
moe · arxiv2024arXiv:2401.04088in-productionSparse MoE quality was available only in very large or gated research models; Mixtral targeted dense-7B serving cost while beating 70B-class dense models.papers/mixtral
verifiedhigh · 0.95
Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness
quantization · arxiv2023arXiv:2310.02410research-onlyMoE deployment is memory-bound because ALL expert weights must be resident, even though only a few experts run per token.papers/moqe
verifiedmedium · 0.82
Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
inference-time-compute · arxiv2024arXiv:2404.02258research-onlyTransformers spread FLOPs UNIFORMLY across all input positions and all layers, so easy tokens and hard tokens cost the same - compute is not allocated where it is needed.papers/mixture-of-depths
verifiedhigh · 0.85
Mixture-of-Experts with Expert Choice Routing
routing · neurips2022arXiv:2202.09368research-onlyTop-k gating gives every token the same number of experts regardless of how important the token is, and load imbalance leaves some experts under-trained and under-specialized.papers/expert-choice-routing
verifiedhigh · 0.85
MoBA: Mixture of Block Attention for Long-Context LLMs
attention · neurips2025arXiv:2502.13189in-productionSparse attention for long context is usually a fixed pattern chosen ahead of time, which either misses genuinely distant information or wastes budget on tokens that do not matter.papers/moba
verifiedhigh · 0.85
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
other · icml2024arXiv:2402.14905in-productionGrowing cloud cost and latency concerns push LLMs on-device, which means sub-billion parameters, a scale where prevailing belief says data and parameter quantity dominate model quality.papers/mobilellm-on-device
verifiedhigh · 0.85
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
moe · arxiv2026arXiv:2610.01950research-onlyRouted experts live in host memory because the device cannot hold the pool, and the two phases want opposite fetches. Prefill activates a broad union of experts across many tokens, so a per-expert cac…papers/moe-core-coordinated-expert-offload
verifiedmedium · 0.80
MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache
moe · arxiv2025arXiv:2401.14361research-onlyOn a single-user machine the whole MoE does not fit in VRAM, so routed experts live in host RAM and every decode step pays a PCIe fetch. Next-layer prefetch (DeepSpeed, Mixtral-Offloading) and LRU (vL…papers/moe-infinity-sparsity-aware-expert-cache
verifiedhigh · 0.85
MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
moe · asplos2025arXiv:2411.11217research-onlyMoE's low per-token FLOPs make it attractive on paper, but its size puts it out of reach on anything below a high-end GPU, so in practice the whole point of the architecture is unavailable to the peop…papers/w4p-moe-lightning-hierarchical-roofline
verifiedhigh · 0.88
MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance
quantization · icml2025arXiv:2505.03804research-onlyPTQ degrades MoE models far worse than dense models, and the usual dense calibration set starves rarely-activated experts.papers/moequant
verifiedhigh · 0.85
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
multimodal · cvpr2025arXiv:2409.17146in-productionOpen-weight VLMs relied on synthetic data distilled from proprietary VLMs, so the community could not build or audit them from scratch.papers/molmo
verifiedhigh · 0.85
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
serving-systems · acm-tos2024arXiv:2407.00079in-productionServing a long-context chat workload needs to keep prefill and decode from interfering while also holding a KV cache far larger than GPU memory.papers/mooncake-kimi
verifiedhigh · 0.88
MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
serving-systems · icml2024arXiv:2404.02015research-onlyWhen LLM popularity is highly skewed, dedicating whole GPUs per model wastes most of each GPU — the hot model needs only part of a GPU while the tail model sits on an under-used device.papers/muxserve
verifiedhigh · 0.85
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
attention · acl2025arXiv:2502.11089in-upstream-engineExisting sparse-attention methods either lack the arithmetic intensity to run fast on real hardware (they degrade memory-bound kernels into launch-bound ones) or cannot be trained end-to-end for pretr…papers/nsa-native-sparse
verifiedmedium · 0.80
Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
ssm · arxiv2025arXiv:2504.03624in-upstream-engineInference-time scaling makes models larger and slower to serve, so the cost of reasoning is paid on every single token of every request - constant per-token cost is the thing worth buying.papers/w4p-nemotron-h-hybrid-mamba-transformer
verifiedmedium · 0.80
Nomic Embed: Training a Reproducible Long Context Text Embedder
embedding · tmlr2024arXiv:2402.01613in-productionOpen embedding models could not be reproduced because training data and code were withheld, and long-context embedders were weak.papers/nomic-embed
verifiedhigh · 0.85
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
quantization · iclr2024arXiv:2308.13137research-onlyPTQ methods hand-craft their quantization parameters (clipping thresholds, equivalent transformations), which is why they degrade at INT3/INT4-W4A4.papers/paper-omniquant
verifiedhigh · 0.85
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
distillation · iclr2024arXiv:2306.13649research-onlySequence-level KD suffers a train/inference mismatch: the student is trained on a FIXED set of output sequences but generates its own at inference, and the errors compound.papers/gkd
verifiedhigh · 0.88
Orca: A Distributed Serving System for Transformer-Based Generative Models
serving-systems · osdi2022no arXivin-productionTransformer autoregressive generation yields one token per request per forward pass, so a serving engine that schedules at request granularity cannot fill the GPU with work from a sequence that finish…papers/orca-osdi22-iteration-scheduling
verifiedhigh · 0.95
Orca: Progressive Learning from Complex Explanation Traces of GPT-4
distillation · arxiv2023arXiv:2306.02707in-productionModels distilled from large foundation models tend to learn the ANSWER STYLE of the teacher rather than its REASONING PROCESS, and the small model's capability is then systematically overestimated bec…papers/orca
verifiedhigh · 0.85
Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decoding
speculative-decoding · emnlp2024arXiv:2402.13720research-onlyUnder the draft-then-verify framing, DRAFTING efficiency - not verification - is the bottleneck capping total speculative-decoding speedup.papers/ouroboros-phrase-by-phrase-drafting
verifiedmedium · 0.80
PaliGemma: A versatile 3B VLM for transfer
multimodal · arxiv2024arXiv:2407.07726in-productionTask-specific VLM fine-tuning was fragmented across many bespoke models and datasets.papers/paligemma
verifiedhigh · 0.85
Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
serving-systems · osdi2024arXiv:2405.19888research-onlyLLM-based applications are DAGs of many dependent LLM calls, but public serving APIs expose only a request-level interface, so the scheduler cannot see application structure and optimises each request…papers/parrot
verifiedmedium · 0.82
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
other · arxiv2024arXiv:2404.14219in-productionCapability at sub-10B scale is dominated by parameter count and data volume, which makes such models too expensive to run on-device even though on-device deployment is where the latency and cost press…papers/phi-3-small-capable
verifiedmedium · 0.80
Phi-4 Technical Report
distillation · arxiv2024arXiv:2412.08905in-productionMost capable LLMs are trained on organic data; previous Phi models distilled GPT-4 and plateaued at the teacher.papers/phi-4
verifiedhigh · 0.85
Pixtral 12B
multimodal · arxiv2024arXiv:2410.07073in-productionFixed-resolution vision encoders force a resolution/aspect-ratio choice at design time and cannot adapt token budget to the input.papers/pixtral
verifiedhigh · 0.85
PLAID: An Efficient Engine for Late Interaction Retrieval
retrieval · cikm2022arXiv:2205.09707in-upstream-engineLate interaction's query-time MaxSim scan over full-resolution document vectors was too slow for interactive search at scale.papers/plaid
verifiedhigh · 0.85
Post Training Quantization of Large Language Models with Microscaling Formats
quantization · enlsp2024arXiv:2405.07135research-onlySmoothQuant, AWQ and GPTQ were each designed for fixed-point targets; none of them can emit the microscaling formats that new hardware natively supports.papers/mx-ftptq
verifiedmedium · 0.82
Power Aware Dynamic Reallocation For Inference (RAPID)
scheduling · arxiv2026arXiv:2601.12241research-onlyDisaggregated prefill/decode inference improves utilisation and throughput under fixed hardware capacity, but as model and cluster scales grow, power rather than compute becomes the dominant limiter o…papers/rapid-power-aware-disaggregated-inference
verifiedmedium · 0.78
PowerInfer: Fast Large Language Model Inference with a Consumer-grade GPU
serving-systems · sosp2024arXiv:2312.12456research-onlyA consumer PC has a GPU with far less memory and bandwidth than a datacenter GPU, so an LLM that fits nowhere must either spill to CPU (PCIe-bound, very slow) or be shrunk to a lower-quality model; ne…papers/powerinfer-consumer-gpu-serving
verifiedhigh · 0.88
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert LLM Inference
moe · isca2024arXiv:2308.12066research-onlyStandard top-k MoE routing runs the router AFTER all experts finish, so expert communication and memory traffic are serialised behind the slowest expert and small-batch MoE inference is dominated by t…papers/pre-gated-moe
verifiedmedium · 0.80
Preble: Efficient Distributed Prompt Scheduling for LLM Serving
scheduling · iclr2025arXiv:2407.00023research-onlyModern prompts are largely REPETITIVE - domain instructions, tool-usage illustrations, long reference documents recur across requests - and engines can reuse that KV state, but every existing reuse me…papers/w4p-preble-distributed-prompt-scheduling
verifiedhigh · 0.90
Pretraining Large Language Models with NVFP4
quantization · arxiv2025arXiv:2509.25149research-onlyFP8 training is now standard, but 4-bit floating-point TRAINING at frontier scale faces stability, convergence and implementation problems.papers/nvfp4-pretraining
verifiedmedium · 0.80
ProMoE: Fast MoE-based LLM Serving using Proactive Caching
moe · arxiv2025arXiv:2410.22134research-onlyWith routed experts in host RAM, a cache miss is only discovered when the expert's GEMM is about to run, so the PCIe fetch sits on the critical path. On decoder-only MoE the expert access distribution…papers/promoe-proactive-expert-prefetch
verifiedhigh · 0.85
Punica: Multi-Tenant LoRA Serving
serving-systems · mlsys2024arXiv:2310.18547in-upstream-engineServing many LoRA adapters over one shared base model wastes GPU memory (a copy of the base per adapter) and, worse, cannot batch: each adapter's low-rank matrices have different shapes, so GPU operat…papers/punica-multi-tenant-lora-serving
verifiedhigh · 0.90
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
kv-cache · colm2025arXiv:2406.02069research-onlyEvery KV-cache compression method allocates the SAME budget to every layer, which contradicts how transformers actually route information through depth.papers/pyramidkv-paper
verifiedhigh · 0.87
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
quantization · arxiv2023arXiv:2310.16795research-onlyA trillion-parameter MoE does not fit in GPU memory at all: SwitchTransformer-c2048 is 1.6 trillion parameters and 3.2 TB in bfloat16, and the bytes that matter at decode are the routed expert, which …papers/qmoe-sub-1-bit-compression
verifiedmedium · 0.82
QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
quantization · mlsys2025arXiv:2405.04532research-onlyState-of-the-art INT4 PTQ methods only speed up low-batch edge inference; in large-batch cloud serving the dequantization overhead eats the entire gain (measured at 20-90%).papers/qserve
verifiedhigh · 0.85
QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
quantization · arxiv2024arXiv:2406.08155research-onlyMoE quantization is done at one uniform precision for the whole model, which is wrong because MoE's routing gives its experts structurally different activation statistics.papers/quantmoe-bench
verifiedmedium · 0.80
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
quantization · neurips2024arXiv:2404.00456in-upstream-engineActivation outliers in LLMs make W4A4 (and even KV4) quantization collapse, and every fix so far keeps a handful of channels in higher precision.papers/paper-quarot
verifiedhigh · 0.85
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
kv-cache · icml2024arXiv:2406.10774research-onlyLong-context decode is slow mainly because attention must read the whole KV cache from HBM, and prior KV-selection work assumed token criticality is query-INDEPENDENT, when in fact which tokens matter…papers/quest
verifiedhigh · 0.89
QuIP: 2-Bit Quantization of Large Language Models With Guarantees
quantization · neurips2023arXiv:2307.13304research-onlyBelow 3 bits, weight quantization error explodes because weights are spread-out and the Hessian directions that matter for rounding are badly aligned with the coordinate axes.papers/quip
verifiedhigh · 0.85
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
quantization · icml2024arXiv:2402.04396research-onlyWeight-only PTQ at <=4 bits was not yet Pareto-optimal, and generic random orthogonal incoherence transforms are slow and lack tight guarantees.papers/paper-quip-sharp
verifiedhigh · 0.85
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, and Text Recognition
multimodal · arxiv2023arXiv:2308.12966in-productionGeneralist VLMs of comparable scale lacked grounded text reading and localisation ability.papers/qwen-vl
verifiedhigh · 0.85
Qwen2-Audio Technical Report
speech · arxiv2024arXiv:2407.10759in-productionAudio-language models needed complex hierarchical task tags, which made pre-training data curation brittle and instruction-following weak.papers/qwen2-audio
verifiedhigh · 0.85
Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
position-encoding · arxiv2024arXiv:2409.12191in-upstream-engineA single 1-D RoPE position index cannot express the 2-D/3-D spatial structure of an image or video, and naive per-modality index assignment degrades cross-modal alignment - so a VLM cannot exploit hig…papers/qwen2-vl-mrope
verifiedhigh · 0.87
ReAct: Synergizing Reasoning and Acting in Language Models
agentic · iclr2023arXiv:2210.03629in-productionAn LLM's abilities for REASONING (chain-of-thought) and for ACTING (action-plan generation) had been studied as separate topics, so reasoning traces could not ground themselves in external information…papers/react-reasoning-and-acting
verifiedhigh · 0.90
RecurrentGemma: Moving Past Transformers for Efficient Open Language Models
ssm · arxiv2024arXiv:2404.07839research-onlyAttention-based decoding needs a KV cache that grows with context, which bounds long-sequence inference memory even for small models.papers/recurrentgemma-griffin
verifiedmedium · 0.80
Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
speculative-decoding · arxiv2024arXiv:2402.14160research-onlyExisting tree-based draft methods generate tokens INDEPENDENTLY at each tree level and therefore waste the tree's diversity; and their reported superiority implicitly grants tree methods more compute …papers/recursive-speculative-decoding-rsd
verifiedmedium · 0.80
REST: Retrieval-Based Speculative Decoding
speculative-decoding · naacl2024arXiv:2311.08252research-onlyDraft models are needed just to make speculative decoding work; they require training, storage, and maintenance, and are hard to obtain for arbitrary target models.papers/rest-retrieval-speculative-decoding
verifiedhigh · 0.85
Retentive Network: A Successor to Transformer for Large Language Models
ssm · arxiv2023arXiv:2307.08621research-onlyTransformers need attention to model token interactions; recurrent models cannot parallelise training. The choice of sequence-mixing primitive forces that trade-off, and no prior alternative offered t…papers/retnet
verifiedhigh · 0.85
Rethinking Attention with Performers
attention · iclr2021arXiv:2009.14794abandonedAttention is O(n^2) in time and memory. To go linear one must approximate the softmax kernel, but existing approximations either assume sparsity or low-rankness - priors that rarely hold for attention…papers/performer
verifiedhigh · 0.87
Ring Attention with Blockwise Transformers for Near-Infinite Context
attention · iclr2024arXiv:2310.01889research-onlySingle-device attention cannot exceed a fixed sequence length on available HBM, and simply spilling sequences to many devices either breaks training parallelism or pays for communication.papers/ring-attention
verifiedhigh · 0.87
RouteLLM: Learning to Route LLMs with Preference Data
routing · iclr2025arXiv:2406.18665in-productionCalling one strong model for every query is wasteful when most queries are easy, so routing between a strong and a weak model trades a little accuracy for a large cost cut.papers/routellm
verifiedhigh · 0.85
RWKV: Reinventing RNNs for the Transformer Era
ssm · emnlp2023arXiv:2305.13048in-upstream-engineTransformers parallelise during training but are quadratic and memory-hungry at inference; RNNs are linear and constant-memory at inference but do not parallelise during training. Architecture familie…papers/rwkv
verifiedhigh · 0.87
S-LoRA: Serving Thousands of Concurrent LoRA Adapters
serving-systems · arxiv2023arXiv:2311.03285research-onlyThe pretrain-then-finetune paradigm produces a large COLLECTION of LoRA adapters over one base model, and serving them naively multiplies memory and fragmentation - yet almost nobody runs thousands of…papers/w4p-s-lora-multi-tenant-adapter-serving
verifiedmedium · 0.75
s1: Simple test-time scaling
inference-time-compute · emnlp2025arXiv:2501.19393research-onlyOpenAI's o1 showed test-time scaling works but did not publish its methodology, and replication attempts were producing complicated systems.papers/s1-budget-forcing
verifiedhigh · 0.85
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration
quantization · iclr2025arXiv:2410.02367research-onlyAttention itself can be quantised to cut the memory traffic of the QK^T and PV matmuls, but naive INT8 quantisation of attention loses accuracy because the softmax output contains large channel-wise o…papers/sageattention
verifiedhigh · 0.85
SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization
quantization · icml2025arXiv:2411.10958research-onlySageAttention's INT8 path works, but pushing attention quantisation to INT4 needs a per-thread (rather than per-tensor or per-block) scale to represent the input distribution, and the FP8 PV matmul ac…papers/sageattention2
verifiedhigh · 0.85
SAM Decoding: Speculative Decoding via Suffix Automaton
speculative-decoding · acl2025arXiv:2411.10666in-upstream-engineRetrieval-based (model-free) speculative drafting matches n-grams to propose drafts, but n-gram matching finds only short matches, so acceptance rates stay low; and it is domain-bound - an n-gram tabl…papers/w4p-sam-decoding-suffix-automaton
verifiedhigh · 0.90
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
kv-cache · neurips2023arXiv:2305.17118research-onlyThe KV cache can exceed the model weights themselves and caps inference batch size; existing compression needed fine-tuning or degraded quality.papers/scissorhands
verifiedhigh · 0.86
Self-Consistency Improves Chain of Thought Reasoning in Language Models
inference-time-compute · iclr2023arXiv:2203.11171in-productionChain-of-thought prompting still uses NAIVE GREEDY decoding, which commits to the first sampled reasoning path even when several paths would have reached the right answer.papers/self-consistency-marginalise-paths
verifiedhigh · 0.90
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
embedding · emnlp2019arXiv:1908.10084in-productionBERT must run both sentences through the network jointly, making similarity search over a corpus quadratically expensive.papers/sbert
verifiedhigh · 0.85
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
moe · arxiv2026arXiv:2609.12978research-onlyWith low VRAM and a lot of host RAM, an MoE decode step only needs a few experts, but those experts are known only after the router runs, so the host-to-device copy lands on the critical path. A predi…papers/seqmoe-graph-compatible-moe-offload
verifiedmedium · 0.75
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
speculative-decoding · arxiv2024arXiv:2402.12374research-onlyExisting speculative-decoding methods cannot scale to large speculation budgets and do not adapt to differing hyperparameters or to different hardware.papers/sequoia-hardware-aware-specdec
verifiedhigh · 0.85
Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
serving-systems · osdi2020arXiv:2006.02464research-onlyReactive serving tricks (batching, warm pools) tame average latency but cannot bound the tail, because per-request execution time is not actually unpredictable — the internal state causing variance is…papers/clockwork
verifiedmedium · 0.82
SGLang: Efficient Execution of Structured Language Model Programs
serving-systems · neurips2024arXiv:2312.07104in-productionComplex LLM applications chain many generation calls with control flow, shared state and structured I/O, but serving runtimes execute each call independently and waste the reuse across them.papers/paper-sglang
verifiedhigh · 0.90
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
kv-cache · icml2025arXiv:2410.21465research-onlyDynamic sparse attention either does not cut GPU memory enough, or gets large decode latency by offloading the KV cache to CPU memory.papers/paper-shadowkv
verifiedhigh · 0.85
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
pruning · iclr2024arXiv:2310.06694research-onlyTraining a competitive 1-3B model from scratch on trillions of tokens is expensive; is there a cheaper route to a small capable model than pretraining?papers/sheared-llama
verifiedhigh · 0.85
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect
pruning · arxiv2024arXiv:2403.03853research-onlyDeep LMs contain many near-duplicate layers and some layers that barely affect function, but pruning them requires complex machinery.papers/shortgpt-layer-pruning
verifiedmedium · 0.80
Sigmoid Loss for Language Image Pre-Training (SigLIP)
multimodal · iccv2023arXiv:2303.15343in-productionSoftmax/InfoNCE contrastive loss requires a global view of all in-batch similarities, which caps the usable batch size and makes large-scale pre-training expensive.papers/siglip
verifiedhigh · 0.85
SimCSE: Simple Contrastive Learning of Sentence Embeddings
embedding · emnlp2021arXiv:2104.08821in-productionSupervised sentence-embedding methods needed a separately trained model per dataset, so embeddings did not transfer.papers/simcse
verifiedhigh · 0.85
Simple linear attention language models balance the recall-throughput tradeoff
attention · icml2024arXiv:2402.18668research-onlyEfficient alternatives to attention (H3, Mamba, RWKV) hold a FIXED-SIZE recurrent state, and that small state is exactly why they fail at recall - grounding a generation in tokens seen earlier in cont…papers/simple-linear-attention
verifiedhigh · 0.86
SliceGPT: Compress Large Language Models by Deleting Rows and Columns
pruning · iclr2024arXiv:2401.15024research-onlyPost-hoc sparsification needs extra sparse data structures and gives constrained speedup on current hardware, because irregular sparsity is not what accelerators accelerate.papers/slicegpt
verifiedhigh · 0.85
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
quantization · icml2023arXiv:2211.10438in-upstream-engineW8A8 activation quantization of LLMs is blocked by a small number of activation outlier channels that blow up the per-tensor dynamic range.papers/paper-smoothquant
verifiedhigh · 0.85
SnapKV: LLM Knows What You are Looking for Before Generation
kv-cache · neurips2024arXiv:2404.14469research-onlyThe KV cache grows with input length, inflating both memory and per-token decode cost, and existing compression methods need fine-tuning or hurt quality on real prompts.papers/snapkv-paper
verifiedhigh · 0.88
SoundStream: An End-to-End Neural Audio Codec
speech · taslp2021arXiv:2107.03312in-productionSpeech codecs are tuned for speech and waste bits on music and general audio; a fixed-bitrate model cannot trade quality against rate per deployment.papers/soundstream
verifiedhigh · 0.85
Sparse Attention with Linear Units
attention · emnlp2021arXiv:2104.07012research-onlySparse attention is usually obtained by sparsifying SOFTMAX, which needs approximations of the softmax function and ties sparsity to the softmax's behaviour.papers/sau
verifiedmedium · 0.84
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
pruning · icml2023arXiv:2301.00774research-onlyOne-shot pruning of 100B+ parameter LLMs to 50% sparsity was thought to require retraining or intractable second-order computation.papers/sparsegpt
verifiedhigh · 0.85
SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
multimodal · icml2025arXiv:2410.04417research-onlyExisting pruning methods needed a learned pruner trained on extra data, adding parameters and fine-tuning cost.papers/sparsevlm
verifiedhigh · 0.85
SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices
speculative-decoding · neurips2024arXiv:2406.02532research-onlyConsumer GPUs cannot fit 50B+ models, so their parameters are offloaded to RAM or SSD, and with offloaded weights the engine can draft hundreds or thousands of candidate tokens for the same memory cos…papers/w4p-specexec-consumer-speculative
verifiedhigh · 0.90
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
speculative-decoding · arxiv2026arXiv:2610.15417research-onlyWith routed experts in host RAM, the host-to-device copy is on the decode critical path, and speculative verification makes it worse: every draft token, including the ones the target rejects, routes i…papers/specfit-offloaded-moe-speculative
verifiedmedium · 0.80
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
speculative-decoding · asplos2024arXiv:2305.09781in-upstream-engineSmall speculative models exist and predict well, but single-sequence drafting wastes their capacity, and sequential verification of drafts leaves latency on the table.papers/specinfer-tree-speculative-serving
verifiedhigh · 0.90
SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
moe · arxiv2026arXiv:2607.24787research-onlyWith routed experts off the accelerator, the expert set for a layer is known only after that layer's router runs, so the copy sits on the critical path: the paper measures exposed expert loading at 67…papers/specprefetch-window-aware-expert-prefetch
verifiedmedium · 0.78
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
speech · emnlp2023arXiv:2305.11000research-onlyCascaded speech systems (ASR -> text -> LLM -> TTS) cannot transfer knowledge between modalities, so speech understanding degrades relative to text.papers/speechgpt
verifiedmedium · 0.75
SpinQuant: LLM Quantization with Learned Rotations
quantization · iclr2025arXiv:2405.16406research-onlyRandom rotations remove activation outliers (QuaRot) but the quality you get depends heavily on WHICH random rotation you draw - some draws are far better than others.papers/paper-spinquant
verifiedhigh · 0.85
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
retrieval · sigir2021arXiv:2107.05720in-productionDense retrievers lose the exact-match and interpretability properties of bag-of-words, while learned sparse retrievers had not matched their effectiveness.papers/splade
verifiedhigh · 0.85
Splitwise: Efficient generative LLM inference using phase splitting
serving-systems · isca2024arXiv:2311.18677research-onlyA single LLM serving request has two phases with opposite bottlenecks — prompt computation is compute-bound and memory-bound prompt processing is bandwidth-bound — so running both on the same device w…papers/splitwise
verifiedhigh · 0.85
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
quantization · iclr2024arXiv:2306.03078research-only3-4 bit quantization causes moderate-to-high accuracy loss, worst on the small (1-10B) models that edge deployment actually targets.papers/spqr
verifiedhigh · 0.85
SqueezeLLM: Dense-and-Sparse Quantization
quantization · icml2024arXiv:2306.07629research-onlySingle-batch generative LLM inference is memory-bandwidth-bound, so weights must shrink - but uniform low-bit quantization loses perplexity badly at 3 bits.papers/paper-squeezellm
verifiedhigh · 0.85
Stateful Large Language Model Serving with Pensieve
kv-cache · eurosys2024arXiv:2312.05516research-onlyLLM serving systems are stateless across requests, so a growing multi-turn conversation log is re-processed on every turn.papers/paper-pensieve
verifiedhigh · 0.85
Striped Attention: Faster Ring Attention for Causal Transformers
attention · arxiv2023arXiv:2311.09431research-onlyRing Attention's contiguous per-device sequence split creates LOAD IMBALANCE under causal masking: early devices own fewer live tokens than late devices, so the slowest device determines throughput an…papers/striped-attention
verifiedmedium · 0.84
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
moe · jmlr2022arXiv:2101.03961in-upstream-engineMoE scales parameters at constant FLOPs, but adoption was blocked by routing complexity, expert communication cost, and training instability.papers/switch-transformers
verifiedhigh · 0.90
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
scheduling · osdi2024arXiv:2403.02310in-upstream-engineInterleaving prefill iterations with batched decode iterations raises the tail of time-per-output-token because a long prefill blocks every decode in the batch, capping achievable throughput at any la…papers/sarathi-serve
verifiedhigh · 0.88
Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter
quantization · arxiv2026arXiv:2609.35465research-onlyLeech-lattice quantization gives good quality at 2 bits/weight, but the codebooks have more than 10^14 points - far too many for a lookup table, so earlier kernels read 4.8 bits/weight from memory to …papers/tetra
verifiedmedium · 0.80
Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5)
embedding · arxiv2022arXiv:2212.03533in-productionEmbedding models supervised on a single task or a small labelled set failed to generalise to retrieval.papers/e5
verifiedhigh · 0.85
The Embedder's Dilemma: LLMs Are Better, but at What Cost?
embedding · colm2026arXiv:2608.12875in-productionTeams are being pushed to swap embedding pipelines for LLMs without a controlled cost comparison.papers/embedder-dilemma
verifiedhigh · 0.85
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
quantization · arxiv2024arXiv:2402.17764research-onlyPure 1-bit weights quantize poorly because the +1/-1 grid cannot express that some weights should be near zero.papers/paper-bitnet-b158
verifiedhigh · 0.85
The Joule Point: an Energy-Optimal Operating Point for AI Inference
serving-systems · arxiv2026arXiv:2609.27926research-onlyData centres run GPU inference at full power by default, which maximises throughput but wastes energy per inference - and the inefficiency is a scheduling choice rather than a hardware limit.papers/joule-point-energy-optimal-power-cap
verifiedhigh · 0.85
The Llama 3 Herd of Models
distillation · arxiv2024arXiv:2407.21783in-productionPost-training quality has been getting less attention than pre-training scale, and the distillation/rejection-sampling recipe is the least documented part of frontier pipelines.papers/llama-3-herd
verifiedhigh · 0.85
ThinQuant: Scalable Rotation Learning for Weight and Activation Quantization of LLMs
quantization · arxiv2026arXiv:2609.36120research-onlyGradient-based rotation learning (SpinQuant) is too expensive at frontier scale, and even gradient-free rotation learning (DartQuant) needs too much calibration data and wall-clock.papers/thinquant
verifiedmedium · 0.75
Titans: Learning to Memorize at Test Time
long-context · neurips2025arXiv:2501.00663research-onlyA model's memory is either its finite context window (short-term) or its weights (fixed at training time); there is no mechanism that ACQUIRES and retains new information at inference the way a cache …papers/titans
verifiedhigh · 0.85
Towards General Text Embeddings with Multi-stage Contrastive Learning (GTE)
embedding · arxiv2023arXiv:2308.03281in-productionEvery embedding task was served by its own specialised model, so a single general-purpose encoder was unavailable.papers/gte
verifiedhigh · 0.85
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
attention · icml2020arXiv:2006.16236research-onlySelf-attention is O(N^2) in sequence length, which is prohibitive for autoregressive prediction over very long sequences.papers/linear-transformers
verifiedhigh · 0.88
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
ssm · icml2024arXiv:2405.21060in-upstream-engineMamba and attention looked like different families of sequence models, and there was no principled account of when or why one should be preferred - which made it hard to tell whether the SSM's weaknes…papers/mamba-2
verifiedhigh · 0.92
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
attention · arxiv2023arXiv:2307.14995abandonedLinear-attention LLMs were claimed to beat softmax attention on accuracy, but in practice suffered attention dilution, unstable training, and no practical inference path.papers/transnormerllm
verifiedmedium · 0.80
TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text
retrieval · emnlp2025arXiv:2410.07590research-onlyConventional RAG concatenates many retrieved chunks and prefills all of them online, and since prefill is the most compute-dense step in the whole serving stack, this dominates time-to-first-token.papers/w4p-turborag-precomputed-chunk-kv
verifiedhigh · 0.88
Tutel / Flex: Adaptive Mixture-of-Experts at Scale
moe · mlsys2023arXiv:2206.03382research-onlyMoE execution was statically parallelised and pipelined even though the router makes expert load dynamic at run time, so every step paid for an average case that was never the actual case.papers/tutel-flex
verifiedmedium · 0.80
Unsupervised Dense Information Retrieval with Contrastive Learning (Contriever)
retrieval · tmlr2021arXiv:2112.09118in-upstream-engineDense retrievers needed labelled query-passage pairs and did not transfer to domains with no training data.papers/contriever
verifiedhigh · 0.85
Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Models
quantization · arxiv2026arXiv:2606.05688research-onlyMoE quantization error is amplified discontinuously: a tiny perturbation can flip the top-k expert choice, changing the entire computation path for that token.papers/vsraq
verifiedmedium · 0.75
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
serving-systems · asplos2025arXiv:2405.04437research-onlyPagedAttention-style KV cache management demands that every attention kernel be rewritten to read block tables and masked pages, so it does not compose with existing hand-written kernels and creates o…papers/vattention
verifiedhigh · 0.88
VisionZip: Longer is Better but Not Necessary in Vision Language Models
multimodal · cvpr2025arXiv:2412.04467in-upstream-engineVLM quality kept improving with longer visual token sequences, multiplying prefill cost - but visual tokens from CLIP/SigLIP encoders carry large redundancy.papers/visionzip
verifiedhigh · 0.85
Visual Instruction Tuning (LLaVA)
multimodal · neurips2023arXiv:2304.08485in-productionLLMs had strong zero-shot behaviour from instruction tuning but vision-language models had not been shown to benefit from the same recipe.papers/llava
verifiedhigh · 0.85
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
speech · neurips2020arXiv:2006.11477in-productionRepresentation learning from raw audio needed a contrastive objective that scales without labelled transcripts.papers/wav2vec2
verifiedhigh · 0.85
wav2vec: Unsupervised Pre-training for Speech Recognition
speech · interspeech2019arXiv:1904.05862in-upstream-engineCharacter-based acoustic models need large amounts of transcribed speech, so they do not generalise to rare domains or languages.papers/wav2vec
verifiedhigh · 0.85
Where Does the Energy Go? Profiling LLM Agent Inference on Blackwell GPUs
serving-systems · arxiv2026arXiv:2609.29707research-onlyHow LLM energy is distributed across hardware components and workload phases in agentic inference is poorly understood, because measuring it requires simultaneous visibility into component-level power…papers/where-does-the-energy-go-blackwell-profiling
verifiedhigh · 0.85
Whisper: Robust Speech Recognition via Large-Scale Weak Supervision
speech · icml2023arXiv:2212.04356in-productionSupervised ASR requires per-domain labelled audio, so one model cannot cover many domains and noisy conditions without retraining.papers/whisper
verifiedhigh · 0.85
WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
moe · arxiv2026arXiv:2606.21868research-onlyWith low VRAM and a lot of host RAM, a static layer offload still copies every expert in the layer on every forward even though routing uses a few of them, and a predictor that prefetches the next exp…papers/wisp-working-set-moe-paging
verifiedmedium · 0.80
WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions
distillation · iclr2024arXiv:2304.12244research-onlyInstruction-following data is expensive to hand-write, and humans struggle to produce the high-complexity instructions that actually teach a model.papers/wizardlm
verifiedhigh · 0.85
XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs
structured-output · cais2026arXiv:2601.04426in-upstream-engineAgentic workloads need DYNAMIC structured generation - tool calls and response protocols that change both ACROSS requests and WITHIN a request - which breaks the assumption every existing structured-g…papers/w4p-xgrammar2-dynamic-agentic-structured-generation
verifiedhigh · 0.90
XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
structured-output · mlsys2025arXiv:2411.15100in-upstream-engineConstrained decoding against a context-free grammar requires walking several stack states over ALL VOCABULARY TOKELS at runtime, which is non-negligible overhead on the decode critical path - so struc…papers/w4p-xgrammar-structured-generation-engine
verifiedhigh · 0.90
XWind: A Cross-site Router for Large Language Model Inference Serving at Renewable Energy Farms
serving-systems · arxiv2026arXiv:2605.23348research-onlyAI power demand is growing while grids struggle to keep up; grid expansion carries high capital cost and transmission losses, yet abundant renewable energy exists at the source, just not matched to de…papers/xwind-renewable-site-inference-router
verifiedmedium · 0.75
YaRN: Efficient Context Window Extension of Large Language Models
position-encoding · iclr2024arXiv:2309.00071in-upstream-engineExtending a RoPE model's context window requires either a huge fine-tuning cost or a trick that degrades short-context quality, and existing methods need far too many training tokens and steps to be p…papers/yarn
verifiedhigh · 0.93
Zamba: A Compact 7B SSM Hybrid Model
ssm · arxiv2024arXiv:2405.16712research-onlyHybrid SSM-attention models of the Jamba style duplicate a separate attention module per layer group, so parameter cost and memory scale with the number of hybrids.papers/zamba
verifiedmedium · 0.80
ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
quantization · arxiv2023arXiv:2303.08302research-onlyNo systematic study existed of how PTQ choices (scheme, model family, bit width, weight vs activation) interact across the 125M-176B scale range.papers/paper-zeroquant-v2
verifiedhigh · 0.85
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
quantization · neurips2022arXiv:2206.01861research-onlyQuantization overhead (quantize/dequantize kernels) and calibration data cost made quantization of large models uneconomic in practice.papers/zeroquant
verifiedhigh · 0.85
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification
kv-cache · neurips2024arXiv:2405.14256research-onlyAdaptive KV compression that picks salient tokens degrades sharply at high compression ratios because salient-token identification is inaccurate.papers/paper-zipcache
verifiedhigh · 0.85
Zipformer: A faster and better encoder for automatic speech recognition
speech · iclr2024arXiv:2310.11230in-upstream-engineConformer encoders were accurate but too slow and too memory-hungry for on-device and high-concurrency streaming ASR.papers/zipformer
verifiedhigh · 0.85
Most referenced
The records other records point at. Useful as an entry point: a heavily referenced accelerator or engine is one many decisions in this repo assume.