MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models

Directly attacks the question of whether 4-bit weight quantization can keep large batch sizes bandwidth-bound. Reports near-maximum (~4x) quantization speedup at batch 16-32, gradually decreasing but still significant through batch 64-128, via asynchronous memory access, task scheduling/pipelining and bespoke quantization support. Up to 2.8x end-to-end when integrated with vLLM.