Marlin: FP16xINT4 LLM inference kernel (GitHub: IST-DASLab/marlin)

Fields

id

sources/qhw-src-marlin-repo

URL

https://github.com/IST-DASLab/marlin

Publisher

IST-DASLab

Kind

repo

Accessed

2026-10-04

notes

Marlin GitHub repository. FP16xINT4 mixed-precision matmul kernel for LLM inference. Claims near-ideal 4x speedups up to batchsizes of 16-32 tokens. Includes benchmark results on A10 GPU and Llama2 models. Requires CUDA >= 11.8, NVIDIA GPU with compute capability >= 8.0 (Ampere or Ada). The paper is arXiv:2408.11743.

Notes

Marlin GitHub repository. FP16xINT4 mixed-precision matmul kernel for LLM inference. Claims near-ideal 4x speedups up to batchsizes of 16-32 tokens. Includes benchmark results on A10 GPU and Llama2 models. Requires CUDA >= 11.8, NVIDIA GPU with compute capability >= 8.0 (Ampere or Ada). The paper is arXiv:2408.11743.

Cross-references

This record references no other records.