inference/research
verifiedmedium · 0.80updated 2026-10-03

TinyChat (AWQ on-device runtime)

cuda

Fields

id
engines/tinychat-awq
Repo
mit-han-lab/llm-awq
Languages
PythonCUDAC++
License
MIT
First release
2023
Design goal
low-latencyminimal-deps
Scheduling
none
Backends
cuda
Formats
awq
Notable features
Platform-specific weight packing for 4-bit SIMD registersFused dequant + GEMM kernels; KV cache pre-allocated and updated inside the attention kernelTargets edge CPUs and GPUs, not datacenter serving
Hardware caveats
W4A16 only - activations and KV cache stay FP16, so no INT4 tensor-core op is usedQuantized conv/packing order is platform-specific: a file packed for ARM SIMD is not optimal on GPU
Best for
single-stream on-device generation on consumer GPUs and ARM CPUs
Avoid for
multi-user serving - no continuous batching or request scheduling
Docs
https://github.com/mit-han-lab/llm-awq/tree/main/awq/kernels
notes
The runtime that ships with the AWQ paper, not a general-purpose engine. Recorded here because AWQ's quality numbers were produced by evaluating quantized checkpoints with this runtime's kernels and the paper's own harness; do not read TinyChat's throughput numbers as engine benchmarks. AWQ's own speed claim in the paper is >3x over HF FP16 on desktop and mobile GPUs, not the ~4x commonly quoted from Marlin. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2025-07-17 (14.6 months before this record), NO GitHub release or tag found, 16 contributors (distinct committers including anonymous), open_issues_count 207 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 3645 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.

Notes

The runtime that ships with the AWQ paper, not a general-purpose engine. Recorded here because AWQ's quality numbers were produced by evaluating quantized checkpoints with this runtime's kernels and the paper's own harness; do not read TinyChat's throughput numbers as engine benchmarks. AWQ's own speed claim in the paper is >3x over HF FP16 on desktop and mobile GPUs, not the ~4x commonly quoted from Marlin. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2025-07-17 (14.6 months before this record), NO GitHub release or tag found, 16 contributors (distinct committers including anonymous), open_issues_count 207 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 3645 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.

Cross-references

This record references no other records — it is a leaf. It is cited by 4.