llamafile Supported Systems docs (IQ-quants on NVIDIA section)

Fields

id

sources/qlab2-llamafile-support-docs

URL

https://docs.mozilla.ai/llamafile/reference/support

Publisher

Mozilla AI (mozilla-ai)

Kind

repo

Accessed

2026-10-03

notes

The IQ-quant interop fact: llamafile's bundled CUDA library is size-optimized and OMITS the IQ-quant kernels (IQ1_*, IQ2_*, IQ3_*, IQ4_*). Offloading an IQ-quantized model to an NVIDIA GPU therefore leaves just those layers on the CPU - output is correct but those layers get no GPU acceleration. Metal and ROCm builds are not size-optimized and include full IQ-quant GPU support. | SOURCE AUDIT 2026-10-03: the 'published' field held a verbatim copy of this notes paragraph instead of a date (field corruption across the qlab2-* batch); published set to null because a GitHub README, doc page or source header has no reliable publication date of its own. Content of the document itself was re-verified live 2026-10-03 and the notes are correct.

Notes

The IQ-quant interop fact: llamafile's bundled CUDA library is size-optimized and OMITS the IQ-quant kernels (IQ1_*, IQ2_*, IQ3_*, IQ4_*). Offloading an IQ-quantized model to an NVIDIA GPU therefore leaves just those layers on the CPU - output is correct but those layers get no GPU acceleration. Metal and ROCm builds are not size-optimized and include full IQ-quant GPU support. | SOURCE AUDIT 2026-10-03: the 'published' field held a verbatim copy of this notes paragraph instead of a date (field corruption across the qlab2-* batch); published set to null because a GitHub README, doc page or source header has no reliable publication date of its own. Content of the document itself was re-verified live 2026-10-03 and the notes are correct.

Cross-references

This record references no other records.