inference/research
verifiedhigh · 0.95updated 2026-10-05

LMDeploy published INT8 and FP8 per-architecture availability lists (docs/en/quantization/w8a8.md) — INT8 from Volta up, FP8 Ada and Hopper only

repo

Fields

id
sources/w6q-lmdeploy-int8-fp8-arch-lists-2026-10
Publisher
InternLM (lmdeploy)
Kind
repo
Accessed
2026-10-05
notes
Read 2026-10-05 from InternLM/lmdeploy main. LMDeploy publishes its INT8/FP8 arch coverage as prose rather than as code, which makes it a transcription-type source and therefore a weaker gate than the vLLM/SGLang/TensorRT-LLM source reads — but it is the engine's own statement and it is narrower than the general expectation. Quoted: 'And the following NVIDIA GPUs are available for INT8/FP8 inference respectively:' INT8 lists 'V100(sm70): V100', 'Turing(sm75): 20 series, T4', 'Ampere(sm80,sm86): 30 series, A10, A16, A30, A100', 'Ada Lovelace(sm89): 40 series', 'Hopper(sm90): H100'; FP8 lists only 'Ada Lovelace(sm89): 40 series' and 'Hopper(sm90): H100'. Two readings. First, FP8 in LMDeploy is Ada-and-Hopper only — it does not claim Blackwell at all, and the doc names H100 rather than the H200/H800 parts, so this list has not been updated for Blackwell or for the H200. Second, the INT8 list starting at Volta (sm70) is the OPPOSITE of the Marlin floor: LMDeploy's INT8 W8A8 runs on V100 where Marlin-based W4A16 does not, which is consistent with INT8 being a cuBLASLt W8A8 path rather than a Marlin path, and it means the repo's 'Volta has no Marlin/AWQ' negative must be stated per format-family, not as 'Volta cannot do quantized inference'. The same doc also records that the quantization is produced by the SmoothQuant pipeline (lmdeploy lite smooth_quant --quant-dtype int8|fp8) rather than by a format-native quantizer, and that inference is selected by backend: `pipe = pipeline(..., backend_config=engine_config)` with `lmdeploy serve api_server ... --backend pytorch` — i.e. the TurboMind backend is not the path this page documents.

Notes

Read 2026-10-05 from InternLM/lmdeploy main. LMDeploy publishes its INT8/FP8 arch coverage as prose rather than as code, which makes it a transcription-type source and therefore a weaker gate than the vLLM/SGLang/TensorRT-LLM source reads — but it is the engine's own statement and it is narrower than the general expectation. Quoted: 'And the following NVIDIA GPUs are available for INT8/FP8 inference respectively:' INT8 lists 'V100(sm70): V100', 'Turing(sm75): 20 series, T4', 'Ampere(sm80,sm86): 30 series, A10, A16, A30, A100', 'Ada Lovelace(sm89): 40 series', 'Hopper(sm90): H100'; FP8 lists only 'Ada Lovelace(sm89): 40 series' and 'Hopper(sm90): H100'. Two readings. First, FP8 in LMDeploy is Ada-and-Hopper only — it does not claim Blackwell at all, and the doc names H100 rather than the H200/H800 parts, so this list has not been updated for Blackwell or for the H200. Second, the INT8 list starting at Volta (sm70) is the OPPOSITE of the Marlin floor: LMDeploy's INT8 W8A8 runs on V100 where Marlin-based W4A16 does not, which is consistent with INT8 being a cuBLASLt W8A8 path rather than a Marlin path, and it means the repo's 'Volta has no Marlin/AWQ' negative must be stated per format-family, not as 'Volta cannot do quantized inference'. The same doc also records that the quantization is produced by the SmoothQuant pipeline (lmdeploy lite smooth_quant --quant-dtype int8|fp8) rather than by a format-native quantizer, and that inference is selected by backend: `pipe = pipeline(..., backend_config=engine_config)` with `lmdeploy serve api_server ... --backend pytorch` — i.e. the TurboMind backend is not the path this page documents.

Cross-references

This record references no other records.