Transformers bitsandbytes integration (load_in_4bit / load_in_8bit via BitsAndBytesConfig)

PROXY ENUM VALUE - DECLARED 2026-10-04: this is the TRANSFORMERS INTEGRATION surface (BitsAndBytesConfig / load_in_4bit / load_in_8bit), filed under 'bitsandbytes-nf4'. It is a genuine near miss - the integration's 4-bit path IS NF4 or FP4, and the enum has no value for the integration layer - but note the config's own default is bnb_4bit_quant_type='fp4', not nf4, so this record is broader than the nf4 value it sits under and covers the FP4 variant too. The sibling bitsandbytes-nf4 record covers the format itself. FILLED 2026-10-04 from the library, and the distinction is the point: bitsandbytes functional.py quantize_4bit docstring says 'blocksize (int, optional): The size of the blocks. Defaults to 64. Valid values are 32, 64, 128, 256, 512, 1024, 2048, and 4096', and bitsandbytes/nn/modules.py Params4bit.__init__ does `if blocksize is None: blocksize = 64`. So 64 is the DEFAULT and a choice, NOT a constant of the NF4 format - the same NF4 data can be legally stored at any of eight sizes, which changes the memory and quality arithmetic by a large factor. KERNEL AVAILABILITY IS NARROWER THAN THE CONFIG SPACE: csrc/kernels.cu instantiates kQuantizeBlockwise for 64/128/256/512/1024/2048/4096 for NF4 and FP4, but kQuantizeBlockwiseSmall (the 4-bit fast path) ONLY for qblock_size 32 and 64. Separately, bitsandbytes-int8 (LLM.int8()) is vector-wise with a 6.0 outlier threshold and has NO group size - the null on that record is correct and must not be filled with 64. THE GOTCHA, captured accurately: bnb_4bit_quant_type selects the 4-bit DATA TYPE (nf4 or fp4) and bnb_4bit_compute_dtype selects the DTYPE THE DEQUANTIZED MATMUL RUNS IN - the documented example sets bnb_4bit_compute_dtype=torch.bfloat16. The doc's guidance is that for inference bnb_4bit_quant_type 'does not have a huge impact on performance', but you should keep bnb_4bit_compute_dtype and dtype consistent 'to remain consistent with the model weights'. NF4 is the QLoRA-derived type for weights initialized from a normal distribution and is the one to use for training 4-bit base models. So the real decision is the COMPUTE dtype and the load path, not nf4-vs-fp4. HARDWARE FLOOR, documented as a table: NF4/FP4 quantization needs NVIDIA Pascal (GTX 10X0 / P100) or newer; LLM.int8() needs Turing (RTX 20X0 / T4) or newer; 8-bit optimizers need Pascal or newer. Supported platforms are NVIDIA GPUs on CUDA 11.8-13.0 (Linux x86-64, Linux aarch64, Windows), Intel XPU (Linux and Windows x86-64), Intel Gaudi HPU (Linux x86-64, Gaudi2 and Gaudi3) and CPU (Linux x86-64/aarch64, Windows x86-64). Note what is NOT on that list: no ROCm/HIP and no AMD GPU entry, so on this repo's AMD accelerators bitsandbytes is effectively absent - that is a real portability limit, not an oversight. OTHER BEHAVIOURAL FACTS worth carrying: non-Linear modules such as torch.nn.LayerNorm are left at the default torch dtype, changeable via the `dtype` parameter (dtype='auto' takes it from config.json). CPU offload works for the 8-bit path (llm_int8_enable_fp32_cpu_offload=True) and weights dispatched to CPU are stored in float32 and are NOT converted to 8-bit - so a partly-offloaded model has mixed precision by construction. 8- and 4-bit training is only supported for training EXTRA parameters. The outlier threshold for LLM.int8() defaults to 6. Pushing quantized weights to the Hub requires recent versions of both transformers and bitsandbytes, and quantized models can be dequantized back (`model.dequantize()`) with possible quality loss.