Unsloth DeepSeek-R1-Distill-Llama-8B (GGUF quant set)

CONTEXT SCALING FILLED - this record previously said native when its own config says otherwise. config.json declares config.json on the GGUF mirror carries the same rope_scaling {"rope_type":"llama3","factor":8.0,...,"original_max_position_embeddings":8192} and rope_theta 500000 as the base model. So the 131072 advertised window is an EXTRAPOLATION of an 8192-token trained base by 16x, not a native length. This is Llama-3.1's rope scheme, not YaRN and not PI: the llama3 type frequency-rescales high and low frequency bands separately (high_freq_factor 4.0, low_freq_factor 1.0), which is why it is neither a uniform position divide (PI) nor the NTK-style base adjustment. Not a distinct model - the repo's config.json is the base DeepSeek-R1-Distill-Llama-8B config unchanged. It is recorded because the artefact people actually run is the quant set, and the file sizes are the serving budget. Measured sizes from the repo tree on the access date: BF16 16.07 GB, F16 16.07 GB, Q8_0 8.54 GB, Q6_K 6.60 GB, Q5_K_M 5.73 GB, Q4_K_M 4.92 GB, Q4_K_S 4.69 GB, Q4_0 4.68 GB, IQ4_XS 4.46 GB, Q3_K_M 4.02 GB, Q2_K 3.18 GB, IQ1_S 2.16 GB. The UD-* dynamic quants sit alongside: UD-Q2_K_XL 3.39 GB, UD-Q3_K_XL 4.20 GB, UD-Q4_K_XL 4.99 GB, UD-Q6_K_XL 7.33 GB, UD-IQ2_XXS 2.50 GB, UD-IQ1_S 2.16 GB. Counter-intuitively the UD-* dynamic quants are SLIGHTLY LARGER than the same-bit standard quants at this size rather than smaller: UD-Q2_K_XL 3.39 GB vs Q2_K 3.18 GB (+6.6%), UD-Q4_K_XL 4.99 GB vs Q4_K_M 4.92 GB (+1.4%), UD-IQ1_S 2.16 GB = IQ1_S 2.16 GB. That is the honest reading of the measured sizes: the dynamic recipe buys accuracy by keeping outlier tensors at higher precision and using larger group sizes, and at 8B those costs are not repaid by the bits saved on the bulk of the weights. The case for dynamic quants is at larger scale, where outliers are a bigger share - see unsloth-hermes-4-405b-gguf, where UD-IQ1_S is 87.22 GB for a 405.9B model. The load-bearing serving fact: an 8B reasoning distill runs in 3-5 GB at these quantisations, so resident KV and KV-cache type, not weights, set the batch size at long context - the repo README's own llama.cpp example passes --cache-type-k q8_0. imatrix_unsloth.dat in the repo root is the importance matrix used for calibration. KV bytes/token at bf16 (2 bytes/element): 32 layers x 8 KV heads x 128 head_dim x 2 (K and V) x 2 bytes = 131072 B/token = 128.0 KiB/token. bf16 KV is 16.00 GiB per sequence at 128k, 3.3x the size of the 4.92 GB Q4_K_M weights. context_scaling is null: context length is native to the pretrained config, no rope_scaling. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=dense-gqa. REQUIRED KERNEL CLASS: ordinary causal attention plus GEMM/GEMV. This is the baseline every engine in this repo implements, so the join here is a floor rather than a filter: no engine record in data/engines/ states that dense GQA is unsupported. SUPPORTED BY (cited): vLLM LlamaForCausalLM / Qwen2 / Qwen3 / Mistral / Phi3ForCausalLM rows [[mdl2-vllm-supported-models-2026-10]]; TensorRT-LLM LlamaForCausalLM, Qwen2ForCausalLM, Qwen3ForCausalLM, Phi3ForCausalLM rows [[mdl2-trtllm-supported-models]]; SGLang generative table (Llama, Qwen, Mistral families) [[mdl2-sglang-generative-models]]; llama.cpp LLM_ARCH_LLAMA [[pap-arch-llamacpp-arch-registry]]. HARD CONSTRAINTS THAT DO APPLY even to dense GQA, and they are model-shape rather than architecture-class: tensor-parallel size must divide the attention head count or the engine refuses to start [[mgpu-tp-size-must-divide-attention-heads]] (affects vllm and sglang); and a GQA KV cache is the thing paged-attention machinery exists for, so an engine with no paged KV (llama.cpp's slot-based pool [[llama-cpp]], exllamav2 [[exllamav2]]) loses the concurrency lever rather than the ability to run. ONE HARDWARE-DEPENDENT KERNEL GATE THAT IS NOT ABOUT GQA: on RDNA, vLLM's custom paged-attention kernel is gated on head_size == 128 AND block_size == 16, so a dense GQA model with head_dim 256 falls to the Triton 2D fallback and decode falls off a cliff - measured 12.1 tok/s at 518 tokens of context to 4.2 tok/s at 32k, with kernel_paged_attention_2d growing 28.3x [[rdna-triton-paged-attn-decode-cliff]]. Check head_dim before blaming the card. CONTEXT_SCALING_METHOD (llama3): config-derived; the existing free text 'llama3 rope scaling (factor 8 from an 8192 base)' agrees.