Unsloth Hermes-4-405B (GGUF quant set)

CONTEXT SCALING FILLED - this record previously said native when its own config says otherwise. config.json declares rope_scaling {"rope_type":"llama3","factor":8.0,...,"original_max_position_embeddings":8192}, rope_theta 500000, max_position_embeddings 131072. So the 131072 advertised window is an EXTRAPOLATION of an 8192-token trained base by 16x, not a native length. This is Llama-3.1's rope scheme, not YaRN and not PI: the llama3 type frequency-rescales high and low frequency bands separately (high_freq_factor 4.0, low_freq_factor 1.0), which is why it is neither a uniform position divide (PI) nor the NTK-style base adjustment. Same weights as NousResearch/Hermes-4-405B - config.json matches - but shipped as the only practical single-node artefact. Sums of per-file sizes measured from the repo tree on the access date: BF16 811.75 GB across 17 shards, Q8_0 431.24 GB across 9 shards, UD-Q5_K_XL 286.50 GB across 6 shards, Q4_K_M 243.07 GB across 5 shards, UD-Q4_K_XL 243.72 GB across 5 shards, UD-Q3_K_XL 197.39 GB across 4 shards, UD-Q2_K_XL 151.36 GB across 4 shards, UD-IQ1_M 94.73 GB across 2 shards, UD-IQ1_S 87.22 GB across 2 shards. These are sharded repos: each directory is one logical quantisation split into <=50 GB parts, so a deployment must fetch every shard in a directory. Serving consequence: the IQ1 dynamic quants (87-95 GB) are the only variants of this model that fit in two 80GB SXM boards, and at UD-IQ1_S the weights alone are ~87 GB - a single sequence's bf16 KV at 128k is 63 GiB, so long-context serving of this model is impossible without KV quantisation or offload even where the weights fit. KV bytes/token at bf16 (2 bytes/element): 126 layers x 8 KV heads x 128 head_dim x 2 (K and V) x 2 bytes = 516096 B/token = 504.0 KiB/token. context_scaling is null: context length is native to the pretrained config, no rope_scaling. SERVICEABILITY JOIN (added 2026-10-03; the architecture x engine matrix this record was missing, with a citation per cell in docs/13-model-compatibility.md): FAMILY=dense-gqa. REQUIRED KERNEL CLASS: ordinary causal attention plus GEMM/GEMV. This is the baseline every engine in this repo implements, so the join here is a floor rather than a filter: no engine record in data/engines/ states that dense GQA is unsupported. SUPPORTED BY (cited): vLLM LlamaForCausalLM / Qwen2 / Qwen3 / Mistral / Phi3ForCausalLM rows [[mdl2-vllm-supported-models-2026-10]]; TensorRT-LLM LlamaForCausalLM, Qwen2ForCausalLM, Qwen3ForCausalLM, Phi3ForCausalLM rows [[mdl2-trtllm-supported-models]]; SGLang generative table (Llama, Qwen, Mistral families) [[mdl2-sglang-generative-models]]; llama.cpp LLM_ARCH_LLAMA [[pap-arch-llamacpp-arch-registry]]. HARD CONSTRAINTS THAT DO APPLY even to dense GQA, and they are model-shape rather than architecture-class: tensor-parallel size must divide the attention head count or the engine refuses to start [[mgpu-tp-size-must-divide-attention-heads]] (affects vllm and sglang); and a GQA KV cache is the thing paged-attention machinery exists for, so an engine with no paged KV (llama.cpp's slot-based pool [[llama-cpp]], exllamav2 [[exllamav2]]) loses the concurrency lever rather than the ability to run. ONE HARDWARE-DEPENDENT KERNEL GATE THAT IS NOT ABOUT GQA: on RDNA, vLLM's custom paged-attention kernel is gated on head_size == 128 AND block_size == 16, so a dense GQA model with head_dim 256 falls to the Triton 2D fallback and decode falls off a cliff - measured 12.1 tok/s at 518 tokens of context to 4.2 tok/s at 32k, with kernel_paged_attention_2d growing 28.3x [[rdna-triton-paged-attn-decode-cliff]]. Check head_dim before blaming the card. CONTEXT_SCALING_METHOD (llama3): config-derived; the existing free text 'llama3 rope scaling (factor 8 from an 8192 base)' agrees.