Hugging Face config.json files for the long-context model families (fetched 2026-10-04)

Raw config.json read from the HF CDN on 2026-10-04 for each model. This is the machine-readable ground truth for claimed context length (max_position_embeddings or seq_length) and for the geometry that sets KV bytes per token. Values read verbatim: NousResearch/Meta-Llama-3.1-8B-Instruct and -70B-Instruct: max_position_embeddings 131072, hidden 4096/8192, 32/80 layers, 8 KV heads, head_dim 128 (32/64 query heads), intermediate 14336/28672, rope_theta 500000, rope_scaling {rope_type llama3, factor 8.0, low_freq_factor 1.0, high_freq_factor 4.0, original_max_position_embeddings 8192}. nvidia/Llama-3.1-405B-Instruct-FP8: 131072, hidden 16384, 126 layers, 8 KV heads, 128 query heads, head_dim 128, intermediate 53248, identical llama3 rope_scaling and theta 500000. Qwen/Qwen2.5-7B-Instruct: 32768, hidden 3584, 28 layers, 4 KV heads, 28 query, head_dim 128, rope_theta 1000000, NO rope_scaling, sliding_window field 131072 but use_sliding_window false. Qwen/Qwen2.5-72B-Instruct: 32768, hidden 8192, 80 layers, 8 KV, 64 query, theta 1000000, no rope_scaling. Qwen/Qwen2.5-7B-Instruct-1M: max_position_embeddings 1010000, same geometry as the 32K 7B, theta 10000000000, no rope_scaling, sliding_window 32768. Qwen/Qwen3-8B: 40960, 36 layers, 8 KV, 32 query, head_dim 128, theta 1000000, rope_scaling null. Qwen/Qwen3-235B-A22B: 40960, 94 layers, 4 KV, 64 query, head_dim 128, theta 1000000, rope_scaling null. Qwen/Qwen3-Next-80B-A3B-Instruct and -Thinking: max_position_embeddings 262144, 48 layers, full_attention_interval 4 (so 12 of 48 layers are full attention, the rest are gated deltanet), 2 KV heads, 16 query, head_dim 256, partial_rotary_factor 0.25, rope_theta 10000000, rope_scaling null. mistralai/Mistral-Large-Instruct-2411: 131072, hidden 12288, 88 layers, 8 KV, 96 query, head_dim 128, theta 1000000, no rope_scaling. CohereLabs c4ai-command-r-plus-08-2024 is gated; geometry read from the ungated mirror mlx-community/c4ai-command-r-plus-08-2024-4bit: 131072, hidden 12288, 64 layers, 8 KV, 96 query, head_dim 128, rope_theta 8000000, vocab 256000, model_type cohere. 01-ai/Yi-34B-200K: max_position_embeddings 200000, hidden 7168, 60 layers, 8 KV, 56 query, head_dim 128, theta 10000000000, rope_scaling null. THUDM/glm-4-9b-chat-1m: seq_length 1048576, hidden 4096, 40 layers, 32 query, kv_channels 128, multi_query_attention true, multi_query_group_num 4, rope_ratio 10000, original_rope true. THUDM/chatglm3-6b-128K: seq_length 131072, hidden 4096, 32 query, kv_channels 128, multi_query_group_num 2. deepseek-ai/DeepSeek-V2: 163840, 60 layers, kv_lora_rank 512, qk_rope_head_dim 64, rope_theta 10000, rope_scaling {type yarn, factor 40, mscale 0.707, mscale_all_dim 0.707, beta_fast 32, beta_slow 1, original_max_position_embeddings 4096}. deepseek-ai/DeepSeek-V3: same MLA geometry with 61 layers, rope_scaling {type yarn, factor 40, mscale 1.0, mscale_all_dim 1.0, beta_fast 32, beta_slow 1, original_max_position_embeddings 4096}, rope_theta 10000. deepseek-ai/DeepSeek-V3.2: identical to V3 including the same yarn factor 40 and 163840. moonshotai/Kimi-K2-Instruct: 131072, 61 layers, hidden 7168, 64 query, kv_lora_rank 512, qk_rope_head_dim 64, rope_theta 50000, rope_scaling {type yarn, factor 32.0, mscale 1.0, mscale_all_dim 1.0, beta_fast 1.0, beta_slow 1.0, original_max_position_embeddings 4096}, 384 routed experts, 8 per token. zai-org/GLM-4.5: 131072, hidden 5120, 92 layers, 8 KV, 96 query, head_dim 128, theta 1000000, rope_scaling null.