DeepSeek-R1 and R1-Distill model card (huggingface.co/deepseek-ai/DeepSeek-R1)

R1 itself is architecturally identical to DeepSeek-V3 (config model_type deepseek_v3, same 61 layers / 256 experts / 8 active / kv_lora_rank 512). The distills are separate architectures: the Qwen-based ones use architectures Qwen2ForCausalLM (model_type qwen2) and the Llama-70B one uses LlamaForCausalLM with rope_scaling type llama3 factor 8. R1-Distill-Qwen-32B and -14B set sliding_window to the full 131072 (i.e. effectively no sliding), and rope_theta 1000000. The distills are MIT. There is a DeepSeek-R1-Distill-Qwen-1.5B (hidden 1536, 28 layers, 12 heads / 2 kv heads, head_dim 128 implied) whose config does NOT set head_dim, so its KV derivation relies on hidden_size/num_attention_heads.