Aleph-Alpha/Kolibri-1 config.json

Raw config.json fetched 2026-10-17. model_type kolibri1, architectures Kolibri1ForCausalLM. hidden_size 2560, num_hidden_layers 50, num_attention_heads 48, num_key_value_heads 4, head_dim 128, hidden_act silu, vocab_size 128000, rope_theta 10000, no rope_scaling key. num_experts 384, num_experts_per_tok 6, moe_intermediate_size 512, shared_expert_intermediate_size 512. use_sliding_window true, sliding_window 513. layer_types is 40 sliding_attention and 10 full_attention, full attention at indices 4,9,14,19,24,29,34,39,44,49. max_position_embeddings in this file is 262144, the card recommended serving length, not the card 1,048,576 nameplate. quantization_config: fp8, activation_scheme dynamic, weight_block_size [128, 128], every MoE gate left unconverted.