config.json, nvidia/Nemotron-H-8B-Base-8K

Fields

id

sources/arch-hf-config-nemotron-h

URL

https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K/raw/main/config.json

Publisher

Hugging Face

Kind

repo

Accessed

2026-10-03

notes

52 layers, hybrid_override_pattern 'M-M-M-M*-M-M-M-M-M*-...' giving 4 attention layers in 48 Mamba-2 layers; hidden 4096, expand 2, ssm_state_size 128, mamba_num_heads 128, mamba_head_dim 64, chunk_size 128, conv_kernel 4, n_groups 8; num_attention_heads 32, num_key_value_heads 8, attention_head_dim 128, intermediate 21504, max_position_embeddings 8192.

Notes

52 layers, hybrid_override_pattern 'M-M-M-M*-M-M-M-M-M*-...' giving 4 attention layers in 48 Mamba-2 layers; hidden 4096, expand 2, ssm_state_size 128, mamba_num_heads 128, mamba_head_dim 64, chunk_size 128, conv_kernel 4, n_groups 8; num_attention_heads 32, num_key_value_heads 8, attention_head_dim 128, intermediate 21504, max_position_embeddings 8192.

Cross-references

This record references no other records.