Meta-Llama-3-8B config.json (open mirror)

Geometry for the GQA + large-vocabulary reference: hidden_size 4096, intermediate_size 14336, num_hidden_layers 32, num_attention_heads 32, num_key_value_heads 8 (GQA g=4), head_dim 128, vocab_size 128256 (4x Llama 2's 32000), max_position_embeddings 8192, rope_theta 500000.0, torch_dtype bfloat16, tie_word_embeddings false. Derived parameter count from these numbers reproduces the released 8.03B exactly. The 128256-token vocabulary is what makes the LM head a second full decode GEMM and what makes grammar-mask work scale with vocab.