Llama-2-13B Hugging Face config.json (raw)

Fields

id

sources/fct2-hf-config-llama2-13b

URL

https://huggingface.co/NousResearch/Llama-2-13b-hf/raw/main/config.json

Publisher

Meta / Hugging Face

Kind

repo

Accessed

2026-10-04

notes

Read 2026-10-04 for the FLOP-crossover record. hidden_size 5120, num_hidden_layers 40, num_attention_heads 40, num_key_value_heads 40 (MHA, so n_q*d_h = 5120 = hidden_size), intermediate_size 13824, vocab 32000, max_position_embeddings 4096. Used because Llama-2-13B is the cleanest second MHA point in the family and its config needs no head_dim inference.

Notes

Read 2026-10-04 for the FLOP-crossover record. hidden_size 5120, num_hidden_layers 40, num_attention_heads 40, num_key_value_heads 40 (MHA, so n_q*d_h = 5120 = hidden_size), intermediate_size 13824, vocab 32000, max_position_embeddings 4096. Used because Llama-2-13B is the cleanest second MHA point in the family and its config needs no head_dim inference.

Cross-references

This record references no other records.