LLMScalelive Hugging Face data

Model anatomy — where an LLM's parameters and memory go

Loading model anatomy…

What the anatomy shows

The anatomy view splits a model's parameters by where they live. Embeddings are the vocabulary table — counted twice when input and output embeddings are not tied. Attention is the query, key, value and output projections. Everything else, mostly the feed-forward network or the experts, is the remainder, so the parts always add up to the model's real total.

  • embeddings = vocabulary × hidden size (× 2 if untied)
  • attention = 2 × layers × hidden size × head_dim × (attention heads + key-value heads)
  • feed-forward / experts = total parameters − embeddings − attention
  • active experts (MoE) = feed-forward × experts per token ÷ experts

The KV cache chart plots one sequence at FP16 from zero to the model's maximum context, up to 128k tokens. For a model with grouped-query attention a dashed line shows what full multi-head attention would have cost — the gap is what GQA saves. The precision card shows what the weights weigh at BF16, FP8 or INT8, and INT4.

Questions

What is grouped-query attention (GQA)?
Several query heads share one set of key and value heads. The KV cache grows with the number of key-value heads, not query heads, so a model with 64 query heads and 8 key-value heads caches 8 times less than full multi-head attention would.
Why do embeddings take such a large share of small models?
The embedding table is vocabulary × hidden size, and modern vocabularies are large. In Qwen2.5 7B the embeddings hold about 1.09 billion parameters against 0.82 billion in attention; the feed-forward layers take most of the rest.
Why does a MoE model show far fewer active parameters?
Only a few experts run for each token. The active figure counts the embeddings, the attention and the share of expert parameters a token actually passes through — memory still has to hold all of them.
Where does the data come from?
From the model's config.json and the Hugging Face model API, fetched live in your browser. Downloads, likes, licence and repository size come from the same place.

Go deeper