MHA, MQA, and GQA: the cache is part of the architecture

Attention architecture is often introduced as a modeling choice. At inference time, it is also a memory-traffic choice.

The shared question

For a sequence length $T$, each generated token attends to cached keys and values from preceding tokens. Multi-head attention gives every query head its own key and value head. Multi-query attention shares one key/value head across all query heads. Grouped-query attention sits between them, sharing key/value heads within groups.

Variant	Query heads	KV heads	Inference intuition
MHA	$h$	$h$	Highest KV-cache footprint
MQA	$h$	$1$	Maximum sharing
GQA	$h$	$g$	Controlled sharing

If each head has dimension $d_h$, KV-cache storage is proportional to $2Tn_{kv}d_h$ per layer. The number of KV heads, not the number of query heads, is the term that changes across these variants.

The systems consequence

The useful engineering question is not which acronym is best in isolation. It is whether the model is compute-bound or bandwidth-bound at the intended batch size, context length, hardware, and serving setup. The KV cache is a first-class part of that answer.

This is a seed note: a place to extend with benchmarks, tensor shapes, and a small implementation walkthrough.