JOURNAL / MODELS & COMPUTATION

What is a KV cache?
The working memory behind each token.

A key-value cache stores attention keys and values already computed for a request. It lets a language model reuse earlier work as it generates the next token—and explains why long contexts can need far more runtime memory than a model’s weight file suggests.

Four computer memory modules arranged on a pale textured surface.SERVING / MEMORY / SYMPLOKA
CONTEXT BECOMES RUNTIME STATE

A prompt is more than a number in a context window.

As a transformer reads tokens, it builds intermediate representations that can be reused at the next decoding step. The KV cache keeps part of that work ready while a sequence is active.

01 / WHAT THE CACHE STORES

Each token leaves reusable attention state.

In a standard autoregressive transformer, every attention layer projects a token’s representation into a key and a value. When the model generates the next token, its query is compared with earlier keys; the resulting attention weights combine the corresponding values. The model appends the new token’s keys and values to the cache as generation continues.

Without a cache, a straightforward decoder could recompute key and value projections for the whole preceding sequence at every step. Reusing them avoids that repeated work. The cache does not contain a second copy of the prompt as readable text: it is numerical state, stored across attention layers for the active sequence.

Short version: the model weights describe what was learned; the KV cache holds intermediate attention state for tokens processed in this run.

For the larger sequence of prompt processing and token generation, see how AI inference works ↗.

02 / CACHE, CONTEXT, MEMORY

Three kinds of “memory” answer different questions.

Model weightsLearned parameters loaded for inference. Their size depends mostly on the model and its numeric format, not on how many tokens one request has processed.
KV cachePer-sequence key and value tensors accumulated during inference. Its size usually grows with active token positions and the model’s attention layout.
Context windowA token limit for what a model call can process. It is a capacity boundary, not a measurement of bytes currently used by cache.
Persistent memoryInformation an application stores for later use, such as saved preferences or conversation history. It is separate product behavior, outside the ordinary KV cache for one active sequence.

A serving engine can keep or share reusable prefixes across requests, subject to its own caching policy. That is an engine-level optimization; it does not make the model’s request state the same thing as a product’s saved memory. The PagedAttention paper ↗ describes on-demand cache blocks, sharing, and the fragmentation problems that arise when variable-length sequences are served together.

For a practical distinction between per-request KV state and prefix reuse across calls, see the prompt-caching guide ↗.

For a product-level view of stored information, read memory as a design choice ↗.

03 / ESTIMATE THE MEMORY

Count layers, active tokens, and KV heads.

For a conventional full-attention transformer, a useful first-order estimate for one sequence is:

cache bytes ≈ 2 × layers × active tokens × KV heads × head dimension × bytes per value

The factor of two counts keys and values. “Active tokens” means positions currently represented in the cache, including the prompt and generated tokens retained so far.

For example, assume 32 layers, 4,096 active positions, 8 KV heads, a head dimension of 128, and 16-bit values. That gives 536,870,912 bytes, or 512 MiB (0.5 GiB), for one sequence. This is an illustration from those assumptions, not a specification for a named model.

Real use can be higher or lower. The estimate leaves out allocator and block rounding, metadata, temporary workspaces, and other runtime buffers. Sliding-window attention may retain fewer positions; model architectures with different KV-head counts change the result. Concurrent sequences can multiply demand, while shared prefixes may reduce duplication in systems that support them.

Do not confuse “fits the context window” with “fits in memory at the target batch size.” For model choice and device budgets, compare the cache with the weight and runtime footprints; the small-model guide ↗ and the token and context guide ↗ cover the related limits.

04 / THREE DESIGN LEVERS

Architecture, allocation, and precision shape the trade-off.

Fewer KV headsMulti-query attention shares keys and values across query heads. Grouped-query attention uses an intermediate number of KV heads. Both can reduce cache size relative to standard multi-head attention; the model architecture sets the layout.
Paged allocationPagedAttention stores cache in blocks allocated as sequences grow. This addresses fragmentation and supports flexible sharing; it changes how memory is managed, not the number of values the architecture needs.
Lower-precision cacheCache quantization stores keys and values with fewer bits, targeting runtime state rather than model weights. It can lower memory demand, with quality and speed effects that depend on the method and workload.
Request limitsPrompt length, output limits, concurrency, and batching determine how many active cache positions the runtime needs to support at once.

The multi-query attention paper ↗ introduces shared keys and values; the grouped-query attention paper ↗ studies a middle ground between one shared KV head and one per query head.

Cache precision is a separate choice from weight precision. The KIVI study ↗ reports results for its particular 2-bit method and tested models; treat those results as experimental evidence, not a guaranteed reduction for every runtime. See how quantization changes model precision ↗.

Another way to reduce serial decoding steps is speculative decoding ↗, which drafts candidate tokens and checks them against the target model.

05 / A PRACTICAL CHECK

Size the workload that will actually run.

  1. Inspect model metadata. Record layer count, KV-head count, head dimension, attention type, and cache data type.
  2. Use realistic token lengths. Include system instructions, retrieved material, conversation history, and expected output—not only the user’s latest message.
  3. Set concurrency explicitly. Estimate simultaneous sequences and whether common prefixes are shared by the runtime.
  4. Measure peak memory. Include weights, KV cache, runtime buffers, and allocator overhead on the target device.
  5. Benchmark the chosen format. If using paged allocation or cache quantization, measure latency, throughput, and task quality with the exact engine and workload.

The important number is not the cache for an empty request. It is the peak memory and response quality under representative prompt lengths and load. Continue with a whole-system cost framework ↗ or the inference overview ↗.

PRIMARY PAPERSKwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) ↗Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019) ↗Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023) ↗Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (ICML 2024) ↗