What is prompt caching?
Reuse the work that repeats.

Keep the part that every request must read.

Before generating a response, a model processes its input: system instructions, tool definitions, conversation context, and the latest message. Processing those tokens creates intermediate attention state. If another request begins with the same prefix and the service can find a matching cache entry, it can reuse that state instead of calculating it again.

The prefix has to match according to the model provider or runtime’s rules. New content after the shared portion is still processed for the current request. Prompt caching can reduce repeated input work and, depending on the service, input cost or the time before generation starts. It does not remove the need to generate a new answer.

The cached state is closely related to the key-value state used by a KV cache ↗. The difference is the reuse scope: a typical KV cache serves one active sequence, while prefix caching lets a runtime reuse eligible state across requests.

Cache, context, memory, and weights are not interchangeable.

These terms can overlap at the implementation level: a prompt cache may reuse KV tensors. They describe different jobs, though. One names the numerical state, another names a cross-request reuse strategy, and application memory names data retained for future decisions.

For the mechanics and memory cost of keys and values, continue to the KV-cache guide ↗. For deliberate, longer-lived information in an agent or product, see memory as a design choice ↗.

Put stable instructions before changing details.

A request series often has a stable part and a changing part. Put content that is identical across requests first: durable instructions, repeated examples, and tool definitions that stay the same. Add the user’s current question, request identifiers, timestamps, and any changing context after that shared prefix.

For example, a coding assistant may send the same role instructions and tool schemas on every turn, then append a different repository question. A support assistant may reuse a policy document while adding a new customer message. Only the identical prefix is a cache candidate; the varied question or document still needs its own processing.

A small change near the beginning can make everything after it a different prefix. The precise matching boundary, cacheable content, and breakpoint controls vary by provider and serving engine. Read the current documentation for the system you run; do not assume every API handles the same request in the same way.

For the broader discipline of selecting and ordering useful context, see context engineering for agents ↗ and how to write a clearer prompt ↗.

Less repeated input work is not less work everywhere.

A cache hit can shorten prompt processing and may reduce the cost of cached input tokens under a provider’s pricing rules. It does not make a new suffix free, shorten the answer the model needs to generate, or guarantee a faster end-to-end workflow. A small prompt with few repeated tokens may see little benefit; a long, stable prefix used repeatedly has more opportunity to repay the extra cache bookkeeping.

Caching is designed as computation reuse, not a change to the model’s instructions or a separate quality technique. The output still depends on the model, the complete request, and the generation settings. Compare response quality as well as speed and spend when changing prompt layout.

To assess the wider trade-off, connect cache measurements with inference latency ↗ and the cost of the full AI system ↗.

Track cache hits beside the answer they served.

1. Record cached reads and writes. Use the usage fields or diagnostics exposed by your provider or engine; field names and availability differ.
2. Measure time to first token. Separate prompt processing from output streaming, then compare matched workloads.
3. Calculate actual input cost. Apply the model’s current cached, uncached, and cache-write rates to observed token counts.
4. Include misses and cold starts. Report a distribution across real request sequences, not only the best cache hit.
5. Check the task result. Confirm that prompt reordering still supplies the same relevant context and produces acceptable answers.

Keep model version, tools, prompt content, region, concurrency, and request mix comparable. A high hit count is useful only if it lowers a measure that matters to the product without weakening the result.

Treat cache scope as part of the data path.

Managed providers and self-hosted runtimes set their own cache keys, eligible content, retention, and isolation boundaries. A cache hit is not evidence that data is stored forever, and a cache miss is not evidence that it was never retained elsewhere. Check the provider’s current data and caching documentation, then include cache behavior in the application’s privacy and threat review.

In a multi-tenant self-hosted service, make tenant boundaries explicit in the cache design. The vLLM documentation, for example, describes cache salts as an input that can help isolate cache keys across tenants. That is a runtime-specific control, not a universal setting; verify the actual engine and configuration you deploy.

Prompt caching also does not replace authorization or retrieval controls. Every request still needs the correct user scope, current access checks, and only the context that user is permitted to receive. See the full data-path view of local AI privacy ↗.

Use it when the same prefix earns its place.

Prompt caching is a good candidate when requests repeatedly carry substantial, unchanged context: long instructions, stable tool definitions, shared reference material, or a growing conversation prefix. First confirm that the model and runtime support the behavior, arrange the prompt so stable content comes first, and inspect real usage fields.

Do not reshape a prompt just to chase a cache hit if that makes the instructions harder to maintain or the context less relevant. Compare cost, first-token latency, response quality, and cache misses on the request pattern people actually use.

Then connect caching to its neighboring ideas: KV-cache memory ↗, draft-and-verify generation ↗, and model quantization ↗ target different parts of inference and should be measured separately.