> ## Documentation Index
> Fetch the complete documentation index at: https://docs.deploy.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt Caching

Improve performance at scale by caching repeated input context.

***

Prompt caching helps reduce repeated input processing when multiple requests share the same context. This can improve response latency and reduce input-token costs.

Cached input tokens are billed at a lower rate. See [Models and Pricing](https://docs.deploy.com/models-and-pricing.md#pricing) for more information.

Support is per-model—check the **Prompt Caching** column under [Available Models](https://docs.deploy.com/models-and-pricing.md#available-open-source-models).

***

# **How prompt caching works**

When an application sends a request to an LLM, the model processes the input tokens before it can generate a response. That input can include system instructions, conversation history, examples, tool definitions, and reference documents. The model calculates intermediate states, known as key-value (KV) states, which let the model refer back to earlier tokens while processing new input and generating output tokens.

Many developer workflows send some of that same context repeatedly. Prompt caching lets the inference system reuse previously computed KV states for subsequent requests, instead of processing the same input from scratch repeatedly.

The model does not cache the final answer—it still generates a response for each request.​

***

# **Enabling prompt caching**

Prompt caching is **automatic**.

We support cache-aware routing to help maximize cache hit rates, and are actively working on improvements to increase the hit rate. Sending requests with shared prompt prefixes and sending these in close succession can improve cache reuse in the meantime.

We cannot guarantee cache retention for a specific period of time or across requests.

***

# **Using caching in requests**

Prompt caching is most useful when requests share a long prefix. The following example sends a request with a stable system prompt and a variable user message:

```python Python theme={null}
import os
from openai import OpenAI

client = OpenAI(
    api_key="DEPLOYAI_API_KEY",
    base_url="DEPLOYAI_BASE_URL",
)

model = "DEPLOYAI_MODEL"

system_prompt = """
You are a technical support assistant.

Answer questions using the provided product documentation.
Be concise, accurate, and explicit when information is unavailable.

Product documentation:
- The platform provides hosted language models.
- Applications can send chat completion requests.
- API behavior depends on the selected model and configuration.
"""

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": "How do I choose a model for my application?",
        },
    ],
)

print(response.choices[0].message.content)
```

The first request processes the prompt. If the selected model supports prompt caching, eligible context may be cached for reuse in a subsequent request.

The second request keeps the system prompt unchanged, but sends a different user question.

```python Python theme={null}
response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": "How can I monitor API usage?",
        },
    ],
)

print(response.choices[0].message.content)
```

***


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.