Improve performance at scale by caching repeated input context.

Prompt caching helps reduce repeated input processing when multiple requests share the same context. This can improve response latency and reduce input-token costs. Cached input tokens are billed at a lower rate. See Models and Pricing for more information. Support is per-model—check the Prompt Caching column under Available Models.

How prompt caching works

When an application sends a request to an LLM, the model processes the input tokens before it can generate a response. That input can include system instructions, conversation history, examples, tool definitions, and reference documents. The model calculates intermediate states, known as key-value (KV) states, which let the model refer back to earlier tokens while processing new input and generating output tokens. Many developer workflows send some of that same context repeatedly. Prompt caching lets the inference system reuse previously computed KV states for subsequent requests, instead of processing the same input from scratch repeatedly. The model does not cache the final answer—it still generates a response for each request.

Enabling prompt caching

Prompt caching is automatic. We support cache-aware routing to help maximize cache hit rates, and are actively working on improvements to increase the hit rate. Sending requests with shared prompt prefixes and sending these in close succession can improve cache reuse in the meantime. We cannot guarantee cache retention for a specific period of time or across requests.

Using caching in requests

Prompt caching is most useful when requests share a long prefix. The following example sends a request with a stable system prompt and a variable user message:

import os
from openai import OpenAI

client = OpenAI(
    api_key="DEPLOYAI_API_KEY",
    base_url="DEPLOYAI_BASE_URL",
)

model = "DEPLOYAI_MODEL"

system_prompt = """
You are a technical support assistant.

Answer questions using the provided product documentation.
Be concise, accurate, and explicit when information is unavailable.

Product documentation:
- The platform provides hosted language models.
- Applications can send chat completion requests.
- API behavior depends on the selected model and configuration.
"""

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": "How do I choose a model for my application?",
        },
    ],
)

print(response.choices[0].message.content)

The first request processes the prompt. If the selected model supports prompt caching, eligible context may be cached for reuse in a subsequent request. The second request keeps the system prompt unchanged, but sends a different user question.

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": system_prompt},
        {
            "role": "user",
            "content": "How can I monitor API usage?",
        },
    ],
)

print(response.choices[0].message.content)

Get Started

Chat Completions

Large Language Models

Security & Compliance

Prompt Caching

How prompt caching works

Enabling prompt caching

Using caching in requests

​How prompt caching works

​Enabling prompt caching

​Using caching in requests

How prompt caching works

Enabling prompt caching

Using caching in requests