Prompt caching

Overview

Prompt caching reuses the work a model has already done on the beginning of a prompt. When a request starts with the same tokens as an earlier request, the model loads the saved state for that shared prefix instead of recomputing it, then processes only the tokens that follow.

Prompt caching is automatic. There’s no parameter, header, or account setting to turn it on, and no cache ID or expiry setting to manage. Caching never changes how the model generates its output.

How prompt caching works

To process a prompt, the model computes intermediate key-value (KV) states for every input token, which let it attend to earlier tokens while it generates. Prompt caching stores the KV states for a prompt’s prefix, so a later request with the same prefix skips that computation.

The prefix is the prompt exactly as the model sees it, after the model’s chat template renders your request into tokens. Everything the chat template renders counts toward the prefix:

• The system prompt and every earlier message, including images.
• Tool definitions in tools.
• Request settings the chat template renders into the prompt, such as reasoning_effort.

A request reuses the longest prefix it shares with a cached entry. Tokens up to the first difference can come from the cache, and every token from the first difference onward is processed from scratch. For example, two requests that share a system prompt and reference material but end with different questions read the shared tokens from the cache and process only the new question.

Cache entries are kept on a best-effort basis and may be evicted sooner under heavy load. On models served by several replicas, a repeat request may be routed to a replica that hasn’t seen your prefix, so hit rates are probabilistic and improve with sustained traffic on the same prefix. The minimum cacheable prefix length and the granularity of cached tokens vary by model and can change, so don’t design your prompts around a specific threshold.

Maximize cache hits

Any change that alters the rendered prompt ends the match at the point of the change. Structure requests so the part that changes comes last:

• Put stable content first: place instructions, tool definitions, reference documents, and few-shot examples at the start of the prompt. Put per-request content, such as the user’s question, timestamps, or retrieved snippets, at the end. A timestamp or username in the system prompt makes every request’s prefix unique from that point on.
• Append instead of rewriting: in a multi-turn conversation, add new messages to the end of the history and keep earlier turns byte-identical, so the reusable prefix grows with each turn. Summarizing, truncating, or reordering the history changes the prefix from the first edited message onward.
• Keep tools stable: send the same tool definitions, with the same names, descriptions, schemas, and order, on every request. Tool definitions are part of the prefix, so changing any of them ends the match where the chat template renders them.
• Set reasoning effort once per conversation: choose reasoning_effort before the conversation starts. Changing it partway through can change the rendered prompt.
• Serialize content deterministically: if your code builds tool schemas or JSON message content, keep the key order the same between requests. The same data serialized in a different order renders as different tokens.
• Stay on one model: KV states are specific to the model that computed them, so a cache entry only serves requests to the same model.

Where dynamic content sits

A timestamp placed before the system instructions leaves almost nothing reusable:

# Every request gets a unique prefix: nothing after the timestamp can be reused.
messages = [
    {"role": "system", "content": f"Current time: {datetime.now().isoformat()}\n\n{INSTRUCTIONS}"},
    {"role": "user", "content": question},
]

The same timestamp placed after the instructions lets the stable content be read from the cache:

# The stable instructions come first; only the tail changes between requests.
messages = [
    {"role": "system", "content": INSTRUCTIONS},
    {"role": "user", "content": f"Current time: {datetime.now().isoformat()}\n\n{question}"},
]

Growing a conversation

When each turn appends to the history, every request reuses most of the previous request’s prompt and processes only the latest assistant reply and user message:

history = [{"role": "system", "content": INSTRUCTIONS}]

while True:
    history.append({"role": "user", "content": input("> ")})
    response = client.chat.completions.create(model="deepseek4-flash", messages=history)
    reply = response.choices[0].message
    # Append the assistant turn verbatim so the next request shares this whole prefix.
    history.append({"role": "assistant", "content": reply.content})
    print(reply.content)

If an earlier turn is summarized or truncated, the next request reuses only the content before the edit. Everything from the edited turn onward is processed again.

Check cached tokens

The usage object on each response reports how many prompt tokens came from the cache in usage.prompt_tokens_details.cached_tokens. This example sends two requests that share a long system prompt and prints the cached count for each:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.reka.ai/v1",
    api_key=os.environ["REKA_API_KEY"],
)

# A long, stable prefix: instructions plus reference material.
system_prompt = "You are a support assistant for Acme Cloud." + "".join(
    f" Plan tier {i} refunds take {i % 7 + 1} business days." for i in range(300)
)


def ask(question: str) -> None:
    response = client.chat.completions.create(
        model="deepseek4-flash",
        max_tokens=64,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    details = usage.prompt_tokens_details
    cached = details.cached_tokens if details and details.cached_tokens else 0
    print(f"prompt_tokens={usage.prompt_tokens} cached_tokens={cached}")


ask("How long do refunds take for plan tier 12?")
ask("How long do refunds take for plan tier 40?")

On a cache miss, some models omit prompt_tokens_details entirely while others return it with cached_tokens: 0. Treat a missing object as zero, as the examples above do.

Running the example against deepseek4-flash prints output like this:

prompt_tokens=3626 cached_tokens=0
prompt_tokens=3626 cached_tokens=3584

The second request reads most of its prompt tokens from the cache. The rest are the tokens after the shared prefix: the new question and the tokens the chat template adds around it. The exact counts depend on the model’s tokenizer and cache granularity, and so does the hit rate: on models served by several replicas, such as glm5.3 and qwen3.8-27b, two back-to-back requests often land on different replicas and the example prints cached_tokens=0 twice. Sustained traffic on the same prefix warms more replicas. The full usage object from the second request looks like this:

{
  "usage": {
    "prompt_tokens": 3626,
    "completion_tokens": 22,
    "total_tokens": 3648,
    "prompt_tokens_details": { "cached_tokens": 3584 }
  }
}

Some models include additional informational fields in prompt_tokens_details, such as created_cache_tokens and, on vision-capable models, multimodal_tokens. They are not billed and may be absent; only cached_tokens affects price.

To track your cache hit rate, divide the total cached_tokens by the total prompt_tokens across your requests. See the API reference for the full shape of the usage object.

Pricing

Cached tokens are billed at the model’s cached input rate. All other input tokens are billed at the standard input rate. There’s no charge for writing to the cache, and no platform fee or markup on top.

reka-edge-2603 and reka-flash-3 report cached tokens and get the latency benefit, but have no cached input price; every input token bills at the standard rate.

For example, the second request in Check cached tokens bills 3,584 cached tokens at DeepSeek V4 Flash’s cached input rate and the remaining 42 tokens at its standard input rate. Current rates for every model are on the Models page.

Requests that hit the cache still count toward your rate limits.

Next steps

• PricingCache-read rates alongside input and output for each model.
• Tool calling best practicesKeep tool definitions stable so they stay cached.
• ParametersRequest settings that affect cost and latency.