Prompt Caching

Prompt caching can reduce repeated input processing for supported models and providers when requests share a reusable prompt prefix.

Caching availability varies by model and upstream provider. Cache hits are not guaranteed, and any latency or cost improvements depend on the selected model's current pricing. Applications should never depend on cache state for functional correctness.

How it works

Some upstream providers may automatically cache eligible prompt prefixes. When you send a request with a repeated prefix, the provider checks for reusable cached input.

Implementation

You do not need to send special parameters to enable caching on supported models. Structure your prompts so that identical, reusable content appears at the very beginning of the request.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DEVUP_API_KEY"],
    base_url="https://api.devupai.com/v1",
)

# Long system prompt that stays the same across requests
SYSTEM_PROMPT = """You are a helpful AI assistant with deep expertise in Python.\n[... thousands of tokens of instructions or context ...]\n"""

# First request - full processing
response1 = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Pro",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "How do I use list comprehensions?"},
    ],
)

# Second request - matching prefix may be reused
response2 = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Pro",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "What are Python generators?"},
    ],
)

Best practices

Common use cases

When constructing your prompt architecture, these elements are typically the best candidates for the static prefix:

Checking cache usage

Cache usage metadata may be available when returned by the selected provider. When supported, the response usage object indicates how many tokens were served from cache:

{
  "usage": {
    "prompt_tokens": 5000,
    "completion_tokens": 50,
    "total_tokens": 5050,
    "prompt_tokens_details": {
      "cached_tokens": 4800
    }
  }
}