Cache AI Responses in Python to Cut Costs

Run a script over a spreadsheet of 500 product names, hit an error on row 470, fix it, and run the whole thing again — and you have just paid twice for 469 identical answers. A response cache stops that. In roughly twenty minutes you will have a cached_completion() function that hashes each request, checks a local SQLite file first, calls the API only on a miss, and forgets entries once they get old. Every rerun after the first is close to free.

This guide sits inside Managing AI API Costs and Tokens, alongside the guides on measuring spend before and after a call. Caching is the one technique on that list that removes cost rather than predicting it.

Prerequisites

You need Python 3.10 or newer and an OpenAI API key. Nothing here needs a database server: sqlite3 and hashlib are part of the Python standard library, so the only third-party packages are the SDK itself and a loader for your key. If you have not made a project folder with an isolated environment yet, Create a Python Virtual Environment for AI covers it in a few minutes.

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install "openai>=1.40" "python-dotenv>=1.0"
pip freeze > requirements.txt

Put your key in a .env file in the project folder:

OPENAI_API_KEY=sk-your-key-here

Then keep it out of version control before you write another line:

echo ".env" >> .gitignore
echo "cache.db" >> .gitignore

Add cache.db to that same file while you are there. The cache holds full model replies, which may include customer data or draft copy you would rather not publish to a repository by accident.

Step 1 — Build a cache key from the model and the messages

A cache key is a short string that stands in for one exact request. Two requests that would produce the same kind of answer must produce the same key; two requests that differ in any meaningful way must produce different keys. You cannot use the prompt text alone, because the same prompt sent to gpt-4o-mini and to gpt-4o gives different answers and should be billed and stored separately.

The reliable recipe is: collect everything the API call depends on into one dictionary, turn it into text with sorted keys so the ordering never wobbles, then run that text through SHA-256. SHA-256 is a hashing function — it takes any length of input and returns a fixed 64-character fingerprint, and the same input always yields the same fingerprint. That fingerprint becomes your key.

import hashlib
import json


def cache_key(model: str, messages: list[dict], **params) -> str:
    """Return a stable 64-character fingerprint for one API request."""
    payload = {
        "model": model,
        "messages": messages,
        "params": {k: params[k] for k in sorted(params)},
    }
    blob = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()


if __name__ == "__main__":
    msgs = [
        {"role": "system", "content": "You write short product blurbs."},
        {"role": "user", "content": "Blurb for a stainless steel water bottle."},
    ]
    print(cache_key("gpt-4o-mini", msgs, temperature=0))
    print(cache_key("gpt-4o", msgs, temperature=0))

Run it and you get two different fingerprints from the same messages, because the model differs. Change a single character of the system prompt and the fingerprint changes completely — that is the property you want, since a changed system prompt means a changed answer. If you are still deciding what belongs in that system message, Write System Prompts that Control Output Format is worth reading first, because every edit you make there invalidates the cache.

sort_keys=True matters more than it looks. Python dictionaries preserve insertion order, so {"a": 1, "b": 2} and {"b": 2, "a": 1} would serialise to different text and hash to different keys despite describing the same request. Sorting removes that trap.

Step 2 — Store responses in a SQLite table

SQLite is a database that lives in one ordinary file. Python talks to it through the built-in sqlite3 module, so there is nothing to install, nothing to run in the background, and nothing to configure. For a script on your laptop it is the right amount of machinery.

Store six things per row: the key, the model, the reply text, the input token count, the output token count, and the moment you saved it. The token columns are what let you prove the cache is working later, and the timestamp is what makes expiry possible in step 4.

import sqlite3
from pathlib import Path

DB_PATH = Path("cache.db")


def get_db(path: Path = DB_PATH) -> sqlite3.Connection:
    """Open the cache file, creating the table on first use."""
    conn = sqlite3.connect(path)
    conn.execute(
        """
        CREATE TABLE IF NOT EXISTS responses (
            key               TEXT PRIMARY KEY,
            model             TEXT NOT NULL,
            reply             TEXT NOT NULL,
            prompt_tokens     INTEGER NOT NULL,
            completion_tokens INTEGER NOT NULL,
            created_at        TEXT NOT NULL
        )
        """
    )
    conn.commit()
    return conn


if __name__ == "__main__":
    with get_db() as conn:
        rows = conn.execute("SELECT COUNT(*) FROM responses").fetchone()
        print("cached rows:", rows[0])

CREATE TABLE IF NOT EXISTS means the function is safe to call on every run: it builds the table the first time and does nothing afterwards. Declaring key as the PRIMARY KEY gives you a fast lookup and blocks duplicate rows for the same request.

Timestamps go in as ISO 8601 text — 2026-07-24T09:15:00+00:00 — rather than a number. SQLite compares those strings alphabetically, and for ISO timestamps alphabetical order is chronological order, so created_at > ? works as a date comparison without any parsing.

Step 3 — Wrap the call in a cached_completion() function

Now join the two halves. cached_completion() takes the same arguments you would pass to the SDK, computes the key, looks it up, and only reaches for the network when the lookup comes back empty. Your calling code never has to know which of those two paths ran.

The function returns a tuple: the reply text, and a boolean saying whether it was a hit. Returning that flag costs nothing and makes the savings visible while you are testing.

import os
import sqlite3
from datetime import datetime, timedelta, timezone
from pathlib import Path

from dotenv import load_dotenv
from openai import OpenAI

from cache_key import cache_key      # step 1
from cache_db import get_db          # step 2

load_dotenv()  # reads .env, which is listed in .gitignore
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

MAX_AGE = timedelta(days=7)


def cached_completion(
    conn: sqlite3.Connection,
    model: str,
    messages: list[dict],
    **params,
) -> tuple[str, bool]:
    """Return (reply_text, was_a_hit). Calls the API only on a miss."""
    key = cache_key(model, messages, **params)
    cutoff = (datetime.now(timezone.utc) - MAX_AGE).isoformat()

    row = conn.execute(
        "SELECT reply FROM responses WHERE key = ? AND created_at > ?",
        (key, cutoff),
    ).fetchone()
    if row is not None:
        return row[0], True

    response = client.chat.completions.create(
        model=model, messages=messages, **params
    )
    reply = response.choices[0].message.content
    usage = response.usage

    conn.execute(
        "INSERT OR REPLACE INTO responses "
        "(key, model, reply, prompt_tokens, completion_tokens, created_at) "
        "VALUES (?, ?, ?, ?, ?, ?)",
        (
            key,
            model,
            reply,
            usage.prompt_tokens,
            usage.completion_tokens,
            datetime.now(timezone.utc).isoformat(),
        ),
    )
    conn.commit()
    return reply, False


if __name__ == "__main__":
    question = [{"role": "user", "content": "Name three uses for a stapler."}]
    with get_db(Path("cache.db")) as conn:
        for attempt in (1, 2):
            text, hit = cached_completion(
                conn, "gpt-4o-mini", question, temperature=0
            )
            print(f"attempt {attempt}: hit={hit} — {text[:60]}...")

Run this file twice in a row. The first execution prints hit=False and takes a second or two; the second prints hit=True and returns instantly, having sent nothing. That instant response is the second benefit people underrate — a cached script is not just cheaper, it is fast enough to iterate on.

Two details are load-bearing. The ? placeholders in the SQL keep your values separate from the query text, which is the standard way to avoid SQL injection and also handles quotes in reply text for free. And the created_at > ? condition in the SELECT means an expired row is treated as a miss even before anything deletes it, so stale text can never be served.

Step 4 — Expire entries so stale answers do not live forever

A cache with no expiry slowly becomes a museum. Model versions shift behind the same name, your prompts improve, and a reply written months ago stops representing what the model would say today. The window you allow is called the TTL, short for time to live.

Step 3 already enforces the TTL on read. Add a delete pass so the file does not grow forever, and run it whenever you start a batch job:

import sqlite3
from datetime import datetime, timedelta, timezone


def purge_expired(conn: sqlite3.Connection, max_age: timedelta) -> int:
    """Delete rows older than max_age. Returns how many were removed."""
    cutoff = (datetime.now(timezone.utc) - max_age).isoformat()
    cursor = conn.execute("DELETE FROM responses WHERE created_at <= ?", (cutoff,))
    conn.commit()
    return cursor.rowcount


def cache_stats(conn: sqlite3.Connection) -> dict:
    """Count rows and the tokens they saved you on every repeat hit."""
    row = conn.execute(
        "SELECT COUNT(*), COALESCE(SUM(prompt_tokens + completion_tokens), 0) "
        "FROM responses"
    ).fetchone()
    return {"rows": row[0], "tokens_per_full_replay": row[1]}


if __name__ == "__main__":
    from cache_db import get_db

    with get_db() as conn:
        print("removed:", purge_expired(conn, timedelta(days=7)))
        print("stats:", cache_stats(conn))
        conn.execute("VACUUM")  # reclaim the disk space the deletes freed

Pick the TTL from the content, not from habit. Product descriptions and classification labels can sit for a month without harm. A summary of a document that people keep editing deserves a day. Anything that mentions "current" or "latest" belongs at an hour or should not be cached at all.

cache_stats() gives you a number to report: the token total across every stored row is what a full rerun would cost you if the cache were not there. Feed that into the pricing arithmetic in Estimate OpenAI API Costs with Python and you have a real figure for what the cache saved this week.

When caching is the wrong tool

Caching is not free of consequences. It trades freshness and variety for cost, and there are jobs where those are exactly the things you cannot give up.

The clearest case is creative generation at a temperature above zero. Temperature controls how much randomness the model uses when picking each word, and asking for five alternative taglines works only because each call rolls the dice differently. A cache defeats that on purpose: the same prompt returns the same stored tagline forever. If you are unsure what setting you are actually using, What Is Temperature in an LLM API? explains the dial. A good rule is to cache only calls where you have pinned temperature=0, since those are the ones meant to be repeatable anyway.

The second case is anything tied to the present moment. Summarising today's support tickets, drafting a note about a live inventory count, answering a question about a document someone is still editing — a cached answer is a wrong answer as soon as the source moves. Either skip the cache or set a TTL short enough that no reader would notice.

There is a third, quieter failure mode. If your prompt embeds a customer's name or account details, the cache key includes them, so entries are naturally per-customer and will almost never be reused. That is correct behaviour but pointless storage. Restructure such prompts so the shared instruction is cacheable and the personal part is added after the model replies, or accept that this particular call is not a caching candidate.

Provider-side prompt caching is a different lever

Providers offer their own caching, usually called prompt caching, and it is easy to assume it makes your work here redundant. It does not, because the two operate at different layers.

Provider prompt caching applies to the repeated opening section of a long prompt — a large system instruction, a style guide, a document you keep asking questions about. The provider stores its internal representation of that prefix and charges a reduced rate when the next request starts with the same text. You still send a request, the model still generates a fresh answer, and you still get a bill. It shines when the prefix is long and the question at the end keeps changing.

Your local response cache works on complete request-and-answer pairs. When it hits, no request is sent, so the charge is exactly zero and the answer arrives in milliseconds. It shines when whole requests repeat.

Run them together. Your cached_completion() wrapper sits outermost and answers everything it has seen before; whatever gets through still benefits from the provider's discount on its shared prefix. Check the provider's documentation for the exact conditions, since minimum prefix lengths and retention windows are set by them and change over time.

Cache key ingredients

Use this as the checklist when you decide what to hash. Anything that changes the answer belongs in the key; anything that varies for unrelated reasons must stay out, or you will never get a hit.

If you also want a per-user cache so one customer's answers are never served to another, add the user identifier as one more entry in the hashed dictionary rather than mixing it into the prompt text. That keeps the key honest and the prompt clean.

Troubleshooting

• sqlite3.OperationalError: no such table: responses — you called sqlite3.connect() directly somewhere instead of get_db(), so the CREATE TABLE step never ran. Route every connection through get_db().
• sqlite3.InterfaceError: Error binding parameter 3 - probably unsupported type — you passed the SDK's usage object or a whole response object into the INSERT instead of plain integers. Pull out usage.prompt_tokens and usage.completion_tokens first.
• TypeError: Object of type set is not JSON serializable — one of the keyword arguments you forwarded into cache_key() is not JSON-friendly. Convert it with str() before hashing, or drop it from the key if it does not change the answer.
• sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread — you shared one connection across threads. Open a connection inside each worker, or pass check_same_thread=False and guard writes with a lock. If the traceback is hard to read, Read a Python Traceback in Five Minutes shows how to find the line that matters.

When to use this vs. alternatives

• Use a SQLite response cache for scripts and batch jobs on one machine, where the same prompts recur across reruns. It costs nothing to install and survives restarts, which an in-memory dictionary does not.
• Use an in-memory dictionary when repeats only happen within a single run and you do not care about keeping them afterwards. functools.lru_cache does this in one line, but it forgets everything when the process ends.
• Use Redis or another shared store once several servers or workers must see the same cache. The key-building and expiry logic here carries over unchanged; only the storage call swaps out.
• Use rate limiting instead when your problem is bursts of distinct requests rather than repeats — Fix the 429 Rate-Limit Error in Python and Rate-Limit AI API Calls in a SaaS with Python cover that pattern.

Caching pairs naturally with the other cost habits in this series. Measure a prompt with Count Tokens in Python Before You Send so you know what a miss costs, keep a ceiling in place with Set a Monthly AI API Spending Limit, and record hits and misses alongside your other call data using Log and Monitor AI API Calls in Production. Together they turn API spend from a surprise at the end of the month into a number you steer.

Back to Managing AI API Costs and Tokens.

Related guides

• Managing AI API Costs and Tokens — the main guide for this section, covering tokens, budgets and limits end to end.
• Estimate OpenAI API Costs with Python — turn the token counts your cache stores into a currency figure.
• Count Tokens in Python Before You Send — measure a prompt before it becomes a cache miss.
• Bulk-Rewrite Product Descriptions with Python — the classic repeat-heavy batch job a cache pays for itself on.
• What Is Temperature in an LLM API? — decide whether a call is repeatable enough to cache at all.