What Is Temperature in an LLM API?

Temperature is the one parameter almost every beginner changes without knowing what it does. By the end of this guide you will know exactly what it changes inside the model, you will have run the same prompt at three settings and compared the results with your own eyes, and you will have a rule for choosing a value from the task in front of you. Budget about twenty minutes and roughly a cent of API credit.

This guide sits inside Understanding LLM APIs, the section that covers how a hosted model call works end to end. It assumes you have already made one successful API call and now want to control the character of what comes back.

Prerequisites

You need Python 3.10 or newer, a virtual environment, and an OpenAI API key. If any of that is missing, work through Create a Python Virtual Environment for AI first, then come back.

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install "openai>=1.40" "python-dotenv>=1.0"

Put your key in a .env file — a plain text file of NAME=value lines that your script reads at startup, so the key never appears in your code:

OPENAI_API_KEY=sk-your-key-here

Then add that file to .gitignore before you commit anything, so the key cannot reach a public repository:

echo ".env" >> .gitignore

Everything below uses gpt-4o-mini, the cheapest current OpenAI text model, because you will be sending the same prompt many times. The ideas transfer to any provider — the equivalents on other vendors are covered in OpenAI vs Anthropic API for Beginners and OpenAI vs Gemini API for Python Beginners.

Step 1 — Understand what the model is actually choosing

A language model does not write a sentence. It writes one token — a short chunk of text, usually a word or word-fragment — then looks at everything so far and writes the next one. To choose each token it scores every candidate in its vocabulary and turns those scores into probabilities: perhaps "coffee" at 41%, "espresso" at 22%, "brew" at 9%, and thousands of unlikely options sharing what is left.

Temperature is applied to those scores just before they become probabilities. Divide the scores by a small number and the gaps between them stretch, so the leader takes nearly all the weight. Divide by a larger number and the gaps compress, so the runners-up become genuinely plausible picks. That is the whole mechanism: temperature does not make the model smarter, more truthful, or more creative in any real sense — it only changes how sharply the odds favour the front-runner.

The loop in that diagram runs once per token. A 200-token reply means 200 separate draws, which is why a small change in the odds compounds into a noticeably different paragraph.

Step 2 — Run one prompt at three temperatures

Reading about the effect is much less convincing than seeing it. This script sends the same prompt twice at each of three settings and prints all six replies, so you can judge both the variation between settings and the variation within a setting.

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()  # reads .env, which is listed in .gitignore
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

PROMPT = "Write a one-sentence tagline for a coffee subscription service."

for temperature in (0.0, 0.7, 1.3):
    print(f"\n=== temperature = {temperature} ===")
    for run in (1, 2):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            temperature=temperature,
            max_tokens=60,
            messages=[{"role": "user", "content": PROMPT}],
        )
        print(f"  run {run}: {response.choices[0].message.content.strip()}")

Save it as temperature_demo.py and run python temperature_demo.py. Three patterns show up almost every time:

• At 0.0 the two runs are usually identical or near-identical, and the tagline is the safe, obvious phrasing — the sort of line you would guess the model would write.
• At 0.7 the two runs differ in wording and rhythm but both stay sensible and on-brief. This is the range most writing tasks live in.
• At 1.3 the runs diverge sharply. You will often get one line you like more than anything at 0.7, alongside one that is clumsy, over-long, or slightly off-topic.

That last point is the honest trade. High temperature does not reliably produce better copy; it produces a wider spread, so you get better best results and worse worst results. If you are generating options for a human to choose between, that is a good deal. If the output goes straight into a database, it is a terrible one.

Swap the prompt for something factual — "List the three primary colours as a JSON array" — and rerun. At 0.0 you get the same clean array twice. At 1.3 you will eventually see extra commentary, a different key name, or a fourth colour, which is precisely the kind of drift that causes the errors covered in Fix JSONDecodeError with AI API Responses in Python.

Step 3 — Choose the value from the task, not from taste

There is no universally good temperature. There is only a good temperature for the job, and the deciding question is simple: does anything downstream depend on the exact wording? If a parser, a rule, or a database column consumes the output, you want repeatability and you should stay near 0. If a human reads it and picks a favourite, you want spread.

Rather than sprinkle magic numbers through your code, name them once. A lookup keyed by task type makes the intent readable and gives you a single place to adjust later:

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

TEMPERATURE_BY_TASK = {
    "extract": 0.0,      # pull fields out of text
    "classify": 0.0,     # choose one label from a fixed set
    "summarise": 0.3,    # faithful, slightly fluent
    "answer": 0.4,       # support replies from known facts
    "rewrite": 0.7,      # same meaning, new voice
    "brainstorm": 1.0,   # many different options
}


def ask(prompt: str, task: str = "answer") -> str:
    """Send one prompt using the temperature that suits the task."""
    temperature = TEMPERATURE_BY_TASK.get(task, 0.4)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=temperature,
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content.strip()


print(ask("Label this review as positive, neutral or negative: 'Arrived late but tastes great.'",
          task="classify"))
print(ask("Give me five names for a decaf coffee blend.", task="brainstorm"))

If a task at 0 still gives you inconsistent shapes, the fix is almost never a temperature tweak — it is a firmer instruction. Write System Prompts that Control Output Format and Few-Shot Prompting in Python with Examples both do far more for consistency than any sampling setting.

Step 4 — Leave top_p alone

Sitting next to temperature in every SDK is top_p, sometimes called nucleus sampling. It works on the same probability list but cuts rather than rescales: sort the candidates by probability, add them up from the top, and stop once the running total reaches top_p. Everything below that line is discarded, and the model samples only from the survivors. At top_p=0.9 the model ignores the least likely tail; at the default of 1.0 nothing is discarded at all.

Why only one? Because they interact in ways that are hard to predict. Lower the temperature and the tail shrinks on its own, which changes what top_p would have trimmed; tighten top_p and the temperature you set is now acting on a shorter list. When output goes wrong you want one variable to move, so you can tell what caused the change. Set temperature explicitly on every call, omit top_p entirely, and you will always know why a reply looks the way it does.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.2,      # the one dial you move
    # top_p intentionally not set: it stays at its 1.0 default
    max_tokens=150,
    messages=[{"role": "user", "content": "Summarise this in two lines: ..."}],
)
print(response.choices[0].message.content)

Step 5 — Make a run as repeatable as you can

For testing, cost tracking, and debugging you often want the same input to produce the same output. Three things get you most of the way there: a fixed prompt string with no timestamps or random ids baked into it, a temperature of 0, and — on the OpenAI chat API — a seed, an integer that asks the provider to start its random draws from the same place.

import os
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

for attempt in (1, 2):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        seed=42,                       # best-effort repeatability
        max_tokens=80,
        messages=[
            {"role": "system", "content": "Reply with one short sentence."},
            {"role": "user", "content": "What is a token in an LLM API?"},
        ],
    )
    print(attempt, response.system_fingerprint, response.choices[0].message.content.strip())

The system_fingerprint value in the response identifies the backend configuration that served your request. If two runs share a fingerprint and still differ, something in your input changed. If the fingerprint itself changed between runs, the provider moved you to a different build, and identical output is no longer expected.

That is the caveat worth internalising: identical output is never fully guaranteed. These models run on hardware where tiny differences in arithmetic ordering can flip a near-tie between two candidates, and providers update their serving stack without warning. So write tests that check properties — "the reply is valid JSON", "the label is one of three allowed values", "the summary is under 60 words" — never tests that compare against a stored string. If you truly need byte-identical results, store the first response yourself; Cache AI Responses in Python to Cut Costs shows the pattern, and it saves money as a bonus.

One more thing to put out of your mind: temperature has no effect on your bill or on latency. You are charged per token in and per token out at the same rate whatever the setting, and the request takes the same path through the same model. The only indirect link is length — an exploratory reply may ramble on for more output tokens than a terse one, so measure with Count Tokens in Python Before You Send rather than guessing.

Quick reference: temperature by task

Troubleshooting

Unsupported value: 'temperature' does not support 0 with this model. Only the default (1) value is supported. Some newer models fix their own sampling and reject the parameter. The fix is to remove temperature= from that call entirely and control the output through the prompt instead.

Invalid value for 'temperature' with a 400 status. You passed a number outside the accepted range, usually a value above 2 or a negative one, or a string like "0.7" instead of the number 0.7. Pass a float within the documented range.

Output at temperature 0 repeats a phrase or loops. Very low temperature makes the model commit hard to its favourite continuation, which can trap it in a cycle. Nudge to 0.2, or fix the real cause — a vague prompt with nothing new to say. Adding a concrete example usually beats moving the dial.

High temperature breaks your parser intermittently. A script that works nine times and fails the tenth is a sampling problem, not a code problem. Drop the temperature to 0 for the parsed call and keep the creative call separate; if failures continue, the retry-and-validate approach in Fix JSONDecodeError with AI API Responses in Python will catch the rest.

Repeated experiments trigger RateLimitError. Looping over several temperatures with several runs each fires a burst of requests. Add a short pause between calls or follow Fix the 429 Rate-Limit Error in Python to back off properly.

When to use this vs. alternatives

• Change temperature when the problem is variety — output that is too samey, or too scattered. It is a blunt, global dial with one knob and no vocabulary of its own.
• Change the prompt when the problem is shape or content — wrong format, missing fields, wrong tone, invented facts. No temperature value can supply an instruction you did not write, and a good system prompt fixes the issue at 0 and at 1.
• Change the model when the problem is capability — reasoning that is out of reach for a small model will not appear at a different temperature. Move from gpt-4o-mini to gpt-4o for hard tasks, or compare vendors before committing.
• Generate several candidates when you want the best of many — running the same prompt three times at 0.9 and picking a winner beats hunting for one magic setting, and it is exactly how batch copy tasks such as Classify Incoming Documents with Python and AI separate the deterministic step from the creative one.

Temperature is worth ten minutes of your attention and no more. Set it deliberately per task, write it in your code rather than relying on a default you cannot see, leave top_p where it is, and spend the time you save on the prompt — that is where the real leverage sits. Back to Understanding LLM APIs.

Related guides

• Understanding LLM APIs — the main guide for this section, covering the whole request and response cycle.
• Write System Prompts that Control Output Format — the setting that fixes consistency problems temperature cannot.
• Fix JSONDecodeError with AI API Responses in Python — recover when a reply is not the JSON you expected.
• OpenAI vs Gemini API for Python Beginners — see how sampling settings are named on another provider.
• Count Tokens in Python Before You Send — measure what a longer reply actually costs you.