LLM Temperature and Top-p Explained: How AI Models Choose Their Next Word

Ask an AI model the same question twice and you may get two different answers. Ask it to extract a date from an invoice and you want the same answer every time. Ask it for ten slogan ideas and you want variety. The settings that control this trade-off are temperature and top-p, and they are among the most misunderstood options in any AI API.

We could not inspect a commercial model’s internal probabilities, so we built a small Python simulation of the standard sampling maths, using made-up scores for four candidate words. All the numbers below come from that simulation. Real models choose among tens of thousands of tokens, and providers may implement details differently, but the behaviour is the same in principle.

How a model picks the next word

At each step, a large language model outputs a score, called a logit, for every token in its vocabulary. A function called softmax turns those scores into probabilities that add up to 1. The model then samples one token according to those probabilities, appends it to the text, and repeats. Temperature and top-p change the probabilities before that sampling step.

Our toy example

Imagine the prompt “The capital of France is”. We gave four candidate next words these scores:

logits = {"Paris": 5.0, "Lyon": 3.0, "London": 2.5, "banana": 0.0}

Paris is clearly favoured, Lyon and London are plausible-sounding mistakes, and banana is nonsense. A real model would assign some probability to thousands of other tokens too.

Temperature: sharpening or flattening

Temperature divides every score before softmax is applied. Dividing by a small number exaggerates the gaps between scores; dividing by a large number shrinks them. Here are the probabilities our simulation produced:

At 1.0, the model’s own distribution is used unchanged. At 0.2, Paris gets essentially all the probability. At 2.0, Paris drops to 58%, and even banana rises to almost 5%. Temperature does not add knowledge or creativity in any deep sense. It only changes how willing the model is to pick less likely options, which can mean fresher wording or simply more mistakes.

What sampling looks like

Probabilities are abstract, so we drew 1,000 samples at three temperatures, using a fixed random seed:

T=0.2  {'Paris': 1000, 'Lyon': 0,   'London': 0,   'banana': 0}
T=1.0  {'Paris': 810,  'Lyon': 117, 'London': 70,  'banana': 3}
T=2.0  {'Paris': 607,  'Lyon': 202, 'London': 146, 'banana': 45}

At temperature 2.0, about 4 answers in 10 named the wrong city, and 45 of 1,000 said “banana”. Now remember that a real answer is hundreds of tokens long. A small chance of an odd choice at every step adds up, and one bad token early in a sentence can send the rest of the answer in the wrong direction. This is one reason high temperatures make hallucinations more likely.

Temperature 0 and determinism

At temperature 0, the maths would divide by zero, so APIs treat it as “always pick the most likely token”, called greedy decoding. That makes output far more repeatable, but in practice not perfectly identical every time. Hosted models can vary slightly because of how calculations are batched and parallelised on the provider’s hardware, and models are updated over time. Some APIs offer a seed parameter for better repeatability. Treat temperature 0 as “highly consistent”, not as a guarantee.

Top-p (nucleus sampling)

Top-p takes a different approach. It sorts tokens from most to least likely, keeps the smallest group whose probabilities add up to at least p, discards the rest, and renormalises. Applied to our temperature-1.0 probabilities:

top_p=0.5   {'Paris': 1.0}
top_p=0.9   {'Paris': 0.881, 'Lyon': 0.119}
top_p=0.99  {'Paris': 0.821, 'Lyon': 0.111, 'London': 0.067}

At 0.5, Paris alone already covered more than half the probability, so it was the only option. At 0.9, Lyon joined. Even at 0.99, banana was cut off. That is top-p’s strength: it removes the long tail of unlikely tokens that cause nonsense, while adapting to the situation. When the model is confident, few tokens survive; when many continuations are reasonable, more survive.

Top-k and min-p

• Top-k keeps a fixed number of the most likely tokens, such as 40, regardless of their probabilities. It is simple but does not adapt: 40 may be too many when the model is sure and too few when it is not.
• Min-p keeps tokens whose probability is at least a fraction of the top token’s probability. It is popular in open-source model tools and is claimed to work well at higher temperatures, though the research is newer.

Not every API exposes every option. Check your provider’s documentation for which parameters are available and what their defaults are.

Using temperature and top-p together

Both settings shape the same distribution, so changing both at once makes behaviour hard to predict. Several providers recommend adjusting one and leaving the other at its default. A reasonable approach is to tune temperature first, since it is easier to reason about, and only reach for top-p if you need to cut off rare oddities at a higher temperature.

Suggested settings by task

These are starting points drawn from common practice, not rules. Model families differ, defaults differ by provider, and some newer reasoning models fix or ignore these settings entirely. Test with your own prompts. Clear instructions, covered in our prompt engineering guide, usually matter more than fine-tuning these numbers.

Common myths

• “Higher temperature makes the model smarter or more creative.” It only makes less likely tokens more likely. Sometimes that is welcome variety; often it is just noise.
• “Temperature 0 means the answer is correct.” It means the most likely answer. If the model is confidently wrong, temperature 0 makes it consistently wrong.
• “Temperature controls length.” Length is controlled by the prompt and the maximum tokens setting, not by temperature.
• “Lower is always safer.” Very low settings can cause repetitive or stilted text in long outputs.

Testing AI features

Sampling makes AI features non-deterministic, which changes how you test them:

• Run each test case several times and measure the pass rate, rather than relying on one run.
• Check properties, not exact strings: valid JSON, required fields present, numbers matching the source, no banned content. This is the same idea as property-based testing.
• Fix the temperature in tests to match production, and record it with the results.
• Re-run your evaluation set whenever you change the model, the prompt or the sampling settings.

Setting it in an API call

Most chat APIs accept these settings as fields in the request body. A typical request for an extraction task looks something like this (an illustration, not run for this article, and field names vary by provider):

{
  "model": "your-model-name",
  "temperature": 0.1,
  "max_tokens": 300,
  "messages": [
    {"role": "user", "content": "Extract the invoice date and total as JSON: ..."}
  ]
}

Keep these values in configuration, not scattered through code, and log them with each request alongside the model name. When an answer looks odd weeks later, knowing the exact settings that produced it saves a lot of guessing. If you serve several features from one model, give each feature its own settings: the summariser and the code assistant rarely want the same temperature.

Final thoughts

Temperature and top-p do not change what a model knows; they change how adventurous it is when choosing each next token. Our toy simulation showed temperature sliding Paris from certain to 58% and top-p cutting off the nonsense tail. Start low for tasks with one right answer, raise temperature only when you want variety, adjust one setting at a time, and test with repeated runs.

Frequently asked questions

Keep learning