Two different dials that both get described as controlling "randomness" — but they act on the token distribution in completely different ways, at completely different stages.
Before either setting ever gets involved, the model has already done its job: for the current context, it has computed a probability for every single token in its vocabulary — tens of thousands of candidates, each assigned a likelihood of being the next token. Temperature and top-p don't change that underlying computation at all. They're two independent, optional adjustments applied to that already-finished distribution, right before a token gets randomly sampled from it. People routinely treat them as two names for the same "randomness knob." They're not — one reshapes, the other filters — and mixing them up leads to genuinely wrong intuitions about what a given setting will do.
Every generation step, a language model outputs a raw score — a logit — for every token in its vocabulary. Those logits get converted into a proper probability distribution by a softmax function, so by the time sampling is even a question, the model has already ranked and weighted its entire vocabulary for this exact context. Sampling directly from that raw distribution is already a valid strategy. Temperature and top-p exist because raw sampling has failure modes at both ends — sometimes too conservative, sometimes willing to pick something absurd — and the two settings fix different halves of that problem.
Temperature is a single scalar that divides every logit before softmax is applied — softmax(logits / T). That single division reshapes the entire distribution at once. A low value like 0.2 sharpens it — already-likely tokens get pushed even further ahead, and unlikely tokens get pushed even further behind. A high value like 1.5 flattens it — the gap between likely and unlikely tokens shrinks, so tokens that were barely in contention become meaningfully more competitive. Crucially, temperature touches every token, all the way out to the least-likely tail — it never removes anything from consideration, it only rescales how competitive each option is relative to the others.
Top-p (nucleus sampling) does something structurally different: it doesn't reweight anything. It sorts tokens by probability, walks down the list adding up probability mass, and keeps only the smallest set of tokens whose cumulativeprobability reaches the threshold p — top-p = 0.9 keeps just enough of the most likely tokens to account for 90% of the total probability. Everything past that point is dropped from consideration entirely, no matter how low temperature is set. The model then samples only from that surviving pool, using each token's original relative probability within it. Because the cutoff is defined by cumulative mass rather than a fixed count, the pool's size changes token-to-token — a handful of candidates when the model is very confident, dozens when the distribution is flatter — unlike a fixed top-k, which always keeps the same number of tokens whether or not that number makes sense for the current step.
Turn temperature up alone, and the full vocabulary — including the genuinely bizarre, near-zero-probability tail — remains eligible the whole time, just reweighted. Almost always the model still lands on something reasonable, but because nothing is ever fully excluded, a high enough temperature can occasionally still let a truly strange token get sampled purely by chance. Set top-p alone, and that failure mode disappears — the implausible tail is cut outright, every time, regardless of how the surviving candidates are weighted relative to each other. But top-p alone says nothing about howsharply to favor the top of that surviving pool over the rest of it. That's exactly why the two are so often combined in practice: top-p removes the tail you never want considered at all, and temperature then tunes how confident or exploratory the sampling is among whatever legitimately survives that cut.
False. Both settings do make output feel more or less "random," but they get there through completely different operations, at different stages of sampling. Temperature is a multiplicative reweighting of the entire probability distribution — dividing every logit by a scalar before softmax — and it never fully excludes any token, however unlikely, it only rescales how competitive each one is. Top-p is a hard cumulative-probability cutoff — it drops the least-likely tokens from consideration entirely once the cumulative mass of the more-likely ones reaches the threshold, and it does not reweight whichever tokens survive that cut. One reshapes the whole distribution; the other decides which part of it is even in play.That's precisely why the two are frequently used together — each addresses a failure mode the other one doesn't touch.
Explains the two most common LLM sampling settings and why they are not interchangeable: temperature (a scalar that divides the logits before softmax, reshaping the entire probability distribution including the unlikely tail) and top-p / nucleus sampling (a cumulative-probability cutoff that filters the candidate pool down to a dynamically-sized set before sampling ever happens). Includes a side-by-side bar chart of a single next-token decision reshaped by low vs. high temperature, and a second chart showing the same distribution cut off by a top-p = 0.9 threshold.
Both settings are marketed and discussed as controls for "randomness" or "creativity," and both do move output in that general direction, which makes it easy to assume they're redundant. They aren't. Temperature is a reshaping operation: it divides every token's logit by a scalar before the softmax step, which changes how sharply the whole distribution is weighted but never removes any token from consideration, however small its probability becomes. Top-p is a filtering operation: it ranks tokens by probability, accumulates their probability mass in order, and keeps only the smallest set that reaches the chosen threshold — everything past that point is excluded outright, before temperature or sampling gets a say in the matter for those excluded tokens.
Temperature: given raw logits z and temperature T, the sampling distribution becomes softmax(z / T). As T drops below 1, dividing by a small number amplifies the differences between logits, so softmax concentrates probability on the highest-scoring tokens — a sharper, more deterministic distribution. As T rises above 1, dividing by a larger number compresses those differences, flattening the distribution so lower-ranked tokens become relatively more competitive. This operation touches the entire vocabulary at once, including the far tail.
Top-p (nucleus sampling): sort all tokens by probability, descending. Walk down the list summing probabilities until the running cumulative total reaches p (e.g. 0.9). Keep exactly that set of tokens — the "nucleus" — and discard everything after it. Sampling then proceeds only within that surviving set, using each token's original relative probability. Because the cutoff is defined by cumulative probability rather than a fixed count, the size of the nucleus varies from step to step: a very confident prediction might need only 2-3 tokens to reach 90% cumulative mass, while a genuinely uncertain one might need several dozen — a dynamic behavior that a fixed top-k cutoff does not have.
Temperature alone leaves the entire vocabulary technically reachable — a high enough temperature makes the tail more competitive but never removes it, so on rare occasions a genuinely implausible token can still get sampled purely by chance. Top-p alone solves that specific problem by hard-excluding the improbable tail, but it says nothing about how sharply to weight the tokens that do survive the cut. Using both together lets an engineer set top-p to reliably exclude the tokens that should never be considered, and use temperature to separately tune how confident or exploratory sampling is among whatever legitimately remains — two independent dials addressing two independent parts of the same decision.
At temperature 0, sampling becomes effectively deterministic (always picking the single highest-probability token, i.e. greedy decoding), so top-p has little practical effect at that specific setting. At any nonzero temperature, though, top-p still matters — it continues to exclude the unlikely tail regardless of how temperature has reshaped the surviving candidates.
Not directly — top-p is defined by cumulative probability, not vocabulary size or token count. What varies is how many tokens end up in the eligible pool for a given step: a confident prediction might need very few tokens to reach the threshold, while a highly uncertain one might need many, independent of how large the overall vocabulary is.
No. Top-k keeps a fixed number of the highest-probability tokens every single step, regardless of how the probability mass is actually distributed among them. Top-p keeps a variable number of tokens, defined by cumulative probability reaching a threshold — so its effective pool size adapts to how confident or uncertain the model is at that specific step.
No. Temperature only rescales relative probabilities by dividing logits before softmax; softmax always assigns every token a nonzero probability, no matter how small. A token can become vanishingly unlikely under a low temperature, but it is never mathematically excluded the way top-p excludes tokens outright.
Common defaults across LLM APIs are roughly temperature around 0.7-1.0 and top-p around 0.9-1.0, though the right values are task-dependent — lower temperature and lower top-p for tasks wanting consistent, precise output (code, extraction, math), higher values for tasks wanting varied or creative output (brainstorming, creative writing).
Not necessarily on every single request — if a token would have been sampled regardless — but structurally the two settings interact whenever the temperature-reshaped tail is large enough to fall near or past the top-p cutoff. In general, combining them gives more predictable control than either setting alone, because each is addressing a distinct failure mode.
Try our AI Engineering Studio
More calculators, simulators, and guides for this discipline.