Chapter 5 of 36 · ~1 min
Sampling and Temperature
The model does not always pick its most probable token. It samples: higher-probability tokens are chosen more often, but lower ones get their turn. Temperature controls how adventurous this is. Near zero, the model almost always takes the top choice and becomes repetitive and predictable. Higher, and it explores, which reads as creativity until it reads as nonsense.
This is why the same prompt gives different answers on different runs, and why "the model said X" is a weaker statement than it sounds. It also gives you a knob: low temperature for extraction and classification, higher for brainstorming.
Experiment 1 of 2
Runs in your browserThe maths first, on an illustrative distribution: slide the temperature and sample from it.
The dog chased the ___
- cat35%
- ball25%
- boy18%
- car10%
- bird6%
- postman4%
- wind2%
The starting probabilities are illustrative; the temperature maths and the sampling are real. Low temperature repeats the favourite; high temperature lets the tail through.
Experiment 2 of 2
Live modelThe same opening continued by a live model at a normal temperature and at a very high one. Generate again and watch how far each side wanders.
Runs against a live model through Chatterfly's server. Your text is sent to the model provider and not stored. A completion model continues the text for up to sixty tokens at each temperature.
Big question
Is variability a defect to remove, or a property to design around?