Decoding (temperature, top-p)

Decoding is the way of choosing the next token from the model's probability distribution. Temperature makes the choice more deterministic (low) or more varied (high). Top-p restricts sampling to the most probable tokens whose sum reaches p; top-k to the first k.

Example

temperature=0.2, top_p=0.9

Related concepts

The whole glossary