Attention and the next token

In attention, each token computes how much to look at every earlier token. The causal mask makes sure no token looks at the future: during generation, the future does not exist yet. To avoid recomputing at every step, the model keeps the keys and values it already computed in the KV cache.

Each step ends with a probability list for the next token. Temperature flattens or sharpens that list; top-k cuts the unlikely options. Low temperature repeats the most likely choice, high temperature takes more risks. In a voice model the tokens are pieces of sound, and this draw becomes variation in rhythm and intonation.