From sound to tokens
One second of speech has tens of thousands of numbers. A voice model does not write them one by one.
Predicting the waveform number by number would take tens of thousands of steps per second, and a single number carries almost no meaning. That is why modern voice models work in two layers. A codec compresses the waveform into a short sequence, tens of pieces per second, and can go back. A language model, the same kind of network behind a chatbot, learns to predict the next piece from the text and the previous pieces.
The pieces can be integer codes, like words in a sound vocabulary, stacked from coarse (what is said) to fine (the timbre and the room). Or they can be vectors of real numbers, generated by diffusion. The codec carries no text: it only describes sound.
Read next
Same sentence, only the melody changes
In a question, the voice rises at the end. And the end of the sentence is where generated voices fail most.ReadSame note, three sounds
Three sounds with the same 220 Hz note. What makes each one sound different is timbre.ReadInside the sound
The same speech seen as an image: time across, frequency up, energy as color.Read