From audio to tensor
Every audio clip that enters a model becomes a tensor. Its shape decides memory and speed.
In PyTorch, audio is a tensor with shape [batch, channels, samples]. Ten seconds at 48 kHz in mono is 480 thousand numbers. In float32 each number takes 4 bytes; in bfloat16, 2. The dtype sets precision and memory, and the device says where the tensor lives: on the CPU or on the GPU.
Moving data between the two costs time. That is why audio is loaded, trimmed and normalized with the GPU batch in mind. Change the duration, sample rate, dtype and batch and watch the memory grow.
Read next
From sound to tokens
One second of speech has tens of thousands of numbers. A voice model does not write them one by one.ReadSame sentence, only the melody changes
In a question, the voice rises at the end. And the end of the sentence is where generated voices fail most.ReadSame note, three sounds
Three sounds with the same 220 Hz note. What makes each one sound different is timbre.Read