More voices per GPU
CUDA graph, MPS and batching: how we doubled real-time voices on the same GPU.
Generating voice in real time for many conversations at once is an engineering problem, not only a model problem. Speech comes out in short steps, and each step launches dozens of kernels on the GPU. With one conversation per process, the GPU spends much of its time waiting for the CPU to send the next kernel.
A CUDA graph records the whole step once and replays it with a single call. MPS lets several processes truly share the GPU. The biggest gain came from batching: every conversation moves in the same step, each in its own KV cache row. The audio is identical to one conversation at a time, with twice the real-time voices on the same GPU.
Read next
From audio to tensor
Every audio clip that enters a model becomes a tensor. Its shape decides memory and speed.ReadFrom sound to tokens
One second of speech has tens of thousands of numbers. A voice model does not write them one by one.ReadSame sentence, only the melody changes
In a question, the voice rises at the end. And the end of the sentence is where generated voices fail most.Read