A new voice without touching every weight
Low-rank adapters: train under 1% of the numbers and keep the same speed.
A linear layer stores a d × d matrix W. In full training, every number changes. With a low-rank adapter, W stays frozen and only two thin matrices are trained, B (d × r) and A (r × d). The update is the product B·A, added to W with weight α/r. With d = 4096 and r = 16, that is 131 thousand trainable numbers instead of 16.7 million: under 1%.
Training gets cheap and fast, and each voice becomes a small file. Later, B·A can be merged into W, so inference is as fast as the original model.
Read next
Attention and the next token
A language model writes one token at a time, looking only backwards.ReadMore voices per GPU
CUDA graph, MPS and batching: how we doubled real-time voices on the same GPU.ReadFrom audio to tensor
Every audio clip that enters a model becomes a tensor. Its shape decides memory and speed.Read