Research

Acoustics · 2 min read

Inside the sound

The same speech seen as an image: time across, frequency up, energy as color.

A spectrogram cuts the waveform into short slices of about 25 ms, measures how much of each frequency is in each slice (with the FFT) and places the columns side by side. The parallel lines in the low range are the voice harmonics. The bright bands that rise and fall are formants, mouth resonances that make an "a" sound different from an "i". The patches up high, with no lines, are the air of "s" and "sh". A dark column is a pause.

Change the sample rate and see what disappears. At 16 kHz, the usual rate for speech recognition, the brightness above 8 kHz is gone. At 8 kHz, phone quality, only what you need to understand the words remains. The mel scale stretches the low range, the way the ear hears it.

Next postSound as numbers

Read next

  1. Acoustics · 2 min read

    Sound as numbers

    Before any model, a voice is a list of numbers. Take a close look.Read
  2. PyTorch · 2 min read

    A new voice without touching every weight

    Low-rank adapters: train under 1% of the numbers and keep the same speed.Read
  3. PyTorch · 2 min read

    Attention and the next token

    A language model writes one token at a time, looking only backwards.Read