How AI vocal removers work, and where they fail
Updated · 6 min read
A finished song is one waveform. The voice, the drums and the guitars were added together into it, and nothing in the file says which part of the sound came from which. An AI vocal remover has to work that out from the sound alone, the way you can pick out a singer in a crowded room. Here's how the model we run, HTDemucs, does it, why every separation leaves some traces, what the scores people quote actually measure, and what runs on your device versus on a server.
What a stem is
A stem is one part of a mix on its own: the vocal, the drums, the bass. In a studio, every stem exists as its own recording before the mixdown. A separation model rebuilds them after the fact. HTDemucs gives 4: drums, bass, vocals, and other, which holds everything that isn't one of the first three (guitars, keys, synths, strings). Add the 4 back together and you get the song again, near enough.
Before AI, the karaoke trick was subtraction. Most vocals are mixed dead centre, so subtracting the left channel from the right cancels anything equally loud in both. It also cancels the bass and kick, which usually sit in the centre too, leaves the vocal's stereo reverb behind, and does nothing for a mono recording. A model doesn't subtract channels; it has learned what a voice sounds like.
How the model hears a voice
Picture a spectrogram: time running left to right, pitch from bottom to top, brightness for loudness. HTDemucs reads the song that way, cutting it into slices about 93 ms long, 43 of them a second, each broken into 2,048 frequency bands. A voice has a shape there: a stack of evenly spaced lines (the harmonics of each sung note) that glide with the melody, plus bursts of noise for S and T. A snare is a vertical smear; a bass is a few thick lines at the bottom.
It also reads the raw waveform at the same time, which keeps timing and attack sharp where the spectrogram blurs them. That's the hybrid in Hybrid Transformer Demucs: two branches, one per view, meeting in a Transformer in the middle that compares every moment with every other one in its window, within each view and across the two. Its window is about 7.8 seconds of sound, so it hears a phrase, not just an instant. Here, long songs are fed through in windows that overlap by 10% and crossfade, so there are no seams.
It learned all this from examples: songs where the real stems were known, mixed and handed to it, with its guesses scored against the originals until they got close. The Demucs README says the default model was trained on the MUSDB HQ set plus 800 extra songs. The model never sees your song's stems; it estimates them from what voices and drums looked like in its training.
The HTDemucs paper: Hybrid Transformers for Music Source Separation
Why bleed and ghosts happen
Separation is an estimate, so it's never perfect. The traces you hear come from a few places, and knowing which helps you judge a result:
- 1Shared frequencies. A voice, a guitar and a synth pad all live in the middle of the spectrum. Where they hold the same note at the same time, there's nothing in the sound that says which energy belongs to whom, so the model splits it as best it can.
- 2Cymbals and consonants. Hi-hats and S sounds are both bursts of high noise. Some hat ends up in the vocal, some sibilance in the drums.
- 3Reverb and echo. A voice's reverb is spread out in time and has none of the voice's sharp shape, so part of it is heard as room and lands in other.
- 4Every singer counts as voice. Backing vocals, doubles and choirs go with the lead; the model has one vocal part, not one per singer.
- 5Processed voices. Heavy autotune, a vocoder or distortion make a voice look like a synth, and the model may treat it as one.
- 6A worn source. An MP3 made from a stream has already lost the fine detail the model separates with.
What the score means
Separation papers score a result with SDR, the signal-to-distortion ratio: the energy of the real stem against the energy of everything wrong in the estimate, in decibels. The higher, the cleaner: each extra 10 dB cuts the leftover energy to a tenth. SI-SDR is the same idea made scale-invariant: it doesn't punish a stem that's right but a little quieter or louder overall, so it measures the separation rather than the level.
Two warnings about quoted numbers. A paper's SDR is usually an average over all four parts and a set of test songs; a single song can come out well above or below it. And the score only says how much is wrong, not how it sounds: a thin, watery vocal and a full one with a faint hat behind it can score alike.
| Score | What isn't the part, against the part's energy | What that's like |
|---|---|---|
| 0 dB | 100% | As much error as part: roughly an unsplit song scored as its vocal |
| 3 dB | 50% | The part is there, still buried in the rest |
| 6 dB | 25% | Clearly the part, with the band loud behind it |
| 10 dB | 10% | Where both tools' vocals landed in our test (10.76 dB and 9.79 dB) |
| 20 dB | 1% | Leftovers you'd hardly notice outside the gaps |
Our test: results and limits
On 6 November 2026 we ran one song with published stems through both tools on a single computer, then measured the vocal and the instrumental against the artist's own parts, both ways. Extractune runs HTDemucs; vocalremover.org is the one other free tool we could test without an account. On this song, our vocal came out cleaner on both measures, and our instrumental by a wider margin.
One song is a small test, and a different song could rank them the other way. We've published the method, the song and the scores so anyone can run it again. We don't make claims about the other tool beyond what this test measured.
The full comparison and its method
| Result | Vocals SI-SDR | Vocals SDR | Instrumental SI-SDR | Instrumental SDR |
|---|---|---|---|---|
| Extractune (HTDemucs, on the device) | 10.76 dB | 8.60 dB | 8.31 dB | 7.51 dB |
| vocalremover.org | 9.79 dB | 8.42 dB | 6.38 dB | 5.97 dB |
| The song, not split | 0.64 dB | Not scored | −2.06 dB | Not scored |
Where the models stand
The Demucs README lists how well-known separation models score on the same shared test set. Newer models score higher, and how a model is built matters as much as what it reads: Band-Split RNN reads only the spectrogram and still beats Hybrid Demucs with no extra training songs. In the two families listed both ways, Band-Split RNN and Demucs, extra training songs raised the score. Our fast mode, for phones without the memory for HTDemucs, uses a model from the Open-Unmix family, the first row; that 5.3 dB is the README's score for Open-Unmix across all four parts, not a measurement of our fast mode, which splits voice from music only.
| Model | Reads the sound as | Extra training songs | Average SDR, 4 parts |
|---|---|---|---|
| Open-Unmix | Spectrogram | None | 5.3 dB |
| Spleeter | Spectrogram | 25,000 | 5.9 dB |
| Demucs v2 | Waveform | None | 6.3 dB |
| Hybrid Demucs (v3) | Both | None | 7.7 dB |
| Band-Split RNN | Spectrogram | None | 8.2 dB |
| Band-Split RNN | Spectrogram | 1,700 (mixes only) | 9.0 dB |
| HT Demucs, fine-tuned (v4) | Both | 800 | 9.0 dB |
On your device or on a server
Server-based vocal removers upload your file and split it there. Extractune's free split runs in your browser instead: the first time, it downloads HTDemucs (about 90MB, stored at half the size of the original file and computing the same thing), keeps it, and runs it on your graphics chip through WebGPU, or on the processor where that isn't available. Your song doesn't leave the device, which is also why there's no queue and no monthly limit, and why a phone takes longer than a laptop.
The cost is your device's power. If your browser can drive the graphics chip, one split can run 6 minutes; if not, 2. Phones without much memory, and iPhones too, switch to fast mode, a lighter model that separates only voice from music, so the split can finish at all. HQ split, with Pro or 3 free tries, is the exception and says so beside its button: the song's audio goes to our server's GPU for a stronger model and is deleted within an hour.
Demucs in the browser against Demucs on your computer
Questions
Is HTDemucs the same as Demucs v4?
Yes. HTDemucs is the model Demucs v4 runs unless you ask for another, and that's the one we run, converted for the browser. The README's fine-tuned and six-part variants aren't offered here.
Why is the very first split slower?
Before it, the browser has to fetch the model, about 90MB. After that it's stored, and the next split begins at once.
What is SI-SDR?
Scale-invariant signal-to-distortion ratio: how much of a separated part is the real part and how much is error, in decibels, ignoring overall volume. Each extra 10 dB leaves a tenth as much error.
Will HQ split sound cleaner?
It runs a stronger model on a server GPU than a phone or laptop can run, so it's built for cleaner vocals. It isn't part of the test above, which scored the free split. Its audio goes to our server and is deleted within an hour.
Does the free split use my song for anything?
No. It runs on your device and the song isn't uploaded, so there's nothing on our side to store or train on.