One Model or Two? ASR Cleanup Explained

A sloth, a donkey, and a curious human Send a prompt and see what they're doing. One hits record with a question on his mind. Two brothers pick it up, see what they find. Corn takes his time. Herman's got the facts. Back and forth, they never hold back. My weird prompts, come on in. That's where we begin. My weird prompts, glad you're here.

This episode was generated using artificial intelligence. Please verify facts.

There's a small text model that exists for the sole purpose of cleaning up after a speech model. Four hundred and eighty-four megabytes, and it won't listen to a single second of audio.

And on paper, it shouldn't need to exist. That's the thing Daniel's poking at.

Daniel's question is this: We've talked about transcript post-processors like S1 Mini before. His point is that if ASR models can be trained to do punctuation, capitalization, paragraph breaks, and error correction themselves, why do we need a whole separate model at all? Where does recognition actually end and cleanup begin? And could the two ever just be one model with one set of weights?

It's a good architectural question because the naive assumption is that speech recognition is physically capable only of turning audio into words, and that assumption is wrong, and it's been wrong for a while.

So the arc is architecture first, then what fine-tuning can buy, then the case for keeping them apart, then whether the weights can ever be merged.

And the answer to that last one is mostly no, but not for the reason people think.

Start with the naive model, Herman, because I think it's load-bearing.

The naive model is a two-stage pipeline. Stage one, the acoustic model turns sound into tokens. Stage two, a language model fixes the mess, and that second stage feels obligatory. You hear it described as if the recognizer is a stenographer who writes everything in lowercase with no punctuation and hands it to an editor.

Which is exactly what LibreSpeech PC says traditional ASR does, lowercased words without punctuation, which it says necessitates a subsequent text processing model.

Right, and that framing is a description of a corpus, not a law of physics. Because then you have Whisper.

Whisper being the pole where the formatting is baked in.

Whisper was trained on six hundred and eighty thousand hours of weakly supervised audio, a hundred and seventeen thousand of it across ninety-six non-English languages, and a hundred and twenty-five thousand hours of translation into English. And the targets weren't cleaned up. The paper is explicit. They train the models to predict the raw text of transcripts without any significant standardization. And they say that removes the need for a separate inverse text normalization step.

Deliberately raw.

Deliberately. They filtered out machine-generated transcripts that lacked complex punctuation, paragraph white space, or capitalization on the reasoning that an all-uppercase or all-lowercase transcript is very unlikely to be human-generated. So the training data itself was curated to teach the model what readable text looks like.

The other pole.

NVIDIA's Parakeet TDT, a transducer, not an encoder decoder. Six hundred million parameters, fast conformer encoder