Merging vs Distillation: Can Two Models Become One?

Sloth, a donkey, and a curious human Send a prompt and see what they're doing One hits a record with a question on his mind Two brothers pick it up, see what they find Korn takes his time, Herman's got the facts Back and forth, they never hold back My weird prompts, come on in, that's where we begin My weird prompts,

glad you're here This episode

was generated using artificial intelligence. Please verify facts.

So the argument I keep coming back to is whether the cleanup step was ever a separate model at all, or whether we just got lazy and split

the job in two. That's the framing Daniel sent us, and it's sharper than most of what I see written about this. His example is the ASR pipeline. You've got a speech recognizer that turns audio into raw text, and then a smaller language model that fixes the errors, restores the punctuation, cleans up the formatting. Two models, one job. And the

question underneath it's the good one. If they're doing one task, is there a way to fold both into a single set of weights so the whole thing runs through one inference pass?

Right. And he's careful to separate the things people lump together under model consolidation. Weight averaging, task arithmetic, lower merging, distillation, multitask learning. He wants to know which of those actually move capability between models, and which of them quietly require the two models to share an architecture or a training history. Then he narrows to the

case I find most interesting. A big foundation model and a small specialist. Can the knowledge of one be poured into the other without just running both in sequence?

And he asks for real papers and a clear split between what's been demonstrated and what's still a proposal. Then he brings it back to the ASR pair specifically and asks what did it actually take. training the recognizer on the corrector's outputs, a multi-task redesign, or some more direct fusion. So the marketing

word is merging, and the actual word for most of what he's describing is distillation, and the two aren't the

same thing at all. They're opposites in a way. Merging takes two models that already share a lineage and combines them without training anything new. Distillation takes two models that don't share a lineage and trains a third one to imitate both. Same goal on the box. Completely different engineering. Which means

the vocabulary trap is the first thing to get out of the way, because everything downstream depends on it.

It does. The dominant survey on this, Yang and co-authors, defines model merging very tightly. You're combining the parameters of several models into one, with no access to the original training data, and no extra inference cost. That's the definition. You get one model, the same size as the ones you started with, running at the same speed. And there's a requirement buried in there that people skip past. Two of them, and they come up in Donato Crisostomi's overview from last year. First, the merged model has the same number of parameters as the endpoints, so merging isn't a way to take a large model and compress it into a small one. It's a way to take two same size models and produce one same size model. No shrinking. Second, no extra