Running One Model Across Mismatched GPUs

A sloth, a donkey, and a curious human send a prompt and see what they're doing. One hits a record with a question on his mind. Two brothers pick it up, see what they find. Colin takes his time, Herman's got the facts. Back and forth, they never hold back. My weird prompts, come on in, that's where we begin. My weird prompts, glad you're here. This episode was generated using artificial intelligence. Please verify facts.

There's a thing people do with graphics cards that almost never works the way they expect, and it's worth an episode because the moment you own two GPUs, the temptation to bolt them together is enormous.

And the hardware makes it look so easy. Two slots, two cards, one machine.

Daniel wants to know what it actually takes to run one open weight model, say a DeepSeek, across several GPUs on a single Linux box, both versions of it. The scrappy path where you've got a card in there already, and you add a second one that doesn't match it, maybe different memory, different generation, even a different vendor. And the deliberate path where you're speccing a new server, and you buy several identical cards on purpose.

And he's asking about the underlying techniques, the tooling, and the honest version of whether that second card is ever worth it, which is a better question than it sounds because the answer is sometimes no.

That's the tension. Capacity and speed look like the same thing when you're shopping, and they aren't. So let's start with the vocabulary because everything else hangs off it.

Right. The first distinction, and it's the one people skip, is between running one model across several GPUs and running several copies of a model on several GPUs. Data parallelism is the second one. Each card holds a complete replica, there's no talking between them, and they just serve different requests. That raises throughput if you have concurrent traffic. It doesn't pool memory, and it does nothing for latency.

A queue of requests, one card each.

Model parallelism is the other thing entirely. You split one model, the cards cooperate on every token, and now you can fit something that doesn't fit anywhere. That's the mode Daniel is asking about, and it's the mode with all the sharp edges.

So the whole episode is pooling memory is possible, and it isn't free.

Every technique is a trade. You're buying capacity or you're buying speed, and the price is paid in communication between the cards. What governs that price is the interconnect, not the compute. That's the theme, and I'll be honest, it took me a while to internalize it properly.

What does that mean for the person who just wants to add a card?

It means there's no magic. Nothing in the stack looks at your two mismatched cards and transparently spreads a model across them by free memory. Pooling is always something you configure explicitly or something a specific backend gives you as a side effect. That's the honest starting point.

All right. Three techniques then. Which is the one people reach for first?

Tensor parallelism, usually because it sounds like the good one, and it's the good one for latency when your hardware supports it. The mechanism is this. You take each layer's weight matrices, and