Containers vs VMs for GPU Inference: What You Actually Pay

A sloth, a donkey, and a curious human. Send a prompt and see what they're doing. One hits a record with a question on his mind. Two brothers pick it up, see what they find. Corn takes his time, Herman's got the facts. Back and forth, they never hold back. My weird prompts, come on in, that's where we begin. My weird prompts, glad you're here.

This episode was generated using artificial intelligence. Please verify facts.

A container adds three hundredths of one percent, not three percent, zero point zero three.

Which sounds like a typo until you understand what a container actually is.

Daniel's question this week is about what you pay when you move the same open weight model off bare metal and into something else. He wants the overhead numbers, not the configuration guide. Container, KVM virtual machine with PCIe passthrough, bare metal as the baseline, and then the harder part, which is where that overhead actually lives, and whether NVIDIA and AMD differ in any way that matters.

And the reason it's worth a whole episode is that the published numbers look like they're describing different universes. You've got papers saying passthrough costs you nothing, and papers saying it costs you a third of your throughput.

Both are true. That's the fun part.

The real question isn't whether virtualization is slow. It's which isolation model on which hardware for which workload shape. Because the benchmark literature is quietly conflating at least four different things under the single word virtualization. There's the container, which shares the host kernel and the host driver. There's single GPU passthrough, where the hypervisor hands one card to one guest and gets out of the way. There's multi-GPU passthrough, where the guest owns several cards and the interconnect between them suddenly matters. And then there's shared GPU virtualization, the MIG and SR, IOV, and vGPU family, which is a completely different product with a completely different overhead profile.

Four different things wearing one word. That's why the numbers don't reconcile.

Right. And before we get into any of it, the baseline has to be pinned down, because bare metal isn't a single number either. Kernel version, driver version, whether the CPU governor is set to performance, whether the process is NUMA aligned to the right socket. A tuned bare metal baseline and a sloppy one differ by more than the entire container overhead we're about to discuss.

So let's start with the number everyone repeats. Containers are free. Find out whether that's actually true.

It's true, and the mechanism is boring in the best way. Docker and Podman share the host kernel. There's no hypervisor, no second kernel, no device emulation. When the inference engine submits a CUDA kernel inside a container, it goes through the same driver stack, the same IOCTL path, the same GPU, and the kernel executes identically. The container boundary is namespaces and scroops. It's a boundary around file systems, process trees, and resource quotas. It's not a boundary around the hardware.

So the GPU never learns. It's in a container.

The GPU has no idea. It's the same instructions on the same silicon.

What's the actual measurement?

The