Batch Inference on Your Own GPU: vLLM, SGLang, Ollama, llama.cpp

A sloth, a donkey, and a curious human send a prompt and see what they're doing. One hits a record with a question on his mind. Two brothers pick it up, see what they find. Corn takes his time, Herman's got the facts. Back and forth, they never hold back. My Weird Prompts, come on in, that's where we begin. My Weird Prompts, glad you're here.

This episode was generated using artificial intelligence. Please verify facts.

Daniel sent us a prompt that starts from a very reasonable place. Batch inference on hosted APIs is one of the best deals in AI right now. You write a file of requests, you send it off, you wait a few hours, you pay half price. DeepSeek does it, a few others do it.

It's the best throughput per dollar on the market, and it isn't close.

And Daniel's question is, why can't I have that on my own box? He's got a Linux server, some GPU, AMD or NVIDIA. He wants to point a file at it and get a file back. Does he need a special framework? Is Ollama enough? What changes when the card is red instead of green?

The short version is that the hosted version is a product, and the self-hosted version is a configuration problem.

Which is the whole episode.

A batch API on a hosted provider is a billing construct. They take your file, they fill their idle GPU capacity with it, and they give you half off because the work has no latency requirement. You're selling them the right to schedule you whenever. That's the deal.

So the discount is arbitrage, not generosity.

Exactly that. Which means the thing you're trying to reproduce locally isn't the API. The API is a thin wrapper. The thing you're reproducing is the scheduling.

Define the thing properly, because batch inference gets used to mean about four different things.

On your own hardware, batch inference is just high-concurrency inference with continuous batching. There's no separate mode you install. The engine adds and retires requests from the running batch on every decode step, so the GPU never sits idle waiting for the slowest sequence in a fixed group to finish.

Say that again, because I think that's the part people skip.

With a naive setup, you take a group of requests, you run them together, and the whole group finishes when the longest one finishes. Everything shorter than the longest one is wasted capacity. Continuous batching drops the finished ones the moment they're done and slots new ones in, so the batch is constantly turning over.

And the file in, file out part. On top of that?

Forty lines of Python if you want it. Or you use the wrapper the engine already ships. But the wrapper is nothing without the scheduler underneath.

So what does the actual work look like on a Linux box? Two shapes.

Two shapes, and picking the wrong one is where most of the lost throughput lives. The first is offline and in process. You load the model in Python, you hand it the whole list of prompts, you wait on the call. VLLM's offline class does this. SGLang's engine does this. No HTTP server at all.

And the second?

Server plus concurrent client. You stand up an OpenAI-compatible endpoint, and you fire many requests at it at once from an async client with a concurrency limit. What you must not do is loop over your prompts