Batch Inference: Four Doors, Half Price

Sloth, a donkey, and a curious human Send a prompt and see what they're doing One hits a record with a question on his mind Two brothers pick it up, see what they find Korn takes his time, Herman's got the facts Back and forth, they never hold back My weird prompts, come on in, that's where we begin My weird prompts,

glad you're here This episode was generated using artificial intelligence. Please verify facts. The same tokens, half the price,

and almost nobody using it properly. That's the strange part, isn't it? The discount has been sitting there for years, and most teams still fire everything synchronously because it's easier.

Daniel's been thinking about this one. He wants the map. Who's actually good at batch inference right now? Does OpenRouter count as a batch provider? Can you run batches on Hugging Face? What happens if you go local? And what are the cost-effective routes somebody might actually use? Four doors into the same room. And the interesting question isn't which provider. It's why this market keeps reshaping itself every 18

months. So before we get into who offers what, it's worth being precise about what batch actually is, because the definition has drifted. Drifted how? The original pitch was simple. You submit a file of requests, you give up any guarantee about when they come back, and in exchange, the provider schedules your work around its own capacity and hands you a discount. That's the deal. Asynchronous, out-of-order results, 24-hour window, 50% off. And the 50% is compensation for what exactly? For giving up latency and streaming. That's it. You're not buying a worse model. You're buying the same weights with the timing taken out. The provider gets to batch your requests together, run them at 3 in the morning when the GPUs are idle, and keep the utilization curve flat. You get the discount because you made their life easier. So the discount is real because the savings are real. The savings are real. Whether the discount is fairly priced is a different question, and we'll get there. But the mechanism is genuine. You're selling your latency back to the provider. And the 50% number has become the anchor for the whole market. OpenAI, Anthropic, Gemini, Vertex, Bedrock, Azure, Maestrol, Grok, Together, Firework. all sitting at or near half price. It's the closest thing to a posted rate this industry has. Except it's fraying. It's fraying at the edges. XAI only gives 20% and only on a handful of Grok models. Deepenfra gives 20%. Bastion gives nothing. Inference.net's published discount was only 10% as of mid-2025. So the anchor holds for the big labs and slips for everyone else. Which tells you the discount isn't a law of nature. It's a marketing decision. It's a marketing decision that most of them made the same way because they're all competing for the same enterprise contracts.

So the first-party labs have all converged on basically the same plumbing, and the interesting

variation is in the edges. Nearly identical. JSONL upload, 24-hour target, 50% off, separate rate limit pool, no streaming. OpenAI caps you at 50,000 requests or 200 megabytes per file. Anthropic at 100,000 requests or 256 megabytes. Gemini takes inline