Spent part of this week reworking a routing layer that sits between a couple of models, and it kept reminding me that the model choice is maybe a third of the actual work. So when I saw a lab publish the part most people keep locked up, I paid attention.
The part of the stack nobody shows you
DeepSeek did something unusual this week. They open-sourced DeepInfer, the inference stack they run in production, along with numbers most companies treat as a trade secret. 155 trillion tokens a day per cluster. A 96% cache hit rate. Margins north of 80% and a payback period under three months on the hardware.
The model releases get the headlines. V4.1 Flash is fast and cheap and benchmarks well, and that's fine. But the thing I kept coming back to is that they showed the serving layer, not just the weights. That's the part I never get to see from the big labs, and it's the part that actually decides whether your AI feature is affordable.
The model was never the hard part
I've built enough systems to know that the interesting work lives between the pieces, not inside them. The retrieval layer. The routing. The caching. Whether you can swap a model without rewriting everything downstream. A separate piece this week put it well: most enterprises moved inference in-house and only a sliver are actually set up to run it at scale. That gap is real. I've watched it firsthand in a different domain, when a disk auto-scaled up at 90% and then never scaled back down. You pay for that headroom forever unless you go force it back. Infrastructure has a memory, and it bills you for it.
DeepInfer's cache strategy is the detail worth sitting with. They hold context for 72 hours where some providers hold it for 12. That's not a model improvement. That's an operations decision, and it's where a huge chunk of that 80% margin comes from. Serving the same tokens twice is nearly free if you built the plumbing to remember them.
What you can actually take from it
Not all of it ports. You're not running their cluster and you don't need to. But the shape is instructive. If you're deploying open models and your cost-per-task feels high, the fix probably isn't a cheaper model. It's looking at how often you're recomputing things you already computed, and how your cache is keyed.
Start by measuring your cache hit rate before you touch anything else. If you don't have that number, that's your Monday. The labs that make money on inference make it there, not in the weights. Now one of them showed the receipts.
3 things worth your time
1. Jev is Vercel's fastest-adopted model ever
A narrow, constrained-output model for classification and routing beat general LLMs to 35% adoption in two weeks. Worth remembering when you reach for a frontier model to do a job a small one does better.
2. Nobody brags about lines of code twice
Feature branches got faster. Production deploys got shakier. The verification tax AI adds is showing up in the DORA and CircleCI data, and it's the number teams actually feel.
3. The inference path is where AI value is won or lost
Pairs well with the DeepInfer story. 78% running inference in-house, 2% ready to do it at scale. The model is table stakes. The path around it is the work.
If you've measured your cache hit rate lately and it surprised you, I want to hear about it. Reply and tell me what you found.