I have been thinking about where AI agents actually break once they touch a business number. Not the reasoning, not the model choice. The part that sits between the agent and your warehouse. Turns out the quietest corner of my map, plain old BI, is where the most interesting fights are happening right now.
The agent isn't the problem. Your metrics are.
There's a case AWS wrote up about an AI agent that answered a revenue question and was off by $3.5 million. The model wasn't broken. It wrote valid SQL against real data and returned a confident, wrong number. The fix wasn't a better agent. It was putting a semantic layer between the agent and the warehouse so it stopped writing SQL at all and started calling defined metrics instead.
That tracks with everything I've seen. An LLM asked "what was revenue last quarter" has to guess which table, which date column, whether to count refunds, how to handle the one region that books differently. A human analyst carries that context in their head. The agent doesn't. So it invents a reasonable-looking answer, and reasonable-looking is exactly the failure mode you can't catch by glancing at a dashboard.
The boring work did the heavy lifting
KOHO built a Claude-powered self-serve analytics tool their team actually uses. The part worth sitting with is what made it work. It wasn't the LLM. It was five years of dbt governance and metric ownership they'd already done, including a full pipeline rebuild back in 2022. The model was the easy piece to bolt on at the end. The hard problem was defining what every number meant and who owned it, and they'd solved that years earlier.
Snowflake and dbt are both pushing the same idea now under different names, business ontology, semantic layer. Define the metric once. Approve it. Let agents reference it instead of inferring it. When Sigma runs 2,883 dbt models so every surface in the company agrees on the same number, that's not a vanity stat. That's the whole game.
What this means if you're smaller
You don't need 2,883 models. But you do need the three or four numbers your decisions actually hang on to mean exactly one thing. Revenue. Active customer. Churn. Write them down as code, not as tribal knowledge, before you point any agent at your data.
I've watched this from the other direction. "No workflows found" once nearly made me wipe a box, until I checked the actual volume and saw the real picture. Absence of an error is not proof of correctness. Same lesson here. An agent that returns a clean number isn't the same as an agent that returns the right one, and the only thing standing between those two is a definition you controlled ahead of time.
The agent is a consumer. Feed it governed metrics and it's useful. Feed it raw tables and a prompt, and you've built a very fast way to be confidently wrong.
3 things worth your time
1. Ramp's agents crossed a million sessions
Their agents now generate 10-15x more analytical queries than humans. The trick was treating dashboards and queries as code agents could edit. Analytics engineers. It turns out, have an edge over software engineers here.
2. DuckDB on the quant's desktop
Quants swapping heavyweight real-time platforms for embedded DuckDB and local Parquet caching, cutting 20-minute queries to instant on cached results. Worth a look for anyone paying remote compute to re-run the same job all day.
3. The middle layer makes or breaks enterprise agents
Financial services teams don't let LLMs do math. They pipe deterministic calculations through Python and feed results back for reasoning. The model's becoming a commodity. The connective tissue is where the real work lives.
If you've pointed an agent at your warehouse, I'd love to hear whether it agreed with your humans on the basic numbers. Reply and tell me.
