July 15, 2026
Context Beats Capability: What Happened When We Gave Two AI Models the Same Finance Question
Precog
TL;DR
We asked AI agents a routine CFO question against our real business data, and changed only two things: how much business context the agent could understand, and which model answered (a mid-tier model vs. a frontier model). Without the context layer created with Precog, both models failed. Confidently, and wrong by one to two orders of magnitude. With it, both succeeded. The most accurate answer came from the frontier model with full context. The cheapest correct answer came from the mid-tier model with full context, for a fraction of the cost of the most expensive wrong answer.
Context beat capability. On accuracy, and on cost.
The experiment background
There's a widespread assumption that as models get more capable, the "context problem" just takes care of itself: point a frontier model at your data warehouse and it'll figure out what you mean. We wanted to test that directly. Our product is an AI context engine for business data, so this is precisely the gap we exist to close.
So we built a simple test harness. Same question. Same underlying data. The only variables were context depth and model choice. Every answer was scored by an independent LLM judge against a verified ground-truth result, using a rubric that grades method, not just whether a number looked plausible.
The setup
One real finance question, the kind a CFO asks every month, answered under three levels of data access:
- Native connectors: the agent reaches the business systems through their own out-of-the-box MCP server, with no unifying context.
- Warehouse SQL: the agent has direct query access to the data warehouse (Snowflake, in this case), but no business-context layer telling it what the terms mean.
- Full context: the agent has the same warehouse access plus Precog's context engine for AI: the definitions, the calculation methodology, the entity resolution, and the data-source map that encode how the business actually works.
Each level was run with two models: a mid-tier model, Claude Sonnet, and a frontier model, Claude Opus, three times each, to capture consistency as well as correctness.
Scoring was done by an independent judge against a verified answer key, on three dimensions:
- Accuracy: did it get the right result?
- Source: did it use the right data?
- Nuance: did it handle the ambiguities a competent analyst would?
The results
Without the context layer, both models failed, regardless of capability.
Given only native connectors or raw warehouse SQL, the agents didn't get a slightly-wrong answer. They answered a different, easier question and presented it with full confidence. Depending on which convenient data source they reached for, the result was wrong by one to two orders of magnitude. And the frontier model failed here too, which is the whole point. More capability didn't rescue a missing definition — it just produced a more articulate wrong answer.
The judge marked every no-context run frustrated (its lowest verdict). On our normalized scale, the no-context conditions scored around 0.16 (native connectors) and 0.05 (warehouse SQL).
With the context layer, both models succeeded.
Supplied with the full Precog context engine, the agents looked up the real definition, applied the documented methodology, drew from all the right data sources, and reproduced the verified answer. The frontier model scored a perfect 10/10 from the judge. The mid-tier model scored 7.5/10: not perfect, but a genuinely useful, correct answer. The full-context condition scored about 0.70 normalized, against 0.16 and 0.05 for the two no-context conditions.
That gap, roughly 4x to 13x higher scores, came entirely from context. Same models. Same data. The only thing that changed was whether the agent could interpret and understand the meaning behind the numbers.
The cheapest correct answer beat the most expensive wrong one.
Since every run actually completed, we can compare real costs. The pattern that matters for anyone deploying this at scale: the most expensive configuration was the frontier model running through native connectors, the one that was also wrong. The cheapest correct configuration was the mid-tier model with full context. Full context didn't just make the answers right; by giving the agent a direct, well-described path to the data, it made them cheaper too. The no-context runs burned tokens and tool calls thrashing around looking for an answer they never reliably found.
The three findings that matter
1. Model capability is not a substitute for context. The frontier model, given no business context, failed exactly like the smaller one. If you're betting that the next model release will fix your AI-analytics accuracy, it likely won't. The bottleneck isn't reasoning power. It's meaning.
2. The failed answer sounded confident. None of the no-context runs said "I'm not sure." They produced fluent, authoritative answers to the wrong question. In a finance setting, a confidently-wrong number that looks right is far more dangerous than a visible failure.
3. Context improves accuracy and unit economics. The cheapest correct answer cost a fraction of the most expensive wrong one. Better context means the agent takes a shorter, surer path to the data, which shows up on both the accuracy line and the cost line.
The real test: a live month-end close
A controlled experiment is one thing; a live financial month-end close is another. The more convincing test came later: the same context layer, our own books, our own close — actual money, actual deadlines, real stakes.
The session began with a routine ask: build a renewal schedule from every sales invoice the company had ever issued, then reconcile it against our recurring-revenue base. A first pass over the raw invoices looked complete but was mostly wrong. Most historically billed accounts came back flagged as overdue renewals. One major renewal showed a value of zero — the latest invoice on file was still a draft. Another was dated a full year late, a mislabeling the team already knew about but that lived nowhere in the data. And the headline total, while it resembled ARR, was something else entirely.
On the second run, with Precog loaded, the agent made a single context call. Twelve saved factors came back: definitions, identity mappings, contract facts, and close figures deposited by earlier sessions. With those loaded, it rebuilt the recurring-revenue base to the exact penny on the first attempt, matched the renewal book against it, and by the end of the session every variance was explained and exactly one genuine overdue renewal remained. The agent asked zero questions about definitions or terms; every question it did ask was a real business decision, not "what does ARR mean here?"
The traffic ran both ways. The session posted 18 corrections back into the context layer: churn events nobody had recorded, a contract consolidation complete with the journal schedule to post, and one business rule, stated once in plain English, that will apply to every future renewals analysis. Several of the "errors" it surfaced turned out to be real bookkeeping gaps, confirmed and fixed the same day. With definitions and business logic accounted for, what remained were actual findings.
The time impact: rebuilding the revenue definition from scratch would have cost an estimated 2–4 hours of founder Q&A across dozens of clarifying questions (where a single wrong answer can corrupt everything downstream), and the reconciliation itself is 1–2 days of analyst work, reduced to minutes because the identity mappings and close facts were already there. And some of the saving can't be measured in hours at all, because the information simply didn't exist in the structured data — facts like a mislabeled invoice date, a deal whose terms exist only in an old email, or a customer that changed names twice. They exist only because a human said them once and the context layer kept them.
Even a conservative extrapolation makes the business impact obvious: a finance team running a similar analysis monthly is looking at one to two analyst-days plus several hours of executive time saved per close, on this single workflow, before accounting for the compounding effect of each correction making future sessions faster.
What this means if you're deploying AI on your business data
The market is racing toward "point a powerful agent at your warehouse and ask questions." This experiment is a direct test of that assumption — and it didn't hold. Raw access plus a frontier model gave us expensive, confident, wrong answers. The same models, given a curated context layer, definitions, methodology, entity resolution, a map of where the truth actually lives, gave correct answers — cheaper.
The differentiator in product-grade AI analytics isn't model capability or power. It's who has a governed context layer that tells the model what the data actually means, specific to the intended use case. That's the layer we build.
See it for yourself
If your team is running AI on business data and need accurate and trustworthy answers, the fix probably isn't a model upgrade. It's context.
.svg)
