On trusting numbers from a language model
Every AI finance tool says its numbers are trustworthy. Here is how to tell.
Three different architectures hide behind the same sentence, and they fail in different ways. This is a test you can run on any vendor, including us.
Julian Simpson, founder of StartOS. September 2026.
Ask any AI accounting product whether it makes up your numbers and you will get a version of the same answer. It reads from your ledger. It does not guess. It follows rules. The sentences are close enough to identical that they cannot be compared, which is a problem, because the things they describe are not the same and do not fail the same way.
There are broadly three claims in circulation, and it is worth separating them before believing any of them.
Three claims wearing one sentence
Retrieval. The model is given your real data rather than working from memory. This is a real improvement and it is now table stakes. What it fixes is the source. What it does not fix is the arithmetic, because the model is still the thing doing the arithmetic. A model handed four correct figures can still hand you a fifth that is wrong.
Review. A qualified human checks the output. This is genuine assurance and it is the oldest one we have. Its limit is arithmetic of a different kind: nobody reviews every answer to every question asked at eleven at night, so review tells you about the month, not about the sentence currently on your screen.
Computation. The model never produces a number at all. Each figure is computed by ordinary code against the database and passed to the model, which writes the sentence around it. This is the strongest of the three, and it is what we build.
It is also, on its own, not enough. I know that because we shipped it and then watched it fail.
The number nobody made up
Iskra, our finance agent, answers questions by calling typed tools that query the ledger directly. 62 of them at the time of writing. The model picks the tools; the database produces the figures. We had a rule in the system prompt stating that every number in an answer must come from a tool result.
Someone asked for a one page summary of the financial statements. The answer contained a cost-of-revenue figure of $434,666.49. No tool returned that number. The tool that had been called returned revenue, gross profit and gross margin, but it did not expose cost of revenue as a field. So the model subtracted gross profit from revenue, which is correct as an identity, and reported the result as though it had been read.
The rule said do not invent numbers. The model did not invent one. It derived one, and derivation looks exactly like reasoning.
This is the part worth sitting with. The arithmetic was right. The inputs were real. Nothing was hallucinated in the sense people mean when they use the word. And the figure was still not a fact about the business, because it had never been computed against the ledger. It was a plausible reconstruction presented with the same confidence as a retrieved value, and no reader could tell the two apart.
A second one, in the same family: Iskra reported a Rule of 40 of 284 when the correct figure was 68.4. Two different tools had computed that metric over two different periods, and the model, finding them inconsistent, reconciled them by recomputing from components. Both inputs were genuine. The output was nonsense.
Neither failure is fixable by asking the model more firmly. A prompt is a request. If the only thing standing between your ledger and a confidently wrong figure is a sentence asking the model to behave, you do not have a guarantee, you have a preference.
So the question is not whether it retrieves. It is whether you can check.
Which turns a vague comparison into four questions with right answers. These are worth asking any vendor, and they are worth asking us.
1. Where did this number come from?
A good answer: The system names the call it made, the arguments it passed, and the raw values that came back. You can read them without asking anyone.
A weak answer: “It reads from your ledger.” That describes the input, not this answer.
2. Would the same question give the same number tomorrow?
A good answer: The figures are computed by code, so they move only when the underlying data moves. The wording around them may vary.
A weak answer: Nobody can say, because the number is produced by the model and the model is sampled.
3. What happens when it does not know?
A good answer: It says so, and says what it would need. A missing field produces an admission, not an estimate.
A weak answer: It produces something plausible, and plausible is the failure mode.
4. Can you take the evidence with you?
A good answer: You can export the working: the calls, the values, the arithmetic, in a file you can hand to someone who does not have a login.
A weak answer: You can take a screenshot.
Notice that none of the four asks about the model. They ask about the evidence the system can produce. A vendor who cannot answer them is not necessarily careless, but they are asking you to take the numbers on trust, and trust is the thing accounting exists to avoid needing.
What we did after the COGS bug
We stopped relying on the prompt and made the property checkable. Every figure in an answer is now extracted and matched against the numbers the tools actually returned. Anything that does not trace triggers one bounded correction round in which the model must either cite the tool value or say it does not have the figure. Whatever still does not trace after that is logged and shown to the reader rather than quietly passed through.
A separate check compares figures across tools, so two tools reporting the same metric for the same period cannot disagree without being flagged. That is the Rule of 40 failure, caught structurally rather than argued about.
The honest scope, which is also documented in the source: bare integers are not checked, because they are overwhelmingly years, dates and list positions, and the counts that matter come from dedicated counting tools. Answers drawing on general knowledge rather than your data are exempt from the check and are labelled as such, so you know which kind of answer you are reading.
The useful side effect is that every flag is a gap in tool coverage. The COGS failure was fixed by exposing cost of revenue as a field, not by telling the model to try harder. The check makes the product better by telling us what it cannot yet answer honestly.
What determinism does not buy you
It does not make your books correct. If a transaction is miscoded, a deterministic system will report the miscoding precisely, with a full audit trail, in a downloadable file. Traceable and wrong is still wrong. Determinism is a property of the path from your ledger to the sentence, and it says nothing about how the ledger got that way.
It does not cover judgment. Whether to raise now, whether a customer is worth keeping, whether a variance matters: these are not tool calls, and anyone presenting an opinion with the same confidence as a computed figure has confused the two.
It does not remove the accountant. It removes the part of their job that consists of re-deriving numbers by hand to find out whether they can be trusted, which is the part none of them wanted.
Any vendor claiming determinism end to end, this one included, is describing an ambition. Ask which specific figures carry a trace, and what happens to the ones that do not.
Run the test
The four questions are the point of this piece, more than anything they happen to say about us. Ask them of whatever you are evaluating. Ask them of us: every Iskra answer carries the calls it made, the values that came back, and the arithmetic, and you can download the lot as a spreadsheet. Here is one, for a made-up company, with no login. Open the Calculations sheet and change a number: the gross margin recalculates, because the cells reference the source rows rather than restating them.
If a finance product cannot show you its working, the number on the screen is not evidence. It is a claim, and you are the one who will have to defend it.