82% accurate. Every amount wrong by a factor of a hundred.

Automation invents more exceptions than it removes.

The documents contained 26 genuine exception cases — the reverse charges, credit notes and malformed VAT numbers the person in the chair had learned to spot. The pipeline generated 47 of its own: extractions that failed, abstained or contradicted themselves.

They behave differently, too. Document exceptions concentrate — 62% came from four counterparties, which is fixable with a phone call. Extraction failures spread across 26 counterparties, because they follow the extractor rather than the sender. Supplier-level remediation does not touch them.

Nobody plans for these, because they do not exist until you automate. This is the single strongest argument for finding out before you commit the budget.

The accuracy number was the thing hiding the error.

The pipeline scored 82% field-level accuracy on its first pass. Underneath it, every monetary value was wrong by two orders of magnitude — the model had silently decided to report pence, and never mentioned it.

What caught it was not a better model. Each extracted value was required to carry the exact line of text it was read from, and the two disagreed. That check needs no labelled data and no tax expertise, and it runs in production indefinitely. One added sentence took the disagreement rate from 73% to zero, and whole-document accuracy from 11% to 50%.

The model that scored better was the one to avoid.

A local model scored 50% on whole-document accuracy against a hosted model’s 45%. It scored higher because it guessed: on expense categorisation it declined 3 times out of 120, where the hosted model declined 43 — and was right 97% of the time on the questions it chose to answer.

In operational terms, the willing model pushes more wrong documents into the ledger unchallenged. Any metric that counts “I don’t know” as a wrong answer selects for exactly the wrong model, which in a regulated process is backwards.