A Tiny Eval Harness for Finance Claim Drift

Language-model benchmarks measure average skill on public tasks. Finance teams usually need something narrower: can a reviewer notice when a polished summary quietly changes the period, the unit, or the definition behind a number?

The Drift Test page is that narrower exercise. It loads seven synthetic claim/fact pairs from a local fixture file and asks you to classify the mismatch. There is no model call and no upload. It is not scored against a foundation model leaderboard.

What the cases cover

The fixtures intentionally mix failure modes that show up in finance workpapers:

• Value drift (the figure moved)
• Period drift (the window changed)
• Unit drift (thousands vs millions, rates vs amounts)
• Definition drift (same label, different accounting cut)
• Missing or unsupported claims

Each case is invented. The point is the review habit, not the industry anecdote.

How it fits the rest of the site

The harness pairs with the essay Small AI errors can drift into important decisions and with the lineage argument in Data lineage is a practical AI bottleneck. Together they argue for controls that compare recurring outputs to an independent source over time—not only a one-pass “does this sound right?” skim.

Unit tests under src/lib/drift-test.test.ts lock the evaluator behavior so the lab cannot silently change what “drift” means. That is the same discipline I want when a later prototype introduces model-assisted text: keep the checkable layer separate from the generative layer.

For the guided reading path, go to Start Here.