Skip to main content
Pedro Alizo

Lab / Synthetic data

Drift Test

A synthetic challenge set for plausible errors in finance summaries: amount, period, unit, definition, and omitted or unsupported facts.

Prototype: no uploads, external requests, or model calls. This is not an LLM benchmark and does not score prose quality. Read the method and limitations.

0 of 7 attempted · 0 correct

A clean match

A reviewer is checking a generated sentence against a synthetic operating statement.

Metric
operating-expense
Period
2026-Q2
Value
1,250,000 USD
Definition
Booked operating expense for the quarter

Operating expense was $1,250,000 in 2026-Q2.

What issue do you find?

Method and limitations

Each fixture contains synthetic source facts and a candidate summary with a separately annotated structured claim. The evaluator compares metric, period, unit, value, and definition, then flags missing source facts or claims without a matching source. It does not parse the summary text; the annotations are hand-authored.

These seven cases are a prototype challenge set, not a representative benchmark. They do not test a language model, establish model rankings, assess materiality, or show how often people catch errors in practice. Expand and independently review the cases before drawing broader conclusions.