RA-Bench: receipt extraction, fake detection, evidence and duplicates

RA-Bench 0.2.0. These definitions, thresholds and exclusions were written before the run and change only in a new suite version. Who and what. Receipt Audit runs this benchmark on its own product. Every target gets the same cases, the same files, the same timeout (180 s per attempt by default) and the same scoring; only the target changes. A target is our engine ("self", in process, with the duplicate network on an empty in-memory backend that lives for one repetition) or an agent's own model asked with one plain prompt ("diy"). DIY targets get the same file preparation as our engine (upright, at most 3072 px and 3.75 MB) so that API size limits don't count against them. Transport failures (network errors, HTTP 408, 429 and 5xx) are retried up to 3 times with exponential backoff; nothing else is retried, and a wrong answer is a result. Models are pinned: no fallback to another model. Units and intervals. A case run several times counts once, with the share of its runs that succeeded. Rates carry a 95% Wilson interval over cases (operations rates: over calls). ROC AUC, detection at a fixed false-flag rate and line-item F1 carry a 95% percentile bootstrap interval (2000 resamples, genuine receipts and fakes resampled separately, seeded per row so a re-scored run gives the same interval). Latency and cost have no interval. Every row reports n. Rows are given for all cases and for every stratum (a "dimension=value" tag on the case, or a label-derived group) with at least 10 cases. Which cases count. Extraction, checks and authenticity use base cases only: cases that aren't a transform of another case (no lineage). Duplicate transforms count only for the duplicates task. Operations use every call. Extraction. Scored only on fields the label covers; a field labelled null means the document doesn't show it, and only a null answer is right. Merchant name: lowercase, accents stripped, "&" read as "and", punctuation dropped, then a match at edit-distance similarity ≥ 0.9. Date: the exact ISO date (YYYY-MM-DD). Subtotal, tax total and total: within 0.01. Currency: the exact ISO 4217 code. Receipt number: equal after keeping only letters and digits, lowercased. "All fields right": every labelled scalar field of the document right in that run (line items excluded). Line-item F1: labelled and extracted items paired one to one when descriptions are at least 0.8 alike (same normalisation) and totals match within 0.01, micro-averaged over all documents. Valid output: the call succeeded and returned readable fields. A failed call or an unreadable answer gets every field wrong. Checks. totals_add_up is scored against the label "the printed amounts add up". A positive is a document whose amounts don't add up; the check flags it when it reports failed. Recall over documents that don't add up; precision over documents it flagged; false alarms over genuine documents that do add up; coverage is the share of documents where the check ran (a check that couldn't run isn't output). Targets without deterministic checks report this task as not supported. Authenticity (genuine vs fake). Higher risk means more likely fake. ROC AUC uses each target's risk as the score; a result without a risk scores 0.5 (no opinion), ties count half. Detection at a 1%, 5% and 10% false-flag rate: the share of fakes flagged by the lowest threshold that flags at most that share of genuine receipts (ties at the threshold are all flagged). At the documented thresholds risk ≥ 0.5 and risk ≥ 0.7: the share of fakes flagged (detection) and of genuine receipts flagged (false flags); a result without a risk is never flagged. Strata from the labels: each fake class ("class=ai_full", …) and fakes whose amounts add up or don't ("arithmetic=consistent", "arithmetic=inconsistent"), each compared with every genuine receipt; and provenance metadata ("provenance=intact", "stripped", "none"), genuine receipts and fakes alike. Duplicates. Every case is submitted by a different account, so a repeat is "someone else submitted this receipt before". Recall: the share of duplicate transforms (re-encoded, cropped, re-shot copies of an earlier case) the target recognised as seen before; exact recall counts only matches on the identical image hash. False matches: the share of distinct receipts reported as seen before, for hard negatives (different receipts that look alike: same merchant, template or day; "false_match_rate") and for the other genuine receipts ("false_match_rate_genuine"). Originals always run before their transforms. Targets without memory across checks (DIY prompts) report the task as not supported, not as zero. Decisions. A decision is the yes/no a caller acts on: flagged when risk ≥ 0.5. Consistency: over base cases run at least twice, the share whose decision is the same in every repetition; total consistency, the share whose extracted total is the same (within 0.01, or absent) in every repetition. Calibration error: expected calibration error of risk against "is fake" over every run with a risk, 10 equal-width bins, weighted by run count (lower is better; it depends on this set's share of fakes). Expected cost per 1,000 receipts at risk ≥ 0.5 and ≥ 0.7: 1000 × ((1 − p) × false-flag rate × C_review + p × (1 − detection) × C_miss). Defaults, each from a cited, graded source (research/41-benchmark-cost-model.md): p = 1% of submitted receipts are fake (SAP Concur's share of receipts flagged as AI-generated; no study measures the true share, so this is the best available assumption, not a measurement); C_miss = $100, the face value of a missed fake (AppZen's mean AI-generated fake, 12 months to May 2026); C_review = $17, one false flag's review (18 minutes, GBTA, at a loaded auditor wage of $57.47/h, BLS). At these defaults one point of false-flag rate costs as much as about 17 points of detection. The sweep varies one assumption at a time over its published spread ("assumption=…": fakes 0.1% and 5%; a missed fake $32 and $182; a review $8 and $73). Approving everything costs 1000 × p × C_miss ($1000 at the defaults), reported as its own row: a detector above it loses money under those assumptions. The formula counts only mistakes (it treats reviewing a fake that was rightly flagged as free) and leaves out unmeasured costs: an employee's time and trust after a false flag, and repeat fraud or tax penalties after a miss. This set has far more fakes than real submissions, so the cost re-weights the measured rates by p rather than using the set's own mix. Evidence. A yes/no is easy to get from any model; a reviewer also needs to know what is wrong. Edited field named: over fakes whose label names the edited field (AI-inpainted and hand-edited fakes), the share of runs that flag the receipt and whose first 3 reasons point at that field. A reason points at a field through an explicit "@field" tag, a deterministic check about it (an arithmetic check points at the total and subtotal), or, for free-text reasons, a pre-registered keyword (total, amount due; subtotal; tax, VAT, GST; date; merchant, store, vendor, logo, name; item, line, quantity, price). Verifiable evidence. A finding is proof of tampering when a caller can check it without trusting a model and it shows the document was generated or changed: an AI-generator, AI source-type or receipt/invoice-generator tag read from the file, PDF modification records, a thumbnail that doesn't match the image, a photo taken before the printed purchase date, printed amounts that don't add up (with the numbers), or a duplicate-network match. Other file facts (an editor or office tool, missing camera data) and plausibility or policy checks (tax rate, the receipt's age, currency) are real but don't show tampering, and don't count. A model's visual judgement is an opinion, however confident. Fakes caught with proof: the share of fakes flagged with at least one proof among their reasons. Genuine receipts with proof: the guard against false proof (lower is better). Only targets that can return verifiable findings report this task; a plain prompt returns opinions by construction and reports it as not supported. Operations. Latency: wall time of the final attempt of each successful call, measured by the harness (p50, p95, p99). Cost per check is what the caller pays for one check: for Receipt Audit, the list price of its smallest credit pack per check; for a DIY baseline, its model tokens × the AI Gateway list price on the run's price date. Model cost per check is the model tokens × that list price for every target, our engine's own model call included (infrastructure not included). Both are averaged over successful calls. Error, refusal and timeout rates over all calls. Findings. Each of our targets is compared with each baseline on the headline rows of every task (stratum "all"): a win or a loss only when the 95% intervals don't overlap, otherwise a tie; for latency and cost, which have no interval, only when one value is at least 1.2 times the other. A comparison resting on fewer than 10 cases (or calls) on either side is always a tie. A task a baseline can't do by construction is reported as such, not as a score of zero.