Scanned-record agents need an evidence-chain acceptance contract.

The answer is the last link, not the evidence

Organizations keep decades of budgets, statements, schedules, and reports as scanned PDFs. An agent can reach those drives through MCP, invoke OCR, and return a figure in seconds. That convenience compresses a ten-step evidence path into one confident number.

TreasuryQA Bench, submitted October 28, 2026, restores the hidden work. It gives an agent one question and access to 1,435 scanned U.S. Treasury publications containing 118,524 image-only pages. The agent must search the archive through a Dropbox MCP server, locate the relevant volume and page, read the scan with hosted OCR or its own tools, and return one exact figure. A Python grader checks the response against an accepted range.

The paper reports that the best tested configuration solved 64.8 percent of attempts; the weakest solved 16.7 percent. Every agent degraded on blurrier scans. Most failures returned a wrong number instead of declining to answer, and digit misreads were the most common cause. These are publisher-reported, benchmark-specific results. Sixty-seven of seventy tasks are private, so outside teams cannot reproduce the leaderboard end to end today.

The operational lesson is broader than Treasury records. A scanned-record agent should be accepted as independently testable stages. Search quality cannot excuse OCR error. OCR confidence cannot prove the correct period. A cross-foot cannot prove the right row. A final script match cannot prove the route was safe for a new archive. Human approval cannot repair evidence that was never preserved.

TreasuryQA closes two convenient shortcuts

Many benchmarks hand the relevant file to the model, removing archive search. Others provide extracted text, removing image reading and layout. TreasuryQA keeps both. That resembles a shared drive where filenames, contents pages, tables, footnotes, repeated periods, and scan quality all matter.

Use public results to choose experiments, not approve production. The 64.8 percent best result says even a strong agent-tool combination leaves a large unresolved error surface. It says nothing about your invoices, legacy ledgers, handwritten schedules, multilingual scans, permissions, or materiality.

An internal suite should sample document family, year, scan source, image quality, table type, unit, page density, handwriting, and answer materiality. Include questions whose correct outcome is “not enough evidence,” “ambiguous,” or “requires another record.” If every task has one discoverable number, the suite rewards persistence without testing judgment.

Split the pipeline before measuring it

A production receipt should expose seven stages: resolve the authorized corpus; retrieve candidate files; select pages; extract bounded crops; bind tokens to table coordinates and headings; normalize units and calculate; then validate and release. Preserve alternatives, not only the winning path. Nearly identical volumes and disagreeing OCR reads are part of the evidence.

question
  -> authorized corpus snapshot
  -> candidate files + retrieval scores
  -> page images + immutable hashes
  -> OCR crops + engine/version
  -> table coordinates + row/column headers
  -> normalized value + unit + period
  -> deterministic checks + reviewer decision

Financial tables depend on bindings. A digit belongs to a cell; a cell belongs to a row and column; headings define period, unit, and scope; footnotes can redefine the number. A text dump that preserves characters but loses alignment can be less trustworthy than a lower-confidence extraction with coordinates.

def accept(candidate, policy):
    assert candidate.corpus_hash == policy.corpus_hash
    assert candidate.page_hash in policy.authorized_pages
    assert candidate.cell_bbox and candidate.row_header
    assert candidate.column_header and candidate.unit
    assert candidate.source_year == candidate.question_year
    assert abs(candidate.cross_foot_delta) <= policy.max_delta
    return candidate.reviewer_status == "APPROVED"

Not every question supports cross-footing, and an allowed range must reflect documented rounding rather than hide uncertainty. Make each applicable check explicit and stop when a required check is impossible.

Bind acceptance to an exact corpus and toolchain

Identify the archive snapshot, indexer, retriever, OCR engine, vision model, MCP server, prompts, permissions, answer schema, deterministic checks, challenge set, and reviewer. A material change creates a new cohort. “Same model” is not enough when a hosted OCR endpoint or drive index changes independently.

scanned_record_acceptance: v1
scope:
  corpus_snapshot: treasury-archive-2026-10-28
  permitted: [single_cell, bounded_sum, cross_foot]
  excluded: [legal_interpretation, unreviewed_handwriting]
toolchain:
  drive_mcp: {version: "1.8.2", read_only: true}
  ocr: {provider: "pinned", version: "2026-10-20"}
evidence:
  retain: [query, candidates, file_hash, page_hash, crop, bbox,
           raw_ocr, headers, unit, formula]
checks:
  second_read_on_low_confidence: true
  period_and_unit_match: required
  wrong_answer_must_not_be_silent: true
release:
  named_human_review: required
  approval_expires_on_change: true

Set error budgets by consequence. A historical figure for an internal note can use a different review route from a value entering a regulatory report. Materiality is not only dollar magnitude: a small percentage, date, identifier, or covenant threshold can change a decision.

Keep access control outside the model. The MCP server must enforce identity, tenant, folder, and file permissions; the agent must not receive documents because semantic search found them relevant. Log denials without exposing restricted filenames. Test authorized decoys so the system proves both retrieval and non-use of forbidden evidence.

Verify the route, reading, and math separately

Use three lanes. Retrieval asks whether the correct file and page entered the candidate set. Extraction asks whether characters, structure, and coordinates were captured. Answering asks whether normalization, arithmetic, and formatting were correct. One exact-match score cannot localize the failure.

Challenge digit confusions such as 3/8, 5/6, 0/8, decimal points, commas, parentheses, minus signs, and faded superscripts. Add wrong-year volumes, duplicate table titles, revised editions, continuations, rotated pages, split tables, blank OCR layers, and a correct number in an unrelated context.

Answer keys need provenance. The range should reject known misreads and allow only documented rounding. Record who built the key, source pixels, formula, and independent review. A deterministic grader can reliably enforce a bad key.

{
  "question_id": "TQA-041",
  "corpus_hash": "sha256:...",
  "source": {"file":"Treasury-1968.pdf","page":417,"page_hash":"sha256:..."},
  "cell": {"bbox":[812,1320,1040,1398],"row":"Customs receipts",
           "column":"FY 1968"},
  "reads": [{"engine":"ocr-a@pinned","text":"$1,247.6"},
            {"engine":"vision-b@pinned","text":"$1,247.6"}],
  "unit": "USD millions",
  "checks": {"period_match":true,"cross_foot_delta":0.0},
  "answer": 1247.6,
  "review": {"status":"APPROVED","owner":"finance-data-reviewer"}
}

Build the challenge set from real document variation

Start with a document inventory rather than a random sample of questions. For each family, record scan generation, age, page geometry, language, table density, revision pattern, access class, and likely consequence of error. Then deliberately allocate cases across those dimensions. If most questions come from clean annual reports, a high score may say little about the faded schedules and bound-volume scans that consume reviewer time.

Each challenge should carry two labels: the expected evidence route and the permitted decision. The route specifies the authorized file, acceptable page variants, target crop, row and column ancestry, unit, period, transformation, and independent check. The decision is not always a number. It may be ABSTAIN_LOW_IMAGE_QUALITY, ESCALATE_PERIOD_AMBIGUITY, or DENY_UNAUTHORIZED_SOURCE. Scoring these decisions prevents the benchmark from treating relentless guessing as useful behavior.

Separate discovery from reading during diagnosis. First run the retriever against a known file and page; then run extraction on a frozen crop; finally test the entire chain. This creates an actionable confusion matrix. A missing file points to indexing or vocabulary. A correct crop with a wrong value points to OCR or table binding. Correct inputs with a wrong result point to normalization or arithmetic. End-to-end accuracy alone sends all three failures to the same queue.

Define evidence completeness as a schema rather than accepting a citation string. Validate the receipt as data: file hash, page hash, crop coordinates, row and column labels, unit, period, raw reads, chosen read, formula, check results, model and tool versions, access decision, reviewer identity, and timestamps. Make required fields conditional. A direct cell lookup may not need a formula, while a calculated ratio must retain every operand and rounding rule.

Failure modes that a demo misses

Measure wrong confident answers separately from abstentions. TreasuryQA reports that failed attempts usually returned a wrong number. In production that is more dangerous than a visible stop because spreadsheets can treat the value as authoritative. Stop on OCR disagreement, retrieval ambiguity, missing unit, unmatched period, or failed totals.

Do not use a second language model as the only reviewer. Models may share extraction errors. Independent verification changes the evidence path: another OCR engine, direct pixel review, deterministic table total, structured system of record, or a qualified human inspecting the crop and headers.

A 30-day acceptance pilot

1. Days 1-5: select document families, define users, hash the archive, and create 40-80 questions including abstention cases.
2. Days 6-10: two reviewers record file, page, crop, row, column, unit, formula, value, and near misses.
3. Days 11-15: capture retrieval ranks, versions, crops, coordinates, transformations, latency, cost, and access decisions.
4. Days 16-20: inject blur, rotation, duplicate years, revisions, split tables, restricted decoys, and digit-confusion cases.
5. Days 21-25: shadow the manual process without writing to authoritative systems.
6. Days 26-30: approve only bounded task classes with acceptable error severity, evidence completeness, reviewer load, and rollback.

Report results by stage and slice. A 90 percent average can hide zero percent on blurry microfilm. Track candidate recall, page recall, cell accuracy, unit/period accuracy, arithmetic, evidence completeness, abstention precision, wrong-answer severity, reviewer minutes, and cost per accepted answer.

• Every accepted answer has an immutable receipt.
• Every material value has deterministic or independent reperformance.
• Every ambiguity produces escalation, not a guessed number.
• Every tool change triggers canaries.
• Every release names the accountable person and evidence hash.

Release a bounded task, not a universal agent

Predefine the release rule before seeing results. Require complete receipts on every accepted answer, zero unauthorized-source use, zero unreconciled high-materiality errors, a tested abstention path for every defect class, and a maximum reviewer workload. Do not trade a severe miss against many easy successes by averaging them together. A single wrong covenant threshold can outweigh dozens of correct archival lookups.

The first approval should name a narrow operating envelope: specified collections, document families, question types, user roles, materiality bands, OCR versions, and downstream destinations. Everything else remains unapproved. Put an expiration on the decision and rerun canaries when the index, permissions, OCR, model, prompt, schema, or deterministic checker changes. If performance falls outside the envelope, disable release while preserving read-only diagnostic access.

Compare shadow results with the current manual process. Measure total time to an accepted answer, reviewer corrections, unresolved cases, evidence retrieval time, and downstream rework. An agent that generates more candidates but doubles verification effort has not delivered a reliable productivity gain. The useful unit is a verified answer with a replayable receipt, not a generated answer.

Keep the production queue observable

After release, publish a daily control report by document family and risk band. Show accepted, abstained, escalated, and rejected counts; the oldest unresolved item; reviewer overrides; receipt-schema failures; and canary drift. Sample accepted low-risk answers as well as every high-risk case. A queue that reports only completed work can hide rising abstentions, slow reviews, or a retriever that quietly stopped finding one archive series.

Give operators one reversible response for each alert: pause a task class, pin the previous tool cohort, route to manual lookup, or quarantine receipts produced after a change. Recovery should never require trusting the same agent that triggered the alert.

FAQ

What does TreasuryQA prove?

It shows that combined archive search, image-only reading, and numeric extraction remain difficult. It does not prove performance on another archive or security boundary.

Should we require two OCR engines for every page?

No. Route a second read by risk and evidence quality: low confidence, material values, degraded scans, ambiguous characters, or failed totals.

Is a citation to the PDF enough?

No. Identify the edition, page image, crop or cell, headings, unit, period, and transformation. Document-level citation is too coarse.

Can an LLM judge replace the script?

Not for exact figures. Use deterministic grading for numbers and schemas. A model may assess explanation quality but should not be sole authority over correctness.

When is write access appropriate?

After the read-only pilot passes bounded tasks, receipts are complete, wrong answers fail safely, reviewer load is acceptable, and a separate write authorization and rollback design is tested.