Autonomous testing
The Crucible
The generator is tested the way its reports test others. Benchmark questions run through the full pipeline; an independent judge re-researches each answer from its own fresh searches and scores it. Every run is on the record — including the failures.
Verdict accuracy
81%
Dates on claims
100%
Stale sources caught
79%
Bottom line (of 10)
5
Run time
31s
Benchmark cases
“What is the current federal funds rate target range?”
- Oct 18accuracy 88% · dates 100% · stale 100% · bottom line 2/10 · 21sthe report
- Oct 18accuracy 71% · dates 100% · stale 0% · bottom line 3/10 · 23sthe report
- Oct 17accuracy 71% · dates 100% · stale 100% · bottom line 3/10 · 29sthe report
- Oct 17accuracy 50% · dates 100% · stale 0% · bottom line 2/10 · 27sthe report
“What is the 10-year Treasury yield right now?”
- Oct 17accuracy 33% · dates 100% · stale 100% · bottom line 5/10 · 33sthe report
- Oct 17accuracy 100% · dates 100% · stale 100% · bottom line 2/10 · 68sthe report
“Will the Federal Reserve cut rates at the December 2026 FOMC meeting?”
- Oct 18accuracy 95% · dates 100% · stale 100% · bottom line 7/10 · 23sthe report
- Oct 18accuracy 80% · dates 100% · stale 100% · bottom line 6/10 · 23sthe report
- Oct 17accuracy 100% · dates 100% · stale 100% · bottom line 9/10 · 31sthe report
- Oct 17accuracy 100% · dates 100% · stale 70% · bottom line 8/10 · 31sthe report
- Oct 17accuracy 100% · dates 100% · stale 100% · bottom line 8/10 · 36sthe report
The graveyard
Approaches that failed, kept so they are not repeated.
Same benchmark case swings run to run: accuracy 50% then 71%, staleness catch 0% then 100%
Search-result freshness is the generator's weakest link: when the search index returns a months-old snapshot page, the report quotes it as current. Watched via the arena; fix directions are fresher query phrasing and a recency filter on fetched pages. First Crucible pass, Oct 17, 2026.
First Crucible pass, Oct 17, 2026
Judge could not see the what-would-change section and marked it missing
The independent judge scored only the bottom line, stale check and claims. The section existed in every report; the judge prompt now includes it. Harness defects are recorded here like method defects.
First Crucible pass, Oct 17, 2026
Snippet-only extraction stamped stale claims PROVEN
Search-result snippets are evidence, not proof: a report built on snippets alone stamped stale 2025-vintage forecasts PROVEN. Fix: fetch the primary page and re-quote numbers from it before stamping PROVEN.
ORACLE test run, Oct 17, 2026
Verdict pass without a supersession rule kept old facts current-looking
When a newer source contradicted an older one, the ledger left the old fact looking current. Fix: the supersession rule — older claims keep PROVEN for their own date but the note must say what changed and when, and the bottom line rests on the freshest snapshots only.
Generator test runs, Oct 17, 2026
Forecast queries without a this-month query surfaced year-old outlooks
Query plans that omitted an explicit latest-news-this-month query returned 2025 outlooks as the top results. Fix: one planned query is always about the latest news in the current month.
Generator test runs, Oct 17, 2026