Tested against published outcomes
Before anyone relies on a verdict, it helps to see the method reach answers that someone else has already published.
How the test works
Each set is a group of claims whose outcome someone else has published. Every claim was run through the ordinary method with the published answer hidden from it. Only then was each verdict set beside the published outcome.
Case studies
Elon Musk’s self-driving timelines
26 dated predictions. The outcome comes from Wikipedia’s list, CC BY-SA 4.0. They either happened or did not, and news reports say so.
Seoul AI safety commitments
19 promises, one per company: publish a safety framework by the Paris summit. The outcome comes from METR’s list. Frameworks are published documents with dates.
White House AI commitments, 2023
56 promises by 16 companies, scored by Wang et al. 2025, CC BY 4.0. It mostly tests whether evidence exists outside the answer key: the reference scores what companies said they did, and the method asks for the deliverable itself.
The numbers
“Right direction”: take the claims where both the published record and the machine gave a clear answer, kept or not kept. How often did they give the same answer?
“Same verdict” is stricter: how often the machine’s answer matched exactly, including how sure it was. “Resolved false” beside “not met” counts; “mostly false” beside “not met” does not.
“Unresolvable”: claims where the machine found too little dated evidence to decide, so it gave no verdict rather than guess.
| Set | Right direction | Same verdict | Unresolvable |
|---|---|---|---|
| Musk | 22 of 22 | 21 of 26 | 0 |
| Seoul | 14 of 14 | 11 of 18 | 4 |
| White House | 18 of 20 | 13 of 56 | 23 |
Every verdict in these sets is machine-generated and not yet reviewed by a person.