How the Reliability API measures reliability

Method version 0.1.0+461aa947

Method version 0.1 (the exact version string, including a hash of every prompt, schema, constant and line of pipeline code, is returned by GET /v1/methodology). Every verdict and score is machine-generated and unreviewed. Error rates against human judgement have not yet been measured.

What a score means

A score is an estimate, from 0 to 100, of the share of a subject's checkable public claims of one type that hold up when checked against the record. A score of 70 for factual claims means: of the factual claims we sampled and could check, about 70% held up, after allowing for how few claims we checked.

There are three scores, one per claim type, and an overall score:

Each score comes with an 80% interval, the number of claims it rests on, and a confidence level with the reasons for it. With fewer than 5 checked claims of a type there is no number, only "insufficient record".

Who can be scored

Public figures and organisations only, from their public professional output. A person who is not a public figure is covered only with their documented consent. Serving politicians and officials in post are out of scope, and party-political claims (who wins an election, what a party will do, whether a policy should pass) are counted but never scored. Statements about anyone's private life, family, health, home or location are dropped before anything is stored.

How a report is made

1. Confirm who the subject is. A search-based step checks that the name and URLs point to one subject. A separate decision step, using only that evidence, decides whether the subject may be covered: public figure, not a serving politician. When the evidence is thin, the rule is not to cover.
2. List candidate documents. Web search lists 20 to 40 documents in which the subject itself speaks: its own writing, interviews, transcripts, and news reports quoting it. All candidates are saved before any is chosen. A URL that search did not actually return is never used. The list is stored and reused for 90 days, so repeat reports draw on the same pool.
3. Select documents by rule. No model chooses. Years are taken in a spread order (newest, oldest, then the middle of the largest gap), with a seeded random choice within each year. Some slots are reserved for forward-looking documents (plans, reviews of plans, forecasts), so that promises and predictions reach the sample. A quick report keeps 8 documents, 3 of them forward-looking. A document counts only if it contains statements by the subject.
4. Extract statements and check every quote. One step copies the subject's statements exactly. A mechanical check then requires each quote to appear word for word in the stored page text, after normalising spacing and quotation marks. A quote that fails is kept but marked unverified, never shown as verified. A second step decides who is speaking, the claim type, a self-contained restatement, and any deadline or stated probability. Statements by other people are dropped, as are reported paraphrases.

5. Triage, then sample by rule. Each checkable claim is triaged:

   • resolvable now;
   • not yet due (its deadline is in the future);
   • never resolvable (too vague, counterfactual, or no horizon);
   • checkable only against the subject's own records (internal figures and processes).

   Only resolvable claims are researched. Up to 25 per type are chosen, spread across documents, by a seeded rule. Everything not researched is still counted and shown.

6. Fix the standard before looking. For each sampled claim, resolution criteria are written and stored, with a timestamp, before any research: what the evidence would need to show for true, mostly true, mixed, mostly false and false, and the date to judge at. The criteria are never rewritten afterwards.

7. Gather evidence. A research step searches the web (through Perplexity's search service) for dated evidence, preferring sources independent of the subject. It reads the pages itself, respecting each site's robots.txt; a site that asks for more than 30 seconds between requests is skipped rather than fetched early. It looks for evidence for and against equally. Every evidence quote is checked mechanically against the fetched page. Evidence from the claim's own source document is never used, including copies of it at other addresses and reports that only repeat the claim: a claim cannot be checked against itself.

8. Judge blind. Two judges from different model families see the claim, the frozen criteria and only the verified evidence: judge A is Claude Opus 5.5, judge B is OpenAI's GPT 6.1 Sol. They get the same instructions and the same input, and each decides separately, without seeing the other's answer. Judge A's verdict is the claim's verdict. When judge B reaches a different verdict, the claim is flagged "judges disagree" and both verdicts are shown; no third judge is called. With no verified evidence the claim is "unresolvable" without a judge. Not finding a record of something is not taken as proof that it did not happen: a "did not happen" verdict needs positive evidence, such as a report of a delay or cancellation, or a dated account of the situation after the deadline. The rule is to prefer "unresolvable" to a guess.

   What a subject says about itself. For a factual claim about the subject's own figures, events or appointments (results, money raised or granted, who holds a post), the subject's official records count as evidence: audited or published accounts, regulatory or exchange filings, and dated official announcements. For a promise, the subject saying it delivered is not enough; the deliverable itself, or an independent source, has to show it. When the only support is the subject's ordinary statements (an interview, a blog post, a marketing page), the claim is "unresolvable" and marked "self-reported only". Every verdict records what it rests on: an independent source, only the subject's official records, only its ordinary statements, or nothing; reports show this beside the verdict.

9. Score (next section) and store the report, which is served for 90 days. After that it is served marked stale while a refresh runs. A subject can send a reply or correction to any claim; it is shown on the claim as "response received, not yet reviewed".

How the score is calculated

Each checked claim gets a value:

Unresolvable, not-yet-due, own-records-only and unsampled claims are counted, not scored.

The score starts from a prior and moves towards the evidence as checked claims accumulate:

score = 100 × (α + sum of values) / (α + β + n)

Here n is the number of checked claims of that type. α and β encode the prior: a prior mean of 50 worth the evidence of 4 claims (α = β = 2). So a thin record gives a middling score with a wide interval, never an extreme one.

• With 5 claims all true, the score is 78, not 100.
• With 40 claims all true, it is 95.

The prior strength was fixed before any results were seen. The prior mean will become the average for comparable subjects once about 20 subjects are in the database; until then it is 50.

The 80% interval is the middle 80% of the posterior Beta(α + sum of values, β + n − sum of values). The overall score's interval comes from sampling the type posteriors with a fixed seed, so the same inputs always give the same interval.

Brier score. For predictions made with a stated probability ("70% likely"), the Brier score is the mean of (stated probability − outcome)², where the outcome is the verdict's value. Lower is better: 0 is perfect, and 0.25 is what always saying 50% would score. Bounds such as "more than 10% likely" are left out.

Checkable share. Every report also gives the share of extracted statements that are checkable at all. A low share means the subject mostly says things that cannot be checked. That is not counted as error, but it is made visible.

Confidence

The confidence level follows a fixed rule, not a model's judgement:

A score drops to the next level down when its interval is too wide for its count. The reasons are given in plain words, for example "13 resolved claims from 6 documents, 2013 to 2025; 8 documents sampled from 36 candidates found". They show whether a score is thin because the subject says little or because little was found.

What a score does not tell you

• No adjustment for difficulty. A claim almost everyone got right counts the same as a bold one.
• No weighting by importance. A trivial claim counts as much as a central one.
• The sample is not the subject. The score describes the documents and claims the rules picked, and the report says how they were picked.
• Many sampled claims end unresolvable (38% to 46% in the 10 October reports). Counting the subject's official records as evidence (13 October) brought this down only a little (Ørsted 21 to 19 of 50, GiveWell 14 to 12 of 37): most of the rest had the subject's records but not the exact figure the criteria ask for, so deeper research, not a looser rule, is what would settle them.
• Some verdicts rest only on the subject's own records. For GiveWell, 19 of 25 resolved claims did. Such a verdict says the claim matches the subject's own official record, which is weaker than independent confirmation; the report says so beside each one.
• Verdicts lean positive. In development, 86% of resolved claims came out true or mostly true. Claims with confirming evidence are easier to resolve, and the judges may be lenient on "mostly true". Human labels will show how much of this is real.
• About one verdict in four changes on a re-run of the same claim, because research finds different evidence each time (on the current method, 10 Oct: 38 of 50 claims kept their verdict for Ørsted, 27 of 37 for GiveWell; most changes are between "unresolvable" and a verdict). Overall scores moved 3.7 and 0.2 points; a type score resting on 6 claims moved up to 7.5 points.
• Two model judges agreeing is consistency, not accuracy. In development, two Opus judges agreed on 92% to 94% of claims, but they were the same model. With judge B from another family, the judges gave the same verdict on 43 of 59 known-answer claims. Nearly every difference was the GPT judge saying "unresolvable" where the Opus judge gave a verdict. The verdict is judge A's, so a flagged claim rests on one model's reading. Both judges see the same evidence.

Validation (not yet done)

Before the method is used in anyone's decisions, its error rates are to be measured against human labels on a gold set of about 300 claims from 20 subjects (SPEC section 8). Proposed gates:

Until then, GET /v1/methodology reports error rates as "not yet measured" and gives judge agreement only as a consistency figure.

Models

Each step uses the cheapest model that does the job well: - Claude Haiku 5.5 for simple extraction and parsing; - Claude Sonnet 5.5 for harder reading and search; - Claude Opus 5.5 for decisions and for interpreting evidence, including judge A; - OpenAI GPT 6.1 Sol for judge B, so that the two blind judges come from different model families.

The current assignment of model to step is returned by GET /v1/methodology and documented in docs/PIPELINE.md.

Open questions

• How far to trust a subject's own records (known issue). The subject's official records count as evidence for its own figures and events, and a verdict resting only on them is marked as such. But the less reliable a subject is, the less its own records should be trusted, and for many facts (internal numbers, appointments) the subject's records are the only source of truth. Weighting self-records by the subject's own track record would make the score depend on itself, so it is not done; how to handle this is open.

• What counts as a promise. Since 13 October a promise names a deliverable or a measurable target with a date ("cut emissions 50% by 2030"), even when worded as an aim; plans and intentions without either are not checked. On three subjects' stored statements this changed little: dated targets were already counted, vague intentions now drop out, and few subjects reach 5 checked promises. Whether company guidance ("we expect EBITDA above DKK 28 billion") is a promise or a prediction is unsettled; the classifier now mostly calls it a prediction.

• The second judge. Judge B's disagreements with judge A are mostly abstentions, so they are flagged rather than adjudicated (tested against an Opus and a Gemini tie-breaker, neither of which did better: docs/ADJUDICATOR_RESULTS.md). Whether a different judge B, for example Gemini, would add more signal is open.