How it works
Reliability API reads what an organisation or public figure said in public, checks each checkable statement against the record, and shows its working. This page explains how, in plain words.
What counts as a claim
A claim is a statement by the subject that the public record can show to be right or wrong. There are three kinds, each scored separately.
Factual
About the present or the past, checkable against records.
Measured: the share that hold up.
Promise
A commitment about the subject’s own future action, with a deliverable or a measurable target and a date.
Measured: the share delivered, and whether on time.
Prediction
About future events outside the subject’s control.
Measured: the share that came true.
What is not checked. Opinions, values and vague statements. Claims that are not yet due. Claims that could never be settled because they are too vague, counterfactual or have no time limit. Claims that only the subject’s own internal records could settle.
These are counted and shown, never scored. Each report also says what share of a subject’s statements were checkable at all, so a habit of saying nothing checkable is visible without being counted as error.
How a report is made
The same steps, in the same order, for every subject. Where a rule can decide, a rule decides, not a model.
Find documents
A web search lists documents in which the subject itself speaks: its own writing, interviews, transcripts and news reports that quote it. Every candidate is saved before any is chosen.
Start with a random, representative sample
A quick report starts with a random sample of 8 of the documents found, spread across the years so it represents the whole record, with some places kept for plans and forecasts so that promises and predictions are included. A standard report takes 15, and more can be added later.
Take out the claims, with quotes matched word for word
The subject’s statements are copied exactly. A mechanical check then requires each quote to appear word for word in the stored page. A quote that fails is marked unverified and never shown as verified. Statements by other people are dropped.
Fix the test before any research
For each claim, what would count as true, mostly true, mixed, mostly false or false is written down and time-stamped first. It is never rewritten afterwards.
Research, preferring independent sources
The research looks for dated evidence for and against, preferring sources independent of the subject. Every evidence quote is checked against its page. A claim is never checked against its own source.
A panel of judges decides
A panel of frontier AI models, working from instructions refined through our testing, sees the claim, the fixed test and the verified evidence, and each decides independently. Where they disagree, the verdict is flagged. Not finding a record of something is not taken as proof that it did not happen.
Score
The checked claims of each kind become a score with a range, explained below. The report is stored, so the next request for the same subject returns at once.
The six answers a claim can get
Every checked claim ends with one of these. The more of the circle is filled, the more of the claim held up, and that fill is also what the claim counts for in a score.
- Resolved trueIt happened as said. For a promise: delivered on time.
- Mostly trueIt happened, with a gap the evidence records.
- MixedParts held up and parts did not.
- Mostly falseSomething related happened, not what was said.
- Resolved falseIt did not happen as said, or not by the date.
- UnresolvableThe record is too thin to decide, so there is no guess.
For a promise, whether it was delivered on time is recorded beside the verdict. “Unresolvable” is a deliberate answer: the rule is to prefer it to a guess.
What a score means
A score is the share of checked claims of one kind that held up, from 0 to 100, with a range that shows how sure it is.
A true claim counts 1, mostly true three quarters, mixed a half, mostly false a quarter and false nothing. Unresolvable claims and claims not yet due are counted and shown, but not scored.
Fewer than 5 checked claims of a kind get no number, only “insufficient record”.
Small samples are pulled towards the middle. Every score starts from 50, given the weight of 4 claims, and moves towards the evidence as checked claims add up. So a thin record gives a middling score with a wide range, never an extreme one: 5 claims that all held up give 78, not 100.
A worked example: Ørsted’s promises
- 6 of Ørsted’s promises were checked. Counting each by its verdict, they add up to 4.
- Start from the middle: 2 out of 4. Add the evidence: (2 + 4) out of (4 + 6) is 6 out of 10.
- So the score is 60. With only 6 claims the range is wide: likely between 40 and 79.
score = 100 × (2 + sum of values) ÷ (4 + number checked)
Low confidence: 6 checked claims. The Ørsted case study
There is also an overall score: the three kinds averaged by how many claims each rests on. It is a weaker signal than the scores for each kind.
Confidence
Every score carries a confidence level. A fixed rule sets it, not a model’s judgement.
| Level | Checked claims | Range no wider than |
|---|---|---|
| None, no score shown | Under 5 | Not computed |
| Low | 5 to 14 | Any |
| Medium | 15 to 39 | 30 points |
| High | 40 or more | 15 points |
A score drops to the level below when its range is too wide for its count. The reasons are given in words beside it, for example: “6 resolved claims from 4 documents, 2017 to 2021; 8 documents sampled from 32 candidates found.” They show whether a score is thin because the subject says little or because little was found.
What it will not do
- Cover private people. Public organisations and public figures only, from their public professional work. A person who is not a public figure is covered only with their documented consent.
- Cover politics. No politicians or officials in office. Party-political claims, such as who will win an election or whether a policy should pass, are counted but never scored.
- Look at private life. Nothing about anyone’s family, health, home or location. Such statements are dropped before anything is stored.
- Guess at motives. A verdict says whether a claim held up, never why someone said it. The words are neutral, such as “resolved false”.
- Get past logins or paywalls. Only pages anyone can read, collected within each site’s robots.txt.
Known limits
- Not yet measured against human reviewers. Error rates against human judgement have not been measured. Until they are, every verdict and score is labelled machine-generated and unreviewed.
- Claims can be misread. Where a claim is phrased in a way that can be read more than one way, the system can reach the wrong verdict, even when its judges agree. Agreement shows consistency, not that a verdict is right.
- No allowance for difficulty. A claim almost anyone would get right counts the same as a bold one.
- Every claim counts equally. A trivial claim counts as much as a central one.
- The sample is not the whole subject. A score describes the documents and claims the rules picked, and the report says how they were picked.
- Many claims end unresolvable. Often the record holds something close to the claim but not what the fixed test asks for.
- Re-runs can differ. Research can find different evidence each time, so some verdicts change on a second run, mostly between “unresolvable” and a verdict.
If a report is about you
- See how each claim was checked. Every claim shows the quote and its source, the test fixed before research, the evidence and the reasoning.
- Reply with counter-evidence. You can answer any claim with a statement and a link to your evidence.
- Your reply is shown on the claim, marked “response received, not yet reviewed”, until a person has looked at it.
The full method
The complete published method, including the formula, the rules for each step and the open questions, is versioned. Any change to a rule, a prompt or a constant changes the version.
Current version: 0.1.0+461aa947