Case studiesPublic-interest research
Did AI companies publish the safety frameworks they promised?
The promise
In May 2024, at the AI Seoul Summit, AI companies signed the Frontier AI Safety Commitments, and more joined later. We checked 19 of them. Each promised to publish a safety framework, focused on severe risks, before the next summit in Paris on 10 February 2025.
What we did
We ran each company’s promise through Reliability API. It decided in advance what would count as delivered, searched the public record, and gave a verdict with its evidence. It could not see METR’s list of published frameworks, which records the answer independently.
“…to demonstrate how they have achieved this by publishing a safety framework focused on severe risks by the upcoming AI Summit in France.”
What it found
One row per company. “Resolved true” means a framework focused on severe risks was public by 10 February 2025. Select a company to read its reasoning and every piece of evidence it cites.
-
01.AI Resolved false Later independent reviews list it among signatories with no published framework.
The reasoning
The SaferAI Frontier Risk Management Tracker, an independent source dated after the deadline, names 01.AI among six companies that did not publish their frameworks despite the commitment [E1]. If no framework existed at that later date, none had been published by 2025-02-10. A Frontier Model Forum post from April 2025 counts 12 developers with published frontier AI frameworks, and its list does not include 01.AI [E2]. Under the criteria, a dated third-party review listing 01.AI as not having published resolves the promise false.
Evidence
- E1SaferAI14 July 2026
- E2Frontier Model Forum22 April 2025
-
Amazon Resolved true Its framework page is dated 9 February 2025, the day before the summit.
The reasoning
Amazon's own publication page for its Frontier Model Safety Framework states it was published on 9 February 2025, one day before the summit began, and describes the framework as covering severe risks from frontier models (E1). This page is the published output itself, not just a compliance statement. Amazon's May 2025 blog post (E2) and a January 2026 paper (E3) consistently place publication in early 2025, around the Paris summit. All the evidence comes from Amazon or is tied to it, but the dated framework document directly meets the criteria.
Evidence
- E1Amazon Science9 February 2025
- E2Amazon Science29 May 2025
- E3alphaXiv27 January 2026
-
Anthropic Resolved true Its Responsible Scaling Policy was public from September 2023 and updated in October 2024.
The reasoning
Anthropic's own dated posts show it published a Responsible Scaling Policy in September 2023 focused on catastrophic risks such as bioweapon misuse and autonomous harm, defining ASL standards to manage them (E1, E3). It published an updated RSP on 2024-10-15, described as a risk governance framework for catastrophic risks from frontier AI systems (E2). The criteria state that the RSP or an updated version counts if it was public by 2025-02-10 and covers severe frontier risks. These are subject-authored sources, but they are dated records of the published framework itself, which the rules allow to count.
-
Cohere Resolved true Its framework is dated February 2025; a third-party card dated 1 February suggests it was public in time.
The reasoning
Cohere's own document, the Secure AI Frontier Model Framework, is dated February 2025 and describes itself as the first published version (E1). Its content addresses large-scale societal harms (E2), and an independent summary card describes High and Critical risk tiers covering cyber offense and CBRN uplift (E3), so it meets the severe-risk test. On timing, the third-party card dated 2025-02-01 indicates the framework was public before the 10 February 2025 deadline, and METR later lists Cohere among companies with published frontier safety policies (E4). The card's date is not independently verified, so the on-time finding rests on limited evidence.
The two model checks disagreed on this one, so it is flagged.
Evidence
- E1CohereFebruary 2025
- E2CohereFebruary 2025
- E3kpyakhmi.github.io (third-party safety framework card)1 February 2025
- E4METR (evals.alignment.org)December 2025
-
G42 Resolved true A press release on 6 February 2025 announced its framework.
The reasoning
G42's press release dated 6 February 2025 announces the publication of its Frontier AI Safety Framework (E1). It describes capability thresholds for biological, cybersecurity and autonomy risks, which are severe-risk categories (E2). An independent blog post dated 10 February 2025 separately reports that G42 had released a Frontier AI Safety Framework (E3). This corroborates that the document was public by the deadline, although the framework text itself is not among the evidence.
The two model checks disagreed on this one, so it is flagged.
Evidence
- E1PRNewswire6 February 2025
- E2PRNewswire6 February 2025
- E3Peter Wildeford (blog)10 February 2025
- E4G42undated
-
Google Resolved true An updated framework was published on 4 February 2025 and reported the same day.
The reasoning
Google DeepMind introduced its Frontier Safety Framework in May 2024, describing it as focused on severe risks from model-level capabilities (E1). It published an updated version on 2025-02-04, before the 10 February 2025 deadline, covering severe risks and Critical Capability Levels (E2). Independent outlets Semafor and MediaPost reported that publication on 2025-02-04 (E3, E4), so a qualifying framework was public by the deadline.
Evidence
- E1Google DeepMind17 May 2024
- E2Google DeepMind4 February 2025
- E3Semafor4 February 2025
- E4MediaPost4 February 2025
-
IBM Mostly false It published a governance post on 7 February 2025, but the post does not address severe risks.
The reasoning
On 2025-02-07, before the deadline, IBM published a post describing its AI governance framework and culture as aligned with the Seoul commitments (E2). An independent tracker found that this document summarised existing governance practices without addressing severe AI risks, so it did not qualify as a frontier risk management framework (E1). The Frontier Model Forum's April 2025 list of 12 published frontier AI frameworks also omits IBM (E3). Under the criteria, a deadline publication that lacks the severe-risk focus but is presented as fulfilment resolves mostly false.
Evidence
- E1SaferAI14 July 2026
- E2IBM Newsroom7 February 2025
- E3Frontier Model Forum22 April 2025
-
Inflection AI Unresolvable No dated record shows whether a framework was published by the deadline.
The reasoning
E1 is an undated Inflection page that describes a further technical safety paper as a future milestone and does not itself present a severe-risk safety framework, but it does not show the state of affairs as of the 2025-02-10 deadline. E2 and E3 confirm Inflection was a Seoul signatory with a target of publishing before the February 2025 summit, but they do not say whether Inflection published a framework. No dated tracker or report establishes either publication or its absence by the deadline, so the claim cannot be resolved.
Evidence
- E1Inflection AIundated
- E2CASRAI Editorial Boardundated
- E3CASRAI Editorial Boardundated
-
Magic Mostly true Its AGI Readiness Policy dates from July 2024; the date rests on its own page.
The reasoning
Magic's own page [E1], dated 2 July 2024, presents an AGI Readiness Policy (version 1.0) that addresses catastrophic risks, including dangerous capability evaluations and threat models, well before the 2025-02-10 deadline. The Frontier Model Forum, an independent body, lists Magic's AGI Readiness Framework among published frontier AI frameworks [E2], which corroborates that the document exists but not when it was published. The document is labelled a readiness policy rather than a 'safety framework', a minor labelling difference that the criteria treat as mostly true, and its publication date rests on Magic's own page.
The two model checks disagreed on this one, so it is flagged.
Evidence
- E1Magic2 July 2024
- E2Frontier Model Forum22 April 2025
-
Meta Resolved true Its Frontier AI Framework was published on 3 February 2025.
The reasoning
Meta published its Frontier AI Framework on 2025-02-03, a week before the 2025-02-10 summit start, and presented it as its framework for frontier AI risk focused on cybersecurity and chemical/biological weapons risks (E1). An independent news report (E2) and two third-party trackers (E3, E4) confirm the 3 February publication date, the document's focus on catastrophic risks, and its explicit link to the Frontier AI Safety Commitments. The criteria for true, including the deadline, are met.
Evidence
- E1Meta (about.fb.com)3 February 2025
- E2TechCrunch3 February 2025
- E3ETO AGORA3 February 2025
- E4Policy Window3 February 2025
-
Microsoft Unresolvable Its framework is dated only “February 2025”, which does not settle whether it came before the 10th.
The reasoning
E1 and E2 show that Microsoft published a Frontier Governance Framework. The document traces itself to the May 2024 Frontier AI Safety Commitments and is framed around severe risks: national security and public safety, with CBRN, offensive cyber and autonomy tracked at low to critical risk levels. So its content meets the criteria. However, the document is dated only 'February 2025' (Version 1), and no evidence shows whether it was published before, on, or after the 10 February 2025 deadline, which decides between true, mostly true and mostly false. E3 is a tracker methodology page that does not mention Microsoft, so the timing cannot be settled from the evidence provided.
-
MiniMax Resolved false It joined in February 2025; later lists show no published framework.
The reasoning
E3 reports that MiniMax joined the Frontier AI Safety Commitments on 2025-02-07 and that signatories were expected to share their efforts before the Paris summit, but it does not say MiniMax had published a framework. A dated industry-body post from April 2025 (E4) lists 12 developers with published frontier AI frameworks, and MiniMax is not among them. An independent tracker (E1, E2) lists MiniMax as a Seoul signatory with 'No published framework found', based on checks of the lab's site and external indices. Together these later records show no qualifying framework existed by 2025-02-10, so the claim resolved false.
Evidence
- E1Vorp Labsundated
- E2Vorp Labsundated
- E3MLex7 February 2025
- E4Frontier Model Forum22 April 2025
-
Mistral AI Resolved false Later reports say it had not published a framework.
The reasoning
E3, a later third-party commentary, states that Mistral had not yet published its Frontier AI Safety Framework despite the Seoul 2024 commitment, so none existed by 2025-02-10. E2 independently lists Mistral AI as not having published a safety framework. E1, a watchdog report dated 2025-02-19, names Mistral among signatories that did not publicly fulfil every component of the Seoul commitment, which fits the other sources. Together these post-deadline accounts show no qualifying severe-risk framework was published by the summit.
Evidence
- E1EIN Presswire (press release from The Midas Project)19 February 2025
- E2LongTermWiki29 January 2026
- E3askme.it29 May 2026
-
Nvidia Unresolvable Its risk document is undated, so the evidence does not show when it first appeared.
The reasoning
NVIDIA's Frontier AI Risk Assessment (E1, E2) describes a risk framework for identifying and mitigating frontier AI risks. The document is undated, and its statement 'Applicable from August 2025' gives an effective date, not a first publication date, so it does not show whether an earlier version existed by 11 February 2025. The independent tracker (E3) assesses the framework's content but gives no publication date. The RUSI commentary (E4) does not name NVIDIA, so the evidence does not settle whether the commitment was met on time.
Evidence
- E1NVIDIA (authors Barnaby Simkin, Nikki Pope, Leon Derczynski, Christopher Parisien)undated
- E2NVIDIA (authors Barnaby Simkin, Nikki Pope, Leon Derczynski, Christopher Parisien)undated
- E3SaferAI2 July 2026
- E4RUSI7 March 2025
-
OpenAI Resolved true Its Preparedness Framework was published in December 2023.
The reasoning
OpenAI's pre-summit update dated 2025-02-07 (E1) points to its Preparedness Framework, published in December 2023, as its framework for evaluating and mitigating frontier risks, and lists risk categories such as CBRN, cybersecurity, persuasion and autonomy. An independent source (E3) confirms that Version 1.0 of the framework was published on 18 December 2023, well before the 2025-02-10 deadline. The resolution criteria expressly allow an existing framework such as the Preparedness Framework to qualify if it was published by then, so the promise resolved true and on time.
Evidence
- E1OpenAI7 February 2025
- E2OpenAI7 February 2025Quote not matched to the page
- E3EdgeChat AI17 September 2026
-
Samsung Electronics Mostly false A document dated 10 February 2025 does not address severe risks.
The reasoning
Samsung published a document titled 'AI Safety Framework' dated 2025-02-10 (E1). In the text shown, it centres on on-device consumer AI, fairness, transparency, accountability, data governance and red teaming, with no mention of severe or catastrophic risks, risk thresholds or intolerable-risk levels. An independent third-party review rates Samsung's post-Seoul framework only 'Partial' and says it largely restates existing practices without meaningful new commitments (E2). Under the criteria, a related document by the deadline that does not address severe risks resolves mostly false.
Evidence
- E1Samsung Electronics10 February 2025
- E2Longterm Wiki28 December 2025
-
xAI Mostly true A draft framework appeared around the deadline; the exact date is uncertain.
The reasoning
E1, an independent tracker, says xAI released a draft Risk Management Framework on the same day as the February 2025 Seoul Tracker, and its heading dates that release to 10 February 2025. E3 is xAI's own draft framework, marked DRAFT, and it addresses severe risks including CBRN and cyber misuse and loss of control; under the criteria a public draft counts. The evidence is not fully consistent: E3's filename is dated 2025.02.20 and E1 also mentions a missed February 10 deadline, so the exact publication date is uncertain. The weight of the evidence points to release on or by the deadline, with a minor discrepancy over the date and version.
The two model checks disagreed on this one, so it is flagged.
Evidence
- E1The Midas Project13 May 2025
- E2AI Safety Facts20 February 2025
- E3xAI (data.x.ai)undated
-
Zhipu AI Unresolvable Sources suggest something was published, but give no date.
The reasoning
E1, a later third-party wiki, lists Zhipu AI's post-Seoul publication as 'Limited' with 'minimal public detail', which suggests something was published but gives no date and does not show whether it addresses severe risks. E2 reports that 12 of 16 signatories had published frameworks by December 2024 but does not say whether Zhipu was among them. E3 shows that a later METR analysis did not cover Zhipu, but it is not a dated record of Zhipu's status as of 2025-02-10. Taken together, the evidence neither confirms an on-time qualifying framework nor clearly shows that none existed by the summit.
Evidence
- E1longterm-wiki (third-party wiki)28 December 2025
- E2longterm-wiki (third-party wiki)28 December 2025
- E3Tekai3 April 2026
Every verdict on this page is machine-generated and not yet reviewed by a person.
How that compares with the independent list
Here each verdict sits beside METR’s list, which was hidden from Reliability API while it worked. Read it in steps.
-
For 14 companies, both Reliability API and METR gave a clear answer. They agreed on all 14 about whether the framework was published in time.
The next two steps split those 14.
-
For 11 of those, the verdicts match exactly.
Companies where the verdicts match exactly Company Reliability API METR’s list 01.AI Resolved false Not on the list Amazon Resolved true Published by the deadlineFrontier Model Safety Framework, 10 Feb 2025 Anthropic Resolved true Published by the deadlineResponsible Scaling Policy v1.0, Sep 2023 (v2.0 Oct 2024) Cohere Resolved true Published by the deadlineSecure AI Frontier Model Framework, 7 Feb 2025 G42 Resolved true Published by the deadlineFrontier AI Safety Framework, 6 Feb 2025 Google Resolved true Published by the deadlineGoogle DeepMind Frontier Safety Framework v1.0, May 2024 Meta Resolved true Published by the deadlineFrontier AI Framework v1.0, Feb 2025 MiniMax Resolved false Not on the list Mistral AI Resolved false Not on the list Naver Resolved true Published by the deadlineAI Safety Framework, 7 Aug 2024 OpenAI Resolved true Published by the deadlinePreparedness Framework (Beta), Dec 2023 -
For 3, the direction matches but Reliability API was less certain: it said “mostly”.
Companies where the direction matches Company Reliability API METR’s list IBM Mostly false Not on the list Magic Mostly true Published by the deadlineAGI Readiness Policy, 2 Jul 2024 Samsung Electronics Mostly false Not on the list -
For 4 companies, it said “unresolvable”: it could not find dated evidence either way, so it did not guess.
Microsoft’s framework, for example, is dated only “February 2025”, which does not settle whether it came before the 10th.
Companies Reliability API found unresolvable Company Reliability API METR’s list Inflection AI Unresolvable Not on the list Microsoft Unresolvable Published by the deadlineFrontier Governance Framework v1.0, Feb 2025 Nvidia Unresolvable Published after the deadlineFrontier AI Risk Assessment, 17 Feb 2025, after the summit Zhipu AI Unresolvable Not on the list -
xAI is left out of the comparison: METR records only a draft.
Companies left out of the comparison Company Reliability API METR’s list xAI Mostly true Draft onlyDraft Risk Management Framework, Feb 2025 (file dated 20 Feb 2025); timing against the summit unclear