🔍 Read the full analysis: OpenAI’s Agents Learn Inside Your Software: The Ironclad Terms To Review on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI described training GPT-6 Astra on legal, commercial and procurement tasks inside hosted copies of Ironclad’s contract-management software. The model met an average 55% of task criteria, according to OpenAI, while estimated task times were simulated rather than measured customer savings. The work also marks an invitation for other software companies to help train models on difficult workflows.
OpenAI said on October 6 that it trained its frontier model GPT-6 Astra on legal, commercial and procurement tasks inside hosted copies of Ironclad’s contract-management software. The company reported that Astra met an average of 55% of the criteria for 11 selected tasks; the results describe a research exercise, not evidence that the model is ready to handle contract workflows without human review.
Ironclad employees and OpenAI staff who use the product selected 11 tasks, including setting up nondisclosure agreements, creating procurement approval processes and updating a reusable clause to reflect a selected jurisdiction. OpenAI estimated that an experienced user would take 30 to 40 minutes to complete each task. Each task was scored against a rubric of eight to 50 criteria, depending on its complexity.
Ironclad provided hosted copies of its product for the model to practise in. OpenAI said it created synthetic training tasks from publicly filed contracts in the SEC’s EDGAR database, filtered to remove personal information. The company also said the exercise used no OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data.
OpenAI reported that GPT-6 Astra met 55.0% of criteria on average, compared with 41.6% for GPT-5.6 Sol in the high setting. An internal model used in Astra’s development reached 63.7%. On one example task, Astra met about 94% of the criteria. These figures represent rubric criteria satisfied, not the percentage of tasks completed successfully or a general measure of contract accuracy.
The reported task-time comparison was 19.2 minutes for Astra against 37.0 minutes for GPT-5.6 Sol. OpenAI’s post says those times are simulated estimates based on assumed processing and generation speeds. They are not measured customer time savings, and the company cautioned that the estimates apply to these 11 research tasks, not Ironclad workflows as a whole.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Software Firms Are Training Agents
The project gives software companies a route to test agents against realistic, product-specific workflows rather than generic computer tasks. OpenAI says it is inviting a small number of software companies to bring examples of work current agents cannot reliably complete, along with domain experts, a secure test environment and data suitable for research. The approach could help identify where agents lose track of rules and what they need to learn before companies trust them with more complicated operations.
For businesses using contract, finance or customer-record systems, the result is relevant but not a readiness signal. A model that meets only some requirements may still miss a required approval or mishandle a nonstandard term. In those settings, the specific criteria it fails matter more than an average score. Human checks and system controls remain important while performance is incomplete.
The partnership model could also affect software vendors’ competitive position. Agents that work well inside a product may make it more useful, but they may also become the main way customers interact with it. That raises the value of the software’s business rules, data structures, audit records and controls, rather than its screens alone. OpenAI presents the work as a reason a full contracting platform remains necessary; that is the company’s framing, not a conclusion established by the reported benchmark.
Popular right now
Vitamix Propel Series 510 Professional-Grade Blender, 3 Programs, Black
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Vitamix Propel Series 750 Professional-Grade Blender, 4 Programs, Black
As an affiliate, we earn on qualifying purchases.
How the Ironclad Test Was Set Up
The October 6 post described two OpenAI releases, with the Ironclad work receiving less attention than a separate publication of 722 mathematics manuscripts. The Ironclad post is specifically about training and evaluating a model within a software product, not announcing a new agent framework named Ironclad. Ironclad is the contract-management software company whose hosted product was used for the tasks.
OpenAI’s stated aim was to train models to understand business rules, carry out multi-step work in specialized software and check whether finished work meets the original requirements. The chosen tasks spanned legal, commercial and procurement work. The scoring rubric was intended to account for task requirements, but the post’s average score does not show which individual requirements were missed across the full set.
The company’s account also distinguishes the source of the training material from the environment where the agent practised. OpenAI said it generated synthetic tasks from public SEC filings and used hosted Ironclad product copies. It said it excluded non-public Ironclad customer data and OpenAI customer data. The results and data descriptions in this article are based on OpenAI’s published account; no independent evaluation is provided in the supplied material.
What the Reported Scores Do Not Show
The published average does not identify which criteria Astra missed on each task, how often errors involved high-risk requirements, or whether the model’s work would pass review under real customer conditions. The score is a share of rubric criteria met, not a pass rate. OpenAI’s example of a procurement process illustrates the limitation: missing one required Finance, Security or Legal approval could make an otherwise partly correct workflow unsafe to use.
The source material provides no independent assessment, customer deployment results or measured reductions in staff time. It also does not specify the full security and operational arrangements for future partner tests beyond OpenAI’s request for a secure test environment and research-appropriate data. Whether other software companies will participate, what terms they will set and how resulting models will be evaluated remain unclear.
OpenAI’s reported data exclusions are statements by the company. The supplied material does not include outside verification of those practices. Nor does it establish whether improvements on the 11 selected tasks will carry over to other workflows, products or contract types.
Next Steps for Partners and Buyers
OpenAI said it is seeking a small number of software-company partners to identify difficult tasks for future research. Prospective partners are asked to provide a concrete example of work agents cannot reliably complete, access to people with deep knowledge of that work, a secure test environment and data that can safely be used for research. The post does not name additional partners or give a timetable for further results.
Companies considering agent use in contract or other business systems can use the reported limitations to frame their own review. They should ask for task-level results, including which requirements were missed; distinguish simulated estimates from measured performance; and determine where human approval is mandatory. They should also clarify what data an agent can access, how actions are recorded and whether existing approval controls remain in force.
Further reporting will depend on whether OpenAI publishes evaluations beyond the 11 tasks, whether partner companies announce projects and whether any systems move from research tests into customer use. Until then, the published figures show progress on a narrow test set, not a demonstrated replacement for experienced staff or established review procedures.
Key Questions
What is Ironclad in OpenAI’s announcement?
Ironclad is a contract-management software company, not the name of a new OpenAI agent framework. OpenAI said the research used hosted copies of Ironclad’s product for selected workflow tasks.
What does GPT-6 Astra’s 55% score mean?
OpenAI says it is the average share of rubric criteria met across 11 tasks. It does not mean Astra completed 55% of tasks successfully, and the published average does not identify every requirement the model missed.
Did the test prove that agents save customers time?
No. OpenAI described the time figures as simulated estimates based on assumed processing and generation speeds. The post did not report measured time savings for Ironclad customers.
What data did OpenAI say it used?
OpenAI said it created synthetic tasks from public contracts filed in the SEC’s EDGAR database, with personal information filtered out. It said it did not use OpenAI customer data, internal OpenAI contracts or non-public Ironclad customer data.
Can companies rely on agents for contract workflows now?
The reported results do not establish that. OpenAI’s score reflects partial performance on a small research task set, and its post says human oversight still matters. Companies would need task-level testing and controls suited to their own requirements before relying on agents.
Source: ThorstenMeyerAI.com
