🔍 Read the full analysis: What Did GPT-6 Astra Catch In Claude’s Model Edit That Green Tests Didn’t? on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A report published Nov. 5 says OpenAI’s GPT-6 Astra, working in Codex, independently reviewed an experiment in which Claude Opus 5.5 edited the weights of a local model. Astra found reproducible test-rig and weight-conversion problems that green unit tests had missed, and challenged several claims in the write-up. The reported review was not a second model-improvement experiment, and some validation and hardware checks remain incomplete.
A report published Nov. 5 says OpenAI’s GPT-6 Astra, running in Codex, found reproducible flaws in the test rig and weight-conversion process for a Claude-led edit of a local language model—problems that had not been caught by green unit tests. The reviewer did not make a competing model edit; it checked the work, reproduced faults and prompted the authors to qualify or retract some claims.
The experiment, conducted by a Claude Opus 5.5 session, attempted to improve legal-answer behavior in the local model Ornith 1.5 by weakening a “loop direction” in its weights, without additional training. The report says the model completed 12 of 15 legal answers after the edit, compared with four beforehand, while its legal score remained zero. The account does not provide enough detail here to independently assess that benchmark result.
GPT-6 Astra was assigned a separate role: review the implementation without changing the main test rig or inspecting hidden benchmark tasks. Its first review, on Nov. 4, identified four reproducible issues. Resuming a run could reuse an answer from a different model or sampling setup; a nonexistent task ID could yield a successful run with no evaluations; a comma conflict broke the actual command line despite passing parser tests; and converting weights changed values even when the edit strength was set to zero.
The author says the Claude session fixed the four issues and added tests that failed on the old code. In a later review round, all 12 author tests passed, but four of Astra’s six provenance checks failed. Those checks concerned whether the recorded results and files could be reliably tied to the stated experiment. The report also says a separate format-conversion measurement changed 915,360 of 1,048,576 values in one matrix, with a 0.51% measured shift. That figure describes conversion, not legal performance or a general error rate.
Infographic 01 · The two review rounds
Green tests, real faults
GPT-6 Astra (OpenAI, in Codex) reviewed a Claude Opus 5.5 session's model-edit experiment while it was still being built. What it found, and what happened next.
Round 1 · 4 Nov · four faults
- Resuming a run could replace a different experimentAnswers were skipped even after model, settings or edit changed; in the reviewer’s test: 0 new inferences, old answer kept! Partly fixed
- A mistyped task ID gave a “successful” empty runZero evaluations, no error✓ Fixed
- Unit tests green, real command brokenCommas in the sampling syntax clashed with the model list; 3 parser tests passed, the real dry run failed✓ Fixed
- Conversion changed weights at edit strength zeroRe-quantising one real matrix: 0.84% change, 105,315 of 1,048,576 values differed! Partly fixed
Round 2 · 4 Nov, evening · the fixes re-checked
- Old result rows got a new historyAccepted after the requested seed was changed to 999✓ Solved another way
- Loop-detector settings not fingerprintedChanging them left the configuration “identical”✓ Solved another way
- Model configuration and chat template invisibleOnly file names, sizes and modification times were compared✓ Solved another way
- bf16 storage still changes the computationThe zero-strength test compared a conversion with itself→ Control added
✓ Fixed with a test that fails on the old code · ! Partly fixed in round 1, finished in round 2 · ✓ Solved another way: resuming was removed entirely, and every run records fingerprints of configuration, tokenizer and code · → Control added: a conversion-only copy with no edit. The reviewer changed none of the main code and read none of the hidden benchmark tasks.
Sources: our experiment records, 4–5 Nov 2026: reviews of 4 Nov and final explanation of 5 Nov by GPT-6 Astra in Codex (OpenAI); results report of the executing Claude Opus 5.5 session (Anthropic).
Powered by Thorsten Meyer AI
Why Independent Checks Found More
The report illustrates a practical gap between unit-test success and confidence that an experiment measured what its authors intended. Tests can verify individual parser functions while missing a failure in the real command line. A run can also finish without errors while quietly reusing stale answers or evaluating no tasks at all. Those faults could make results appear more reliable than the underlying process warrants.
The weight-conversion issue bears directly on interpreting the model edit. If values change during conversion even at zero edit strength, then a measured difference after editing may not come solely from the intended intervention. The author says the review led to changes in both code and wording, including replacing an assertion of exact identity at zero strength and narrowing a causal claim based on a single control. The account is a case study in review methodology, not evidence that the edited model has improved legal accuracy.
Popular right now
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
As an affiliate, we earn on qualifying purchases.
How the Two AI Roles Split
The source account says two AI sessions began work on the experiment on Nov. 4. After it became clear they were addressing the same project, the roles were divided. Claude retained the plan, model edit, benchmark runs and blind grading by three Claude agents. GPT-6 Astra was asked to review and validate that implementation rather than produce another edited model.
The review was conducted under stated limits: Astra was not to alter the main implementation or inspect hidden prompts, topics, answers or reference solutions. It used separate check programs and reproducible examples; the Claude session made the fixes. The report says Astra’s preparatory files were set aside after the role split and did not produce a second edited model. This distinction matters because the review’s findings concern the experiment’s process and claims, not a head-to-head comparison of two model edits.
“Review and validate the other session’s implementation.”
— The report’s description of GPT-6 Astra’s assignment
What the Review Did Not Establish
The report describes an author’s account of the experiment and review; the supplied material does not include the underlying code, complete logs or independent replication. It is therefore not possible from this account alone to verify the reported legal-answer change or determine how much the weight conversion affected the benchmark results. The stated legal score remained zero, despite the increase in completed answers, so completion should not be read as improved legal correctness.
The author also lists limits to Astra’s review: it did not rerun every test during its final documentation step, was not a fourth grader, and measured conversion on a CPU rather than the production GPU path. The report does not fully explain the six provenance checks or the scope of the four failures. It also does not establish that the findings generalize to other models, editing methods or evaluation systems.
Further Validation of the Edit
The account says the Claude session addressed the four first-round faults, but the material provided does not state a schedule for another review or a complete rerun of the final experiment. The next useful checks would be to rerun the full benchmark with the corrected rig, verify provenance across all results, and compare conversion behavior on the production GPU path. Until those checks are documented, the reported gains in answer completion remain a result from this experiment rather than a validated claim about legal capability.
Key Questions
What did GPT-6 Astra review?
It reviewed the implementation and test process for a Claude-led edit of Ornith 1.5. The report says Astra did not create a competing edited model.
What problems did it find?
The first review found that resumed runs could reuse stale answers, unknown task IDs could produce empty successful runs, the real command line failed despite green parser tests, and weight conversion altered values even at zero edit strength.
Did the model become better at law?
The report says it completed 12 of 15 legal answers after the edit, compared with four before, while its legal score stayed at zero. Completion counts alone do not establish improved legal accuracy.
Were all of Astra’s checks passed after fixes?
No. The report says all 12 author tests passed in the second review round, while four of Astra’s six provenance checks failed. It does not provide the full detail of those checks in the supplied material.
What remains to be verified?
The report says not every test was rerun in the final documentation step and that conversion was measured on a CPU, not the production GPU path. A complete rerun and further provenance checks are not documented here.
Source: ThorstenMeyerAI.com
