🔍 Read the full analysis: We Tested Weight-Space Programming With A Local Model And AI Review on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment edited the weights of the local Ornith 1.5 35B-A3B model to reduce looping. In a same-setup comparison, completed answers rose from 4 of 15 to 12 of 15, but the legal score stayed at zero; recommended sampling settings without an edit produced 13 completions.
A home experiment reported November 5 found that a targeted edit to the weights of the local Ornith 1.5 35B-A3B model helped it finish more answers, but did not improve its score on a legal benchmark. In a same-setup comparison, completions rose from 4 of 15 to 12 of 15, while the legal score stayed at zero; a run using the model’s recommended sampling settings without editing its weights finished 13 of 15.
The experiment ran from Wednesday evening to Thursday morning and was inspired by a paper on “weight-space programming” from four MIT CSAIL researchers. The report says this home test did not reproduce the MIT method: a Claude Opus 5.5 session wrote code that ran the model, measured a direction associated with looping, and removed that direction at half strength from the output weights of 27 of the model’s 40 layers. The researchers did not train the model or add legal data, and an AI did not read the weights as text.
The benchmark asked the model 15 questions about German, U.S. and EU law, without internet access or tools. It checked citations against primary sources. The edited answers met 7 of 75 checked criteria, up from 3, but the overall legal score remained zero. The report says the 12 completed answers contained 62 wrong or invented citations, an average of 5.2 per answer. The test was one run per setting, so the results are observations from a small experiment, not a stable estimate of performance.
The report also says GPT-6 Astra, running in Codex, reviewed the code, figures and claims and found real faults. The author describes that as one independent review, not a second experiment. Claude was used both to carry out the edit and by agents from the same model family to grade answers, a potential conflict the report disclosed.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completions, No Legal Accuracy
The result illustrates a distinction that matters when AI systems are used for research or other high-stakes work: finishing an answer is not the same as answering correctly. Reducing repeated text can make a model more usable, but it does not establish that the model knows the subject or can support its claims with reliable sources. Here, the completion count rose while the legal score did not, and the report identified numerous incorrect or invented citations.
The comparison with recommended settings also limits the case for editing weights. Without a weight change, adjusting sampling to the model card’s recommended settings led the original model to finish 13 of 15 answers in the reported test. That is one run, not proof that the setting will always work better. Still, it shows why any claimed gain from an edit needs to be tested against simpler adjustments under the same conditions.
For readers considering local models, the takeaway is narrow: a technical intervention may change how a model behaves without improving the truth or quality of its output. The experiment is not evidence that the model is suitable for legal work, and the report expressly says it is not legal advice.
Popular right now
De'Longhi Dinamica Plus Fully Automatic Espresso Machine
De'Longhi Rivelia Fully Automatic Espresso Machine with Milk Frother, White
Breville BES870XL Barista Express Espresso Machine, Brushed Stainless Steel
As an affiliate, we earn on qualifying purchases.
The MIT Method and This Test
The MIT researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—described an approach they call weight-space programming, in which a general-purpose model examines and edits another model’s weights without a conventional training loop. Their reported Qwen case stopped a looping behavior and raised a mathematics score from 7.9% to 48.1%, according to the source article. The researchers’ stated limit was that an edit might remove an obstacle hiding an ability, but could not add an ability the model lacked.
The home test asked whether that distinction applied to a local model assessed on law. Its author says Ornith was selected because it showed looping: five of its 15 benchmark answers did not finish within a 62,000-token output limit, and the late portions of those transcripts contained few distinct lines. The model’s publisher describes it as a coding- and agent-focused mixture-of-experts model, rather than a legal specialist. Its original benchmark score was reported as 1 out of 100.
The author says other local models did not provide a comparable test: some lacked legal knowledge without looping, while another looping model was too large for the test machine and used a different token budget. The reported edit was therefore a focused, single-model test, not a broad comparison across local AI systems.
“The edit repairs finishing, not legal knowledge.”
— Thorsten Meyer, the experiment’s author
Limits of the One-Run Results
The findings are preliminary. The report describes one run per setting, a small development set and a loop detector that changed during the experiment. It also says a change in the local setup alone shifted the unedited model from 10 completed answers to 4, making the comparison sensitive to operating conditions. The benchmark includes 15 questions, but only 10 tasks are published; five remain hidden.
The graders were Claude agents from the same model family as the session that conducted the edit. The report says GPT-6 Astra reviewed the work and found faults, but does not provide enough detail here to assess the review’s full scope or how each fault affected the final figures. It is also unclear from the supplied account whether the results have been replicated on other models, question sets or hardware setups. The experiment does not establish that directional ablation caused a repeatable improvement in completion, much less legal accuracy.
Replication and Benchmark Updates
The immediate next step would be to repeat the comparison under a fixed setup, with the same questions, token limits and sampling conditions, and report results across multiple runs. Further tests could compare the weight edit directly with recommended sampling settings and other ways to control looping, while keeping grading independent and checking citations against primary legal sources.
The author says the new runs are not reflected on the Lexassay website, which continues to show the original Ornith result. No replication or planned publication date is specified in the source material. Until further tests are available, the reported result should be treated as a limited demonstration that the edit changed completion behavior—not as evidence of improved legal capability.
Key Questions
What did the experiment change?
It removed a measured loop-associated direction at half strength from the output weights of 27 of Ornith’s 40 layers. The report says the change used directional ablation and did not involve training or adding legal data.
Did the edit improve the model’s legal answers?
Not according to the reported score. The edited model completed more answers, but its legal score remained zero. The 12 completed answers contained 62 wrong or invented citations, according to the report.
How did the edited model compare with recommended settings?
In the reported comparison, the edit produced 12 completions out of 15. The original model using its recommended sampling settings, with no weight edit, produced 13 out of 15. Each setting was tested once, so this is not a reliable estimate of typical performance.
Was this the same method used by the MIT researchers?
No. The article says the MIT team’s Qwen demonstration involved a model reasoning about another model’s weights directly. In the home test, Claude Opus 5.5 wrote code that ran Ornith, measured internal signals and applied a directional ablation.
Is Ornith suitable for legal advice based on this test?
No such conclusion is supported. The benchmark score stayed at zero, and the source explicitly says the experiment is not legal advice. More completed answers do not establish that citations or legal analysis are accurate.
Source: ThorstenMeyerAI.com
