A language model learns to rewrite its own weights
Researchers show that a 27B parameter model can modify its weights without traditional training. The effect persists after editing and also improves a damaged copy.
A paper authored by researchers affiliated with UIUC, Carnegie Mellon, Rutgers, Princeton, and Stanford shows that a language model can modify its own weights to improve performance, without gradients or a new training phase. The experiment, conducted on Qwen3.5 27B and dated November 16, 2026 in the provided source, is notable for one simple reason: it blurs the line between inference and learning. The gain achieved remains encoded in the weights, so it does not disappear once the session ends.
Quand le modèle se corrige lui-même, ce n’est plus juste un assistant bavard
The authors call this process self-rewriting: the model observes itself, diagnoses its errors, then writes its own weight modifications. No gradient, no optimizer, no traditional training pipeline. The system only has its weights, corrected examples, its own computation passes, and very limited Python tooling. In practice, this is a form of internal maintenance, somewhat like software tightening its own bolts, which avoids sending the machine back to the workshop.
The clearest result concerns the 27B model: on 490 never-before-seen problems, an edit of its weights improves accuracy by 6.6 points. The authors also indicate that the method holds over several successive sessions, with progress continuing over the first eight. Most importantly, the modification costs nothing per response, unlike some computation methods at inference time.
Une réparation ciblée, puis une amélioration durable
The researchers also tested the method on a deliberately damaged copy of the model. They corrupted eight lines of an MLP output matrix, then asked the model to find the fault without seeing the healthy weights. The 27B identifies the faulty matrix, applies a correction, and raises its copy’s accuracy from 7% to 70% on six image benchmarks it had never seen before. This figure corresponds to 83% of the intact model’s performance on these tests.
In the case of self-editing, the model does not rely on intuition alone. It compares hidden states, detects a direction that separates completed answers from those interrupted due to lack of tokens, then removes 5% of this direction on two matrices in layer 8. The authors note that each intermediate test favors this edited version, and that the improvement is found on the preselected evaluation. The method is therefore less magical than it seems: it resembles a diagnosis guided by measurements rather than a roll of the dice.
Les petits modèles bricolent, le 27B mesure
The contrast between model sizes is central. Within the Qwen3.5 family, the 2B, 4B, and 9B versions fail to self-correct properly. They test, fumble, add noise, or try random changes. The 27B, however, ends up measuring a precise direction in its internal representations and uses it methodically. The authors see this as a capacity threshold: beyond a certain size, the model no longer just imitates a repair, it reasons it.
The protocol remains strict. Each session has a limited budget of passes before ending, and the final edit is accepted only if internal tests support it. On the damaged copy, the final correction mainly targets the corrupted matrix in layer 30, then extends to other output matrices in layers 0 to 10. The detail matters, as it shows the algorithm does not "rebuild" everything: it identifies a breaking point and treats it locally.
Ce que cela change pour les modèles de demain
The significance goes beyond the lab case. If a model can correct a local defect, then retain the correction in its weights, this opens the door to more autonomous systems in operation, capable of adjusting after an incident or partial degradation. There remains an obvious limitation: the experiment is conducted under strong constraints, with closed tooling and tight safeguards. Nothing guarantees that a real deployment would tolerate such freedom to write its own parameters.
For research, however, the signal is clear. A language model is no longer necessarily doomed to choose between two worlds, that of training and that of usage. In this study, it learns to do a bit of both. And without raising its hand to ask for help.
Key points
- Qwen3.5 27B self-modifies without traditional training.
- 6.6 point gain on 490 new problems.
- Eight self-editing sessions show continuous progress.
- A corrupted copy goes from 7% to 70%.
- Versions 2B, 4B, and 9B cannot do it.
- 27B works with 20 passes and 200 Python executions.
By the numbers
- 490 problems — new test of self-editing, according to the provided article.
- 8 sessions — initial sessions where accuracy improves each time.
- 27B parameters — size of the model that succeeds in self-rewriting.
- 7% to 70% — performance of a damaged copy before/after repair.
- 8 lines — corrupted portion of the MLP output matrix.