LLMs learn to read and write network weights

MIT CSAIL researchers describe an unexpected ability of large language models: to read, correct, and even write neural network weights. The idea opens uses in interpretability and optimization but remains fragile outside the lab.

MIT CSAIL researchers describe a new capability in state-of-the-art LLMs, called Weight-Space Programming: reading and writing neural network weights directly, without gradient descent. According to their paper published on November 2, 2026, GPT-6 Astra can identify a model’s task from its weights alone, fix a faulty network, and in some cases write one from scratch. For interpretability and optimization, this is significant as it shifts part of the algorithm’s work to the model’s own reasoning.

Quand un LLM lit les poids comme un humain lit un indice

The simplest result to understand is also the most puzzling. In several experiments, Astra receives raw network weights, sometimes without input or output layers, yet still recognizes what it has learned to do. On small tasks, it identifies the correct task from patterns in the layers without running the network. The authors describe this as reading the geometric structure of weights rather than classical behavior computation.

The most telling test involves three game networks trained on Breakout, Pong, Pac-Man, and Freeway. The model receives the 17,253 weights of a policy network, with the same architecture for each game, then must guess which game it was trained on. Astra hits the mark on all four networks. The same logic applies to diffusion models: from weights alone, it recognizes circle, triangle, and square generators, even without ever seeing a forward pass.

The authors emphasize a key point: the model does not just read the last layers. On three small language models trained on Python, Japanese romaji, and English, it identifies the training language having access only to central blocks. It also finds the copying heads, attention heads that paste pieces of previously seen text. This detail gives the story a hint of “machines have read the manual and understood it.”

Corriger un réseau sans entraînement, la partie la plus intrigante

The most useful demonstration for industry is not just reading. Astra also modifies weights to repair faulty behavior. In a Kinetix environment, an agent supposed to reverse a car before crossing a ramp consistently fails by going straight into a block. With weights, the scene, and a recorded trajectory, Astra spots a local minimum and rewrites a small fraction of the network to introduce a new decision path. The result is one data point: 1,024 successful episodes out of 1,024, after modifying 3.5% of weights.

The most spectacular case concerns Qwen3.5-0.8B. Researchers ask it to improve its mathematical reasoning without gradients, using a simple Python environment that lets it inspect weights and run a few forward passes. Astra spots a specific problem: the model enters reasoning loops, a flaw also noted in its model card. By removing a direction carried by the MLPs of layers 15 and 19, it modifies 131,072 weights, or 0.016% of the total, raising MATH-500 from 7.9% to 48.1%. GSM8K rises from 13.8% to 67.1%, and IFEval climbs from 21.0% to 62.3%.

This correction has a clear limit, and it is important. When the starting model does not hide a blocked capacity but truly lacks skill, editing is no longer enough. On Gemma 3 270M, smaller and less prone to loops, Astra barely improves math. In other words, removing an obstacle is easier than creating an ability. The network does not accept cosmetic surgery.

Écrire un réseau de zéro, puis viser le Go

Beyond correction, the authors also test generating weights from a natural language instruction. On MinAtar, Astra must write networks for Breakout, Seaquest, and Asterix without training data, gradients, or prior execution. The models it produces outperform PPO trained for 100 million steps in all three games. Against PPO trained for 500 million steps, they do better on Breakout, come close on Asterix, and still reach 60% of the score on Seaquest.

On Go, the challenge rises. The target network has 1.48 million weights, far too many to fit in an LLM context raw. Astra then works with a Python sandbox, can reread tensors and perform some forward passes, but cannot run training or tree search. Even with zero forward passes, it wins more than half the games against two small PPO networks. With 50 passes, it even surpasses the equally sized network trained on 20 million games. The authors describe a layered method: marking stones, tracking groups in danger, then converting that into a move score. All without calling a Go library. Not bad for a model supposed to only “predict text.”

Ce que cela change pour l’interprétabilité et l’optimisation

The work does not prove that an LLM can read any network at industrial scale. The authors acknowledge limits of context, harness, and generalization. But the direction is clear: generalist models can already serve as an interface between natural language and weight space. For auditing, this could help detect hidden behaviors without systematically running full tests. For optimization, it opens the door to targeted corrections, less costly than full retraining.

The downside is known: the more a method promises shortcuts, the more it must be verified that it does not produce convincing but false errors. The authors say they want to build a dedicated test bench for this capability. That will probably be the true test, as the topic touches mechanistic interpretability, model safety, and how models are updated in production.

Key points

  • Published November 2, 2026 by MIT CSAIL.
  • Astra reads 17,253 weights to identify a game.
  • Qwen3.5-0.8B: 7.9% to 48.1% on MATH-500.
  • 131,072 weights modified, or 0.016% of the model.
  • MinAtar: better scores than PPO trained for 100 million steps.
  • Go: 1.48 million weights edited without training.

By the numbers

  • 3.5% — weights modified to repair the Kinetix agent.
  • 48.1% — MATH-500 score of Qwen3.5-0.8B after editing.
  • 62.3% — IFEval score after editing on Qwen3.5-0.8B.
  • 1,024 episodes — full success on the Kinetix test.
  • 1.48 million — Go network weights edited by Astra.

To read