What did Astra's weight edit actually program?

What Astra did

In Sparks of Weight-Space Programming, Takehi, Teoh, Zhang and Khattab show GPT-6 Astra editing another network's weights directly, with no gradient descent. Their most striking example is a small reasoning model, Qwen3.5-0.8B. Astra picked 64 neurons in each of layers 15 and 19. For each one, it took the part of the neuron's output along a single direction in the hidden state, call it u, flipped its sign and made it five times larger. Nothing else changed, about 0.016% of the weights, and the model's MATH-500 score rose from 7.9% to 48.1%.

Because we know exactly what changed, this edit is an interpretability test with an answer key: every reading can be checked against the real change. We rebuilt the edited model from the released code, reproduced the result (7.6% to 51.9% under the same protocol), and asked two questions. What did the edit change in the model's behaviour? And how?

The model can solve the problems. It cannot stop.

Give the unedited model a grade-school math problem with its thinking mode on, and most of the time it never answers. In our screen it finishes only 12% of GSM8K problems within 16,384 tokens. When it does finish, it is right 94% of the time. The rest of the time it falls into a loop, copying its own earlier text word for word until the budget runs out.

So did the edit make the model smarter, or just help it get an answer out? Simple controls answer this. Force the model to stop thinking at 8k tokens, and most of the gain returns. Turn thinking off, a switch every user has, and the unedited model scores 56% on GSM8K and 51% on MATH-500: as good as the edit on GSM8K, and better on MATH-500. With thinking off, adding the edit does nothing.

Problem by problem, the edit rescues questions the model could already do. Of the problems the edited model solves, the unedited model with thinking off fails both of its tries on only about one in ten, the same rate you get from sampling noise alone. The original post is careful here too: it credits the gain to fixing loops, not to new mathematical ability.

Why has training not removed such an obvious bug? Pretraining rewards copying: in human text, a passage that repeated once tends to repeat again, and training scores the model only on human text, never on its own runaway output. By that score a loop looks like excellent prediction, and the unedited model does predict its own looped tokens almost perfectly. Long-thinking training then rewards re-checking, without teaching a small model when to stop. Loops like these are common in small reasoning models, and worse in models distilled from larger teachers (Pipis et al.).

The model knows when it is right, and doubts anyway

Why can't a model that solves the problem stop? We looked at the moment right after it states its first answer. At that point its hidden state already separates right answers from wrong ones, as has been found in larger reasoning models (Zhang et al.). The position along Astra's direction u alone does this better than the model's own choice of whether to doubt.

The model does not use what it knows. With thinking on, it doubts almost every first answer, right or wrong: "Wait, let me re-read the question." Give it the same text with thinking off, and it doubts a correct answer about 70 times less often. The thinking mode, not the model's knowledge, is what makes it second-guess correct work.

Astra's edit does not fix this. It pushes the model away from doubt everywhere, and it pushes hardest at wrong answers, exactly where the doubt was warranted. When the edited model still fails, seven times in ten it is still doubting: re-reading the wording, or re-checking an answer that is already right. It just no longer repeats itself word for word. A better edit would cut the copying and keep the model's sense of when it is wrong.

Inside the edit, tool by tool

How does a change to 128 neurons break a loop? We looked with ten interpretability tools. Here is what the main ones saw, in the order the effect travels through the model.

Neurons. Astra picked well. Rank every neuron by how much more it writes along u in loops than in normal text, and 44 of Astra's 64 in layer 15 are in our top 64; chance is about one. A language model reading their strongest activations describes about half of them as going back to re-read the problem, quote it or check the work. Few fire inside the loops themselves.

Lenses. A lens translates a direction in the hidden state into the next words it favours. The Jacobian lens reads +u as hedging: 还是 ("still"), 其实是 ("actually it is"), Actually, is it? (the model often thinks partly in Chinese). It reads −u as stating a result: results, outcomes, computations. So u runs from "doubt and keep going" to "state the result", and the edit pushes towards the result. The simpler logit lens barely sees this, and in the middle layers it gets the edit backwards: it shows the edit raising the token about to be repeated, while the model's output in fact lowers it.

Features. Sparse autoencoders split the hidden state into many features without being told anything about loops. After layer 19, one feature dominates, and a blind explainer labels it as "Wait, if…" re-checking. Applying Astra's rule along that single feature instead of u recovers more than half of the gain. After layer 15 the same signal is spread over many features that restate the problem, copy text or format numbers.

Steering. The direction works on its own. Adding −u to the hidden state while the model generates, with the weights untouched, raises GSM8K accuracy from 8% to 46%. Push twice as hard, and 90% of answers finish but only 3% are right. Astra's edit is itself a hard push: it does not delete the part along u, it reverses it fivefold. The edited states end up past normal text, in a region the model never visits on its own, and that is where its new failures come from, such as answers that end in long runs of whitespace.

Patching. If the edited neurons simply wrote "stop now" into the output, their direct route to the output would carry the effect. It carries 3%. Restoring one part at a time to its unedited behaviour shows that 91% runs through attention in layers 16 to 23. Those layers contain induction heads, which continue a repeated passage by looking back to where it appeared before and copying what came next. After the edit, attention to that earlier copy falls by about a third, and the token the model was about to repeat becomes the one it most avoids. Repeats still start, but they now end after tens of tokens instead of thousands.

A cheap score can pick the worse edit

Astra's logs include earlier attempts. After 10 rounds of search instead of 50, it had already found nearly the same direction and the same dominant feature. But it applied that direction gently across every layer from 8 to 22, and the result scores 24% on GSM8K against the final edit's 57%. Move those same directions into the final edit's recipe, and accuracy jumps to 49%. Where and how hard a direction is applied matters as much as the direction.

A cheap score would have picked the wrong one. A natural quick test for an editing agent, and one we used ourselves to screen edits, is how much an edit raises the model's loss on existing looping text. By that score the 10-round edit looks almost twice as good as the final one. A stronger version of it looks nearly five times as good, and in practice it scores below the unedited model.

Scoring old loops tests whether a model escapes a loop it is already in. It cannot see whether the model falls into new ones. The 10-round edit shortens loops but enters them twice as often, and the stronger version creates new kinds of loops that the old texts never contain. Astra itself judged its final edit the right way, by having the model answer 1,152 held-out practice problems. As AI systems start to edit models and run experiments on their own, the signal they use to judge their changes matters as much as the changes, and only running the edited model gave the right answer here.

An AI-written edit is an answer key for interpretability

Interpretability tools usually explain models whose true mechanism nobody knows, so their readings are hard to check. Here we know exactly what changed. That let us grade the tools, and Astra's own explanation, against the truth.

Astra's report says the edit counteracts "measured repetition directions". That is right, but it misses the attention path and the overshoot, and it argues the edit is not "simply forcing early termination" when forcing termination does about as well. Astra's other experiments show the same gap. In a sycophancy test it pointed to an attention head whose weights turn out not to matter, and its edits to a Gemma model came with convincing explanations but no gain.

The tools did better, but not uniformly. Most pointed the right way. Several gave readings that sounded convincing and were wrong, and only an experiment that changed the model caught them.

What held up was combining tools and checking each reading by changing the model. Every AI-written edit from now on is a new test of this kind. This is the pipeline we suggest for checking one: