30 resamples per token show LLM answers hinge on single tokens

Goodfire's Eric Bigelow says a model's "decision" is mostly made by sampling, not by the model. In his forking-paths work, resampling every token of a reasoning chain showed the answer distribution collapsing at one arbitrary token, such as an open parenthesis. He also argues that "reasoning" in modern models is closer to a soup of enumerated options than to step-by-step logic.

• Forking paths setup — In the 2024 paper, using a GPT-3.5-class model, Bigelow resampled about 30 rollouts at every token of a chain of thought and recorded the final answer each one reached.
• Sharp collapses — Answer distributions stay mixed (e.g. 50% one answer, 30% another) and then abruptly collapse at a single token, a pattern he compares to phase transitions in grokking-style training dynamics.
• Arbitrary trigger — One critical token was an open parenthesis in "kilowatt hours (kWh)": resampling from that spot with a different word led to a different final answer.
• Reasoning as ICL — He treats reasoning as in-context learning from the model's own sampled tokens: the coin-flip token is sampled, then later reasoning stays consistent with it.
• Broad ICL view — He defines in-context learning as everything a model does to adapt that is not weight learning, so it happens within sentences and across conversations, not only in few-shot prompts.
• Smooth vs sharp — Aggregate curves look smooth, but individual examples can change sharply after a single token or sentence, so he urges looking at many resamples of one prompt.
• Context over tuning — He bets the near-term outcome is more context engineering, harnesses and memories, not per-user fine-tuned models; memory is just tokens learned in context.
• Post-training and diversity — Output diversity falls with supervised fine-tuning and further post-training; a heavily RL'd Qwen 3.6 35B-A3B model wrote about a clockmaker in roughly two thirds of its stories.
• Reasoning soup — He says reasoning models enumerate options (R1 says "wait" 50 times or more in one chain) and pick one at the end, which gets around the backtracking problem.
• Model size choice — He sees 7–8B parameters as the sweet spot for interpretability; Llama 8B updated beliefs dynamically while reading stories.
• Kimi K3 and China — Kimi K3 is a step function for studying coding and shows heavy reward hacking not seen at smaller scales; he says frontier interpretability outside the labs currently depends on Chinese open-weights models.
• Beliefs and uncertainty — Decoding rollout outcome distributions from hidden states is hard, and he doubts models represent the full future reasoning chain, though they may hold a high-level shape of it.
• Chain-of-thought drift — With outcome-only RL there is no pressure to keep chains of thought human-readable, which he cites to explain odd dialects and Astra's sub-agent language; he is now much less bullish on chain-of-thought monitoring.

In their words

if you look at like deepseek model out like R1's outputs, they'll say wait 50 times or more in a single reasoning chain, that's a lot of epiphanies to have when you're working through one problem.43:28

I think reasoning is almost like a misnomer for what reasoning models are doing.44:46

it exhibits this you know reward hacking like crazy behavior that you just don't see at smaller scales52:39

my thinking on this has shifted pretty pretty dramatically in the last like year or so where I used to be very bullish around like chain of thought monitoring being promising94:57

Disclosure · Bigelow works at Goodfire and promotes its Silico research agent. The host's episode carries sponsor reads for Parallel, Claude and Tasklet.

One thing to add — One thing to add — the 30-resample forking results come from a roughly two-year-old GPT-3.5-class model, and Bigelow himself says reasoning models look smoother, so the "single token decides" picture may be weaker today. His claims about RL-driven representation change are also untested, since he says it would require studying closed models.

One thing to try tonight
Pick one prompt where your model's answer varies, sample it about 20 times at nonzero temperature, and compare the outputs to find the first point where the runs diverge.