One warm-up task beat four in a browser-agent ablation study

A small ablation study on browser agents found that stuffing more prior experience into context made agents slower, not smarter, and that a single prior task plus a compressed "recipe" beat resuming four full past sessions. The results come from a handful of RealBench tasks run on a personal Claude Code Max plan, not a rigorous benchmark, but they cut against the assumption that more context or more expert demonstrations automatically helps.

• Setup — Sebastian Sosa built the study to compare Open Cloud and Chrome against the official Anthropic Claude and Chrome, plus a Brave browser variant, adding code mode and dense action recordings to Open Cloud and Chrome.
• Ablation variables — He varied browser choice, context load, and information source (experiential — the agent's own past task runs — versus expert demonstrations recorded by a human), delivered either as retrievable analysis, embedded recipe, or raw data.
• Baseline (phase 1) — Claude and Chrome had lower turn counts but higher latency, while Open Cloud and Chrome in plain Chrome had lower latency but more turns; per-tool-call speedup was only about 0.2 seconds, called noise given small sample sizes off a Claude Code Max plan.
• Raw traces hurt (phase 2) — Giving the agent raw Claude Code traces or raw expert recordings in the repository regressed performance versus baseline, increasing turns and latency because the agent had to search the workspace to parse unfamiliar structure before starting the task.
• Compression helps (phase 3) — Compressing prior workspace data into a single analysis artifact significantly cut turns per task for run 3A and cut both turns and latency for run 3B versus the raw-trace runs.
• Resuming sessions (phase 4) — Resuming a session after four prior tasks (run 4A) reduced turn count but caused latency to "skyrocket" because higher context slows token generation, making the approach net unhelpful when both turns and latency matter.
• Recipes and routing (phase 5) — Baking a compressed recipe of combined experiential and expert knowledge directly into the prompt, with dynamic routing per website (RealBench's site A vs. site B), landed close to prior results with a meaningful turn-count gain but only modest latency improvement.
• One task wins (phase 6) — Run 6A tested the hypothesis that a single prior task teaches most of the navigation skill gained from four, and it held for RealBench's simple sites, cutting both latency and turn count significantly.
• Best configuration — Combining one prior experiential task with the dynamic-routing recipe (run 6B) produced the best latency and best turn count of all tested configurations.
• Recording format vs. Anthropic — Open Cloud and Chrome's denser, untransformed action recordings outperformed Claude co-work's recordings, which Sosa attributed to Anthropic distilling recordings into a single skill artifact rather than preserving raw multi-track data.
• Multimodal encoding — Recordings pair low-resolution screenshots, action descriptions, and live narration on a shared timeline, using the same coordinate schema Claude and Chrome already use, rather than continuous video, since current harnesses only support screenshot-based tool calls.
• Cost and scale caveat — All runs were free under Sosa's Claude Code Max subscription with no token-cost tracking, and reported per-task times (e.g., about four minutes for the 6B runs) are aggregated across a leg rather than per-question.

In their words

what was discovered is that these actually regress from the baseline. Um where well they they regress they regress in terms of they regress in in ter in terms of turns per task20:00

turns per task actually decreases for um significantly decreases is for um for 3A but and also significantly decreases both on turns per task and latency for 3B22:49

while turns per task did decrease, latency like skyrocketed. And this is just due to the fact that higher context is just like slows down the results.25:14

that speed up is like 0.2 seconds per tool call which is kind of marginal when you're looking at this the scope of an actual task. So this is really just noise.19:20

Disclosure · Sosa is presenting his own independent research project (an unofficial fork/extension of Anthropic's Claude in Chrome called "Open Cloud and Chrome"), not a company product he sells, but he is showcasing his own tool.

One thing to add — One thing to add — Sosa repeatedly flagged that sample sizes were tiny and results should be taken "with a grain of salt," so the specific numeric wins (e.g., 6B being optimal) are directional signals from a handful of RealBench tasks rather than a validated benchmark. It's still a useful counterpoint to the default instinct to cram more prior experience into an agent's context window.</note> </invoke>

One thing to try tonight
Try compressing your own agent's past session logs into a single short analysis artifact (instead of pointing the agent at raw prior traces or resuming a long session) and see whether turn count or latency improves on a repetitive task.