Vercel's agent eval score doubled when it dropped custom tools for a file system

Andrew Qu expected his data science agent's clever tool chains to keep improving until trusted users called it awful at 30% eval scores. It took watching Claude Code and Opus 4.5 outperform his hand-built system to learn the fix wasn't more sophisticated tooling but a plain file system agents already know how to use.

• Origin — About a year ago Qu, Chief of Software at Vercel, asked marketing, sales, finance, and legal what they hated most about their jobs, and the data team's answer—dropping everything to write SQL for every question—was the most compelling.
• V1: mega prompt — The first version was a mega prompt: Qu pasted a Snowflake schema dump into a system prompt, had the model generate SQL, then copy-pasted and ran it himself.
• V2: agent chain (D0) — The second version, called D0, chained narrowly scoped agents—query, planning, execution, reporting—each with its own system prompt and limited tools like a 'read entity YAML' and 'search schemas' tool.
• V3: single agent — They then merged everything into one mega agent managing its own state across planning, building, executing, and reporting, running up to 100 steps.
• User verdict — Given to trusted users, the verdict was 'awful' despite clearing 30% of internal evals, and manually mapping more scenarios didn't scale.
• Claude Code unlock — Opus 4.5 paired with Claude Code answered the same questions 'without missing a beat,' and the team traced the advantage to a minimal file-system toolset: list file, read file, run bash.
• Rebuild in sandbox — Rebuilding the agent to run in a sandbox with the whole semantic layer dumped in and only bash/read/write file tools plus a few Vercel-specific ones doubled the eval score.
• Blog traffic — A blog post about this leap was reportedly responsible for 70% of Vercel.com traffic the week it published.
• Skills distillation — A recurring job now distills thousands of daily queries into roughly 100 reusable skills, giving each new agent run pre-built context instead of starting from nothing.
• Eve framework — Two weeks before this talk, Vercel released Eve, described as 'Next.js for agents,' using skills, tools, and channels folders with framework-defined infrastructure.
• Eve components — Eve pairs with Vercel Workflows for durability, Sandbox for secure execution, and Vercel Connect for short-lived OIDC connection tokens, while supporting open-source adapters for Postgres, OpenAI's responses API, and Docker.
• Aura case — Beta customer Aura rebuilt its website-testing agent in Eve and saw fewer steps, better success rates, and better insights compared to an off-the-shelf Claude-based setup.
• Scale at Vercel — Vercel now runs roughly 20 agents with product-market fit internally, spanning marketing retros, legal contract redlines, and the data science agent.

In their words

Honestly, I asked them for a dump of of the snowflake schema. I pasted it into a system prompt with a question and then when it generated SQL I actually copy and pasted that in and just ran it myself.3:38

And I would say this gave us a little bit of confidence that you know models today aren't that good but maybe we can harness engineer or make the context around it a little better3:57

So, we got to a few people's hands and the immediate response was it was awful. You know, we thought we were cooking. We thought this was, you know, nailing 30% of our evals, but we couldn't have anticipated some of the questions that were being asked.7:02

we realized that the big unlock was that it was just a file system. You know, we we had a very minimal set of tools, list file, read file, run bash7:57

Disclosure · Qu is Chief of Software at Vercel and is promoting Vercel's own Eve agent framework, AI SDK, Sandbox, Workflows, and Connect products.

One thing to add — One thing to add -- the talk's central claim (file system plus minimal tools beats bespoke tool chains) is presented as Vercel's internal experience rather than a controlled comparison, and the eval details (30% then doubled) are never given a baseline number or methodology. The "70% of traffic" and "20 PMF agents" figures also come with no denominator or timeframe, so they're best read as color rather than rigorous metrics.</note> </invoke>

One thing to try tonight
Try giving an LLM agent only three tools -- list files, read files, run bash in a sandbox -- against a real internal dataset dump, and compare its output to whatever custom-tool agent you already have.