Vals found $1.5M in monthly token spend was 10x its salary bill

A startup built to independently benchmark frontier AI models discovered the same measurement problem inside its own walls: a month-long token-maxing experiment burned $1.5 million, ten times what it paid engineers over the same period. Vals founder Rayan Krishnan argues public benchmarks are already too gamed to trust, and that evaluation itself is becoming the deciding factor in enterprise AI ROI, model routing, and even AI policy.

• Origins — Vals launched in 2024 after co-founder Rayan Krishnan's team decided public benchmarks were no longer sufficient to measure model progress, releasing its first benchmarks that year.
• Llama 4 case — On Vals' held-out private benchmarks Meta's Llama 4 underperformed, while it scored strongly on major public benchmarks whose questions and rubrics are open source.
• Why labs can't self — grade — Labs want a 'rational buying market' to justify billion-dollar investments, which is why figures like Demis Hassabis have called for third-party evaluators.
• Pre — release testing — Vals runs distributed evaluations at maximum rate limits in the hours before a model launch, aided by an internal system called 'Steve, the economic Vals employee.'
• Fuzzy evals — Krishnan says the hardest problem is making implicit human distinctions (like associate vs. partner at a law firm) explicit enough to test, since no clean human eval exists either.
• Training — data conflict — Vals made an early decision never to sell training data to labs, avoiding what Krishnan compares to Enron-style auditor conflicts of interest.
• RSI Index — Vals released a Recursive Self-Improvement Index, using proxies for pre-training, post-training, and harness engineering since training a model to build its successor is too costly to test directly.
• Deprecating — Deprecating benchmarks — Vals retires saturated benchmarks under an 'always a higher peak' motto and updates others, like a legal research benchmark, to reflect current case law.
• Agentic shift — Newer evals run over hours, days, or weeks, need retry-safe infrastructure, and trade huge sample sizes (like ImageNet's millions of images) for fewer, more complex tasks with richer rubrics.
• Token economics — A Fortune 10 company gave engineers a $100-a-day Claude Code budget (since raised to $300), reshaping work patterns around a 4pm rate-limit reset; Vals itself found engineers burning 1-2 billion tokens a day, peaking at 6 billion for one engineer.
• ValSmith launch — Vals released ValSmith, letting companies build internal coding benchmarks from their own GitHub repos to find the highest-ROI coding model, revealing that Sonnet can cost more than Opus despite being 'token hungry.'
• Policy role — Krishnan frames Vals as an evidence-gathering layer for policymakers, running regular briefings for executive and legislative branches on model capability and risk.
• Geopolitics — Krishnan draws a nuclear-arms-control analogy, citing Reagan's 'trust but verify' and flyover verification, as a model for how nations might eventually audit each other's AI capabilities, especially around recursive self-improvement.

In their words

That was a bit of a disaster. And and interestingly what we saw is that on our heldout private benchmarks the model is actually underperforming but on all of the the major public benchmarks where that the questions and rubrics are actually open source um it was it was showing incredible capability.2:39

we had a lot of engineers spending between 1 to two billion tokens a day. I think peak day was one engineer spending six billion.22:50

it looked like in that month we spent roughly $1.5 million worth of tokens. This is free, by the way. I No, don't want um but it was actually 10x more we were spending in tokens than employee salary for that month.23:06

I think the honest answer is that it's forcing a lot of the um more fuzzy or distributed forms of eval to be made explicit like what what is really the distinction between an associate and a partner at a law firm.5:58

Disclosure · Rayan Krishnan is founder and CEO of Vals, the evaluation company discussed throughout; the interview promotes Vals' benchmarks and its new ValSmith product.

One thing to add — One thing to add — Krishnan's numbers on token spend versus salary are anecdotal (one Fortune 10 company, one internal Vals experiment), not a market-wide study, so they illustrate a directional worry rather than a measured trend. It's also notable that a company selling evaluations is the one arguing evaluations should become the industry's central currency.

One thing to try tonight
Pull your own team's GitHub repo and coding-agent usage logs for the past week and roughly tally tokens spent versus lines of useful code shipped, the same rough audit Vals ran on itself before finding it was outspending payroll ten to one.