50% of sites publish llms.txt; almost no agent reads it

Liad Yosef's research lab ran thousands of agents against real websites and found the de facto standard for agent-readiness, llms.txt, is almost never used even when it's published. His talk argues the fix isn't writing better specs for agents, it's watching what they actually do — and that MCP Apps, which he co-created, is what made an agentic web possible in the first place.

• Speaker background — Yosef is co-creator and maintainer of the MCP Apps spec, co-founder of a company called Aura that researches agentic human interactions, and formerly built the agentic storefronts at Shopify.
• MCP Apps basics — MCP Apps launched a few months ago with Claude as first client and has since been adopted by every major chat client except Gemini, letting servers send UI chunks (not just text) into chats so brands like Booking.com keep their visible identity.
• The agentic web shift — Yosef argues two decades were spent perfecting the browser-tab experience, but users no longer need 99% of Booking's or Airbnb's or Jira's dashboards — just atoms of functionality a personal assistant can compose.
• Against browser agents — He dismisses computer-use/browser agents as a 'faster horses' solution and contrasts Google's WebMCP, which exposes website JavaScript tools to agents like Gemini in Chrome, as still too tied to dashboards nobody wants to browse.
• Anecdotal evidence — He cites his 9-year-old and a 70-year-old woman in Georgia whose phone had only WhatsApp, camera and ChatGPT, plus Cloudflare's CEO tweeting that agent traffic has surpassed human traffic on the web.
• Sentry's pivot — David Cramer of Sentry, who a year earlier bet websites wouldn't be obsolete in 25 years, published 'Designing for Agents' arguing products must be API-first; Salesforce, Cloudflare and Sentry have all gone headless despite UI being a core differentiator.
• Brand loyalty lost — Asked to pick an analytics service, Cloud Code recommended PostHog over Mixpanel, which Yosef's team had used for a decade, purely because PostHog had better MCP/API integration — and they switched.
• Aura benchmark tool — Aura.ai offers a free readiness benchmark and leaderboard scoring websites against agentic protocols and best practices, built after raising capital to research web readiness for agents.
• llms.txt finding — About 50% of tested sites published llms.txt, but almost none of the agents used it — most went straight to the docs page then the homepage, and the roughly 40% that did use it only found it because docs referenced the file.
• Aura Journey tool — Journey.aura.ai lets anyone run any intent against any website with any agent and watch the real-time path taken, run tens of thousands of times, showing Cloud Code, Vercel's Eve, and ChatGPT take very different routes on the same site (example given: attio.com).
• Discovery layer question — Yosef says the next milestone is a discovery layer for agentic resources, citing emerging standards like ai-catalog.json (backed by Anthropic, OpenAI, Google) and the Agentic Resource Discovery (ARD) standard.
• Aura Directory — Aura built a directory exposing ai-catalog.json files for scanned domains, such as monday.com's MCP and API servers, fully compliant with the ARD standard.
• Accessibility parallel — Yosef closes by noting that making a website human-accessible (for users with vision disabilities) closely correlates with making it agent-accessible, since LLMs also can't 'see' a site and rely on the same alternate signals.

In their words

we found out that almost 50% of the website that we tested that we ran published lms.txt but none of the agents that we ran on this website actually used llm.txt txt. Actually, almost all the agents went straight to the docs page and then the homepage.13:34

So then it hit us. We said it doesn't make a lot of sense for us as humans to define to agents what they need. No one is doing it anymore.13:59

we asked Cloud Code, what's the best analytic service? And it recommended Postto, which is an analytic service. And we said no, we prefer Mixpanel because we know Mixpanel. We worked with Mixpanel for a decade, we know how to work with it. And Cloud Code insisted on Postto because it said Postto has better MCP and API and I can integrate to it better. So we don't have brand loyalty, right? We went with postto hog11:47

Disclosure · Yosef co-created the MCP Apps spec and co-founded Aura, whose benchmark, Journey, and Directory tools he promotes throughout the talk.

One thing to add — One thing to add — the llms.txt finding is the strongest empirical claim in the talk, but Yosef gives no sample size, agent list, or methodology beyond "thousands of runs," so the 50%/40% figures should be read as directional rather than rigorous. His Mixpanel-to-PostHog anecdote is a single internal decision, not a controlled study, though it's used to argue a much broader claim about brand loyalty evaporating.</note> </invoke>

One thing to try tonight
Run your own product's homepage through journey.aura.ai with a real user intent and compare the path Cloud Code, ChatGPT, and another agent actually take versus what your llms.txt or docs assume they'll do.