tldraw's agent kit is MIT licensed, not a product pitch

Max Drake argues coding agents work because code is text in, text out — the exact medium they were trained on. Ask the same agent to align two shapes in 2D space and it falls apart, which is why tldraw spent real engineering effort teaching agents to read and act on a canvas.

• tldraw the SDK — tldraw is a free infinite canvas app, but Drake says the more important product is the SDK behind it, which powers other companies' canvases, including Replit's new agent canvas.
• Why tldraw exists — Drake says people building canvas apps kept getting stuck on selection, resizing, and matrix math and never reached their actual product, which is why tldraw built the canvas as a reusable engine.
• Agents fail at 2D — Drake claims agents are trained on text in, text out (like Claude Code writing code) and are consequently very bad at understanding and acting in 2D space.
• Teach project — In an earlier project called Teach, tldraw taught an LLM to interpret a canvas from a screenshot plus JSON; a single-shot prompt to 'make the mouse blow out the candle' produced correctly positioned wind and smoke using ordinary shapes, not a special mouse asset.
• Agent starter kit — The Teal Draw agent starter kit, MIT licensed and available on tldraw's site, wraps that canvas-reading capability in a harness so an agent can set its own to-dos and move its viewport to search the canvas, similar to a coding agent searching a codebase.
• Fairies project — Fairies renders each agent as a visible, ownable character (Drake demos one named Joan) that can be thrown around, given a hat, recolored, and even given a 'leg slider,' letting users read agent state at a glance instead of reading a chat log.
• Group chat mode — Selecting multiple fairies creates a group chat where one fairy acts as orchestrator, assigning a task (drafting a fiscal year 2025 financial memo) to another and waiting to review the finished work.
• Dependency graph habit — Drake says that near every launch, tldraw abandons task-tracking software in favor of one massive dependency graph built directly on tldraw.com, as shown by a real graph from the Fairies launch.
• Tech Tree app — Drake's prototype, the Tech Tree app, turns that dependency graph into a live, multiplayer board where each node is a coding agent task that can be kicked off, reviewed as a PR, and merged, comparable to Conductor or OpenAI's Symphony.
• Desktop app harness — tldraw's desktop app exposes the editor instance via a server so agents like Claude Code can write plain JavaScript against it, similar to 'code mode,' and act outside the canvas on the real computer.
• Window manager stunt — A tldraw colleague turned the desktop app into a window manager by having Claude Code generate rectangles and Apple Script to move real windows around, then used the same setup to play Pong with actual desktop windows.
• Live demo close — Drake's opening prompt — asking Claude to find a Notion doc from a Gmail link and build the described demo in the desktop app — was still running after roughly 13 minutes and did not finish by the end of the talk.

In their words

I don't know if you guys have ever, you know, tried to get your agents to do like UI stuff and try to get them align to align something. found that they could not do that whatsoever. Um because it turns out agents are really really bad at working in 2D space and understanding 2D space and actually requires like a lot of engineering work to get them to uh do it.4:32

I want to be very clear this is not um this is not like a special mouse shape. These are just like uh you know the these are just shapes uh on the canvas.5:44

So, you know, it got it, you know, it made the it made the smoke, it made the it made the wind, it got the positions right, and it understood what it was doing.6:17

the code is also MIT licensed. You can find it on um that you can find on the website.6:44

Disclosure · Max Drake is a product engineer at tldraw and is presenting and demoing his own company's SDK, app, and internal projects (agent starter kit, Fairies, Tech Tree).

One thing to add — One thing to add — the talk's most concrete engineering claim (agents need a taught representation of screenshot-plus-JSON to act correctly in space) is undercut by its own live demos, several of which stalled or didn't finish on stage, suggesting the harness is further along conceptually than operationally. It's worth watching whether the MIT-licensed starter kit gets independent adoption outside tldraw's own showcase projects.</note> </invoke>

One thing to try tonight
Clone tldraw's MIT-licensed agent starter kit from their site and try the single-shot canvas prompt pattern (screenshot + JSON in, positioned shapes out) on a simple two-shape alignment task to see where it breaks.