Devtool Arena methodology

This page covers how a Devtool Arena run works, how it's scored, which agents and models run, and why some tools are skipped. It reflects the October 2026 batch.

How a Devtool Arena run works

A Devtool Arena run is one coding agent doing one realistic task with one tool, start to finish, in its own sandbox.

One task per category

Every tool in a category gets the same task, worded the way a developer would ask for it. In Inference, the task is "build me a script that extracts structured data from this invoice and returns JSON", with the vendor's docs URL as the starting point. The agent gets nothing else about the tool, so it has to find the right docs, SDK and endpoints itself.

A live key in a sandbox

The agent writes the script, runs it against the live API with a real key from a self-serve account, and checks the output. We record every tool call, error, the runtime and the model cost. On the Claude Code board (October 2026), OpenRouter finished the Inference task in 58 seconds with 6 tool calls, 0 errors and $0.12 of model usage, for a score of 90. Fireworks AI did the same task in 1 minute 12 seconds with 11 tool calls and 1 error, for 79. The compare page shows both runs side by side.

Two scores, one number

The eval score is the task: did the integration work end to end, and how cleanly. The discovery score checks that agent-ready assets exist and are public, such as an llms.txt file, Context7 docs, MCP docs, a typed SDK, an OpenAPI spec, agent skills and a CLI. The API board's final score is 70% eval and 30% discovery. The discovery score doesn't test whether agents recommend you. The Agent Landscape is where Devtool Arena ranks tools by how often coding agents recommend them.

Which coding agents and models are on Devtool Arena

Devtool Arena has an API board for 12 coding agents as of October 2026. Each agent runs on one model for a whole batch, so every tool on a board faces the same agent.

Claude Code and Codex also have MCP and CLI boards, which run the same kind of task through a vendor's MCP server or command-line tool.

Why some tools are skipped

A tool is skipped when we can't get a working API key the way a new developer would. On the Claude Code API board (October 2026 batch), 128 tools were ranked and 63 more were listed as skipped. The usual blockers were a paid plan with no free credits, a signup that needed a card, a phone number or a sales call, and products that are self-hosted only.

Skipped tools aren't ranked. They're listed below the board with the reason. If your tool is skipped, send an account to devtoolarena@lightsage.com and we'll run it in the next batch.

Common misconceptions about Devtool Arena

"It's really testing the model." Each board uses one agent and one model for the whole batch, so differences between tools in a category come from the tools: their docs, SDKs, errors and signup.

"A high discovery score means agents recommend you." The discovery score checks that assets like llms.txt and an OpenAPI spec exist and are public. Recommendations are measured separately, on the Agent Landscape.

"Giving Devtool Arena an account improves your score." An account only lets the run happen. The task and grader are the same as for every other tool in the category.

How to check your tool on Devtool Arena

1. Find your tool on each board. Start with the Claude Code API board and switch agents from the menu. Open your tool's report.
2. Read the run. The report shows the task, every tool call, errors, runtime and cost. Compare your run with the leader in your category on the compare page.
3. Fix it and get re-run. Changes show up in the next batch. If your tool isn't listed, add your API. If you want the same runs on your own tasks and schedule, that's Lightsage Agent Usability.