Latest results
Oct 2026 vs May 2026
Biggest improvements
Points gained since the May 2026 run
- +50AssemblyAI
- +43Camunda
- +42Semgrep
- +41Netlify
- +38Render
- +23Vonage
- +21Greptile
- +19Groq
- +17Agentmail
- +17Resend
Current top 10 companies
Oct 2026
- 92Deepgram
- 92Agentmail
- 91Resend
- 88Tenki
- 87LiveKit
- 83Sprites (Fly.io)
- 83Extend.ai
- 81Tavily
- 80Firecrawl
- 80Stripe
Previous top 10 companies
May 2026
- 89LiveKit
- 83Daytona
- 79Firecrawl
- 79Jina AI
- 79Steel
- 77Deepgram
- 77Stripe
- 77Sprites (Fly.io)
- 76WorkOS
- 76Browserbase
LightsageClickTap any row to explore the full CLI eval.
Get your CLI graded →
| Rank | Tool | Score | Grade | Task | Eval | Discovery | Cost | Calls | Errors | Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|
1🥇 | Deepgram | 92 | A | 92 | 95 | $0.0959 | 7 | 0 | 1m 49s | View report | |
1🥇 | Agentmail | 92 | A | 89 | 100 | $0.1333 | 12 | 0 | 59s | View report | |
3🥉 | Resend | 91 | A | 88 | 100 | $0.1180 | 9 | 0 | 1m 19s | View report | |
4 | Tenki | 88 | B | 92 | 80 | $0.1352 | 9 | 0 | 54s | View report | |
5 | LiveKit | 87 | B | 85 | 93 | $0.1530 | 11 | 0 | 1m 21s | View report | |
6 | Sprites (Fly.io) | 83 | B | 96 | 53 | $0.0909 | 8 | 0 | 55s | View report | |
6 | Extend.ai | 83 | B | 85 | 80 | $0.1884 | 13 | 0 | 1m 31s | View report | |
8 | Tavily | 81 | B | 84 | 75 | $0.1929 | 16 | 0 | 1m 45s | View report | |
9 | Firecrawl | 80 | B | 83 | 75 | $0.2375 | 14 | 0 | 1m 48s | View report | |
9 | Stripe | 80 | B | 75 | 93 | $0.1656 | 11 | 1 | 1m 23s | View report | |
9 | Steel | 80 | B | 75 | 93 | $0.1519 | 15 | 2 | 1m 31s | View report | |
12 | WorkOS | 79 | B | 73 | 95 | $0.2368 | 15 | 4 | 1m 48s | View report | |
13 | Vercel | 78 | B | 78 | 78 | $0.1072 | 10 | 1 | 1m 33s | View report | |
14 | Jina AI | 77 | B | 80 | 70 | $0.2431 | 15 | 0 | 2m 3s | View report | |
14 | E2B | 77 | B | 85 | 60 | $0.1582 | 12 | 0 | 1m 28s | View report | |
16 | Pinecone | 76 | B | 75 | 78 | $0.1881 | 13 | 1 | 1m 30s | View report | |
17 | AssemblyAI | 75 | B | 78 | 70 | $0.1278 | 8 | 2 | 1m 27s | View report | |
17 | Cloudflare | 75 | B | 64 | 100 | $0.3999 | 35 | 5 | 5m 3s | View report | |
17 | Browserbase | 75 | B | 75 | 75 | $0.1621 | 11 | 1 | 1m 33s | View report | |
20 | Render | 74 | C | 63 | 100 | $0.4943 | 30 | 7 | 5m 11s | View report | |
21 | Nimble | 73 | C | 67 | 87 | $0.3085 | 20 | 3 | 3m 1s | View report | |
21 | Daytona | 73 | C | 75 | 70 | $0.1458 | 12 | 1 | 1m 36s | View report | |
23 | Modal | 70 | C | 79 | 50 | $0.2555 | 18 | 0 | 2m 31s | View report | |
24 | Browser Use | 69 | C | 64 | 82 | $0.5238 | 33 | 5 | 4m 58s | View report | |
25 | Vonage | 68 | C | 85 | 30 | $0.1453 | 13 | 0 | 1m 51s | View report | |
25 | Box | 68 | C | 70 | 63 | $0.2428 | 21 | 1 | 1m 50s | View report | |
27 | Auth0 | 64 | C | 73 | 45 | $0.3386 | 15 | 1 | 1m 45s | View report | |
27 | Descope | 64 | C | 69 | 55 | $0.2769 | 19 | 2 | 2m 47s | View report | |
27 | Rime | 64 | C | 75 | 40 | $0.1863 | 15 | 2 | 1m 26s | View report | |
27 | Netlify | 64 | C | 63 | 68 | $0.6344 | 44 | 4 | 7m 23s | View report | |
27 | Reducto | 64 | C | 75 | 40 | $0.1521 | 14 | 1 | 1m 37s | View report | |
27 | Naive | 64 | C | 63 | 68 | $0.6209 | 37 | 9 | 5m 30s | View report | |
33 | Temporal | 63 | C | 69 | 50 | $0.2688 | 20 | 3 | 2m 50s | View report | |
33 | LlamaParse | 63 | C | 73 | 40 | $0.1770 | 14 | 1 | 2m 14s | View report | |
33 | Datadog | 63 | C | 68 | 53 | $0.2530 | 21 | 3 | 2m 35s | View report | |
36 | ElevenLabs | 59 | C | 68 | 40 | $0.3228 | 22 | 3 | 2m 10s | View report | |
36 | Camunda | 59 | C | 63 | 50 | $0.5489 | 37 | 2 | 5m 20s | View report | |
38 | Semgrep | 57 | C | 69 | 30 | $0.2698 | 18 | 1 | 2m 39s | View report | |
39 | Railway | 46 | D | 27 | 93 | $0.5193 | 33 | 6 | 5m 30s | View report | |
40 | Azion | 38 | D | 23 | 73 | $1.2247 | 69 | 10 | 18m 16s | View report | |
40 | Groq | 38 | D | 31 | 55 | $1.0701 | 60 | 0 | 13m 42s | View report | |
40 | Unstructured | 38 | D | 38 | 40 | $0.0000 | 17 | 5 | 2m 5s | View report | |
44 | Chroma | 35 | D | 24 | 60 | $0.5970 | 36 | 12 | 7m 53s | View report | |
44 | Greptile | 35 | D | 34 | 40 | $0.2921 | 23 | 10 | 2m 38s | View report | |
46 | Scalekit | 32 | D | 32 | 33 | $0.3789 | 25 | 1 | 5m 27s | View report | |
48 | Cartesia | 31 | D | 22 | 55 | $0.2723 | 16 | 0 | 2m 30s | View report | |
48 | Twilio | 31 | D | 29 | 35 | $0.3655 | 25 | 14 | 4m 40s | View report | |
50 | Telnyx | 27 | D | 24 | 35 | $0.0000 | 4 | 1 | 2m 14s | View report | |
51 | OpenRouter | 26 | D | 12 | 60 | $0.3338 | 23 | 1 | 3m 4s | View report | |
52 | Fireworks AI | 25 | D | 14 | 50 | $0.2219 | 15 | 2 | 2m 18s | View report | |
54 | Apollo | 21 | D | 7 | 53 | $0.5321 | 37 | 2 | 3m 5s | View report | |
43 | Qodo CLI disabled | — | — | — | — | — | — | — | — | — | Trace unavailable |
46 | CodeRabbit No API key available | — | — | — | — | — | — | — | — | — | Trace unavailable |
53 | Anchor Browser No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
55 | Nebius AI Cloud API key requires a paid plan | — | — | — | — | — | — | — | — | — | Trace unavailable |
56 | Unblocked No API key available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Nylas Missing CLI auth env vars: NYLAS_GRANT_ID | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Zilliz Cloud Missing CLI auth env vars: ZILLIZ_API_KEY | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | DigitalOcean Signup requires a credit card | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Weaviate Missing CLI auth env vars: WEAVIATE_API_KEY, WEAVIATE_URL | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Pipecat Open-source / self-hosted — no hosted API key available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Mollie No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | MeetGeek No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | LanceDB No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Cerebras No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Circle No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — |  AWS Step Functions Could not obtain API key | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Airwallex API key requires official certification or company verification | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Brave Search API key requires a paid plan | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Clerk No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Qdrant No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Netflix Conductor Open-source / self-hosted — no hosted API key available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Razorpay No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Cloudflare Workers No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Paddle No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Agora No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Triple-A Could not obtain API key | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Meetstream No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Akka Could not obtain API key | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | LemonSqueezy No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Fireblocks Could not obtain API key | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | DeepInfra API key requires a paid plan | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Prefect No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | TogetherAI API key requires a paid plan | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Meeting BaaS No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | DFNS Could not obtain API key | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | You.com No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | SambaNova No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Perplexity API key requires a paid plan | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Chargebee No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Stytch No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — |  CueMeet Open-source / self-hosted — no hosted API key available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Cadence Open-source / self-hosted — no hosted API key available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Adyen API key requires official certification or company verification | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Recall.ai No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | OpenAI Whisper API key requires a paid plan | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Square Could not obtain API key | — | — | — | — | — | — | — | — | — | Trace unavailable |
| — | Coinbase Payments No CLI tool available | — | — | — | — | — | — | — | — | — | Trace unavailable |
Scores calculated from end-to-end CLI evaluations by AI coding agents.
Frequently asked questions
Join the mailing list
Get updates on leaderboard changes, new benchmark releases, and product announcements.