Compare companies

Side-by-side CLI results for companies evaluated by Claude Code. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Showing the first two companies on mobile. Use a wider screen to compare 3 companies.

Selected

Company 1

Company 2

Company 3

Mobile compare is limited to two companies. Use a wider screen to compare all 3.

Deepgram

Voice STT

OpenRouter

Inference

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Voice STT

Inference

Overall rank

Best

#50

Category rank

Best

Full eval

Open report

CLI Task Scoring

Eval score

92%

Best

12%

Task Completed

Passed+30The agent successfully completed all task requirements: (1) installed the official `dg` CLI via `pip install deepctl`, (...

Best

Failed+0The agent successfully extracted structured invoice data with all required fields (invoice_number, vendor, date, line_it...

Used CLI Correctly

Passed+20Agent used CLI commands correctly

Best

Failed+0CLI usage issues: Used @mrgoonie/openrouter-cli (third-party package) instead of the official @openrouter/cli for the in...

Partial Credit

Passed+10Partial credit: 100%

Best

Partial+6Partial credit: 60%

Zero Errors

Passed+10No errors

Best

Failed+0Failed tool calls: 1

Efficiency

Partial+67 tool calls (<=10, great)

Best

Partial+1.023 tool calls (<=25, below average)

Time

Partial+6109s (1-2min, great)

Best

Partial+2.0184s (3-5min, slow)

Cost

Passed+10$0.10 (<$0.10, excellent)

Best

Partial+3.5$0.33 ($0.20-0.40, good)

Runtime (actual)

1m 49s

Best

3m 4s

Cost (actual)

$0.0959

Best

$0.3338

Tool calls

Best

Errors

Best

Tokens used

181,457

Best

707,116

CLI Discovery

Discovery score

95%

Best

60%

CLI exists

Passed5/5dg

Best

Passed5/5openrouter

Best

Install docs

Passed5/5pip

Best

Passed5/5npm

Best

Auth method

Passed25/25env var

Best

Passed25/25env var

Best

Non-interactive

Passed25/25auto detect

Best

Failed0/25none

JSON output

Passed15/15full

Best

Failed0/15none

llms.txt

Failed0/5

Passed5/5

Best

Agent skill

Passed20/20

Best

Passed20/20

Best

Eval Details

Integration task

# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...

# Task Using the openrouter CLI (OpenRouter), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Ser...

Generated files

Trace tool calls

Completed

9/27/2026

Stderr

None captured

Metric	Deepgram Voice STT	OpenRouter Inference	Fireworks AI Inference
Summary
Compare status	Comparable	Comparable	Comparable
Overall score	92 Best	26	25
Grade	A	D	D
Category	Voice STT	Inference	Inference
Overall rank	#1 Best	#50	#51
Category rank	#1 Best	#2	#3
Full eval	Open report	Open report	Open report
CLI Task Scoring
Eval score	92% Best	12%	14%
Task Completed	Passed+30The agent successfully completed all task requirements: (1) installed the official `dg` CLI via `pip install deepctl`, (... Best	Failed+0The agent successfully extracted structured invoice data with all required fields (invoice_number, vendor, date, line_it...	Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...
Used CLI Correctly	Passed+20Agent used CLI commands correctly Best	Failed+0CLI usage issues: Used @mrgoonie/openrouter-cli (third-party package) instead of the official @openrouter/cli for the in...	Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...
Partial Credit	Passed+10Partial credit: 100% Best	Partial+6Partial credit: 60%	Partial+4.0Partial credit: 40%
Zero Errors	Passed+10No errors Best	Failed+0Failed tool calls: 1	Failed+0Failed tool calls: 2
Efficiency	Partial+67 tool calls (<=10, great) Best	Partial+1.023 tool calls (<=25, below average)	Partial+3.515 tool calls (<=15, good)
Time	Partial+6109s (1-2min, great) Best	Partial+2.0184s (3-5min, slow)	Partial+3.5138s (2-3min, good)
Cost	Passed+10$0.10 (<$0.10, excellent) Best	Partial+3.5$0.33 ($0.20-0.40, good)	Partial+3.5$0.22 ($0.20-0.40, good)
Runtime (actual)	1m 49s Best	3m 4s	2m 18s
Cost (actual)	$0.0959 Best	$0.3338	$0.2219
Tool calls	7 Best	23	15
Errors	0 Best	1	2
Tokens used	181,457 Best	707,116	361,995
CLI Discovery
Discovery score	95% Best	60%	50%
CLI exists	Passed5/5dg Best	Passed5/5openrouter Best	Passed5/5firectl Best
Install docs	Passed5/5pip Best	Passed5/5npm Best	Passed5/5binary download Best
Auth method	Passed25/25env var Best	Passed25/25env var Best	Passed25/25env var Best
Non-interactive	Passed25/25auto detect Best	Failed0/25none	Failed0/25none
JSON output	Passed15/15full Best	Failed0/15none	Passed15/15full Best
llms.txt	Failed0/5	Passed5/5 Best	Failed0/5
Agent skill	Passed20/20 Best	Passed20/20 Best	Failed0/20
Eval Details
Integration task	# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...	# Task Using the openrouter CLI (OpenRouter), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Ser...	# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...
Generated files	1	3	1
Trace tool calls	7	23	15
Completed	9/27/2026	9/27/2026	9/27/2026
Stderr	None captured	None captured	None captured