Compare companies

Side-by-side CLI results for companies evaluated by Codex. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Showing the first two companies on mobile. Use a wider screen to compare 3 companies.

Selected

Company 1

Company 2

Company 3

Mobile compare is limited to two companies. Use a wider screen to compare all 3.

Deepgram

Voice STT

OpenRouter

Inference

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Voice STT

Inference

Overall rank

Best

#49

Category rank

Best

Full eval

Open report

CLI Task Scoring

Eval score

85%

Best

17%

Task Completed

Passed+30The agent successfully completed all required steps: (1) downloaded the sample audio file using curl, (2) installed the...

Best

Failed+0The agent successfully extracted structured invoice data and produced valid JSON output with all required fields (invoic...

Used CLI Correctly

Passed+20Agent used CLI commands correctly

Best

Failed+0CLI usage issues: Used @mrgoonie/openrouter-cli instead of the official @openrouter/cli from the task specification, The...

Partial Credit

Passed+10Partial credit: 100%

Best

Partial+6Partial credit: 60%

Zero Errors

Passed+10No errors

Best

Failed+0Failed tool calls: 2

Efficiency

Partial+3.511 tool calls (<=15, good)

Best

Partial+2.019 tool calls (<=20, acceptable)

Time

Partial+6103s (1-2min, great)

Best

Partial+3.5147s (2-3min, good)

Cost

Partial+6$0.11 ($0.10-0.20, great)

Best

Partial+6$0.19 ($0.10-0.20, great)

Best

Runtime (actual)

1m 42s

Best

2m 26s

Cost (actual)

$0.1075

Best

$0.1949

Tool calls

Best

Errors

Best

Tokens used

374,221

Best

817,040

CLI Discovery

Discovery score

95%

Best

60%

CLI exists

Passed5/5dg

Best

Passed5/5openrouter

Best

Install docs

Passed5/5pip

Best

Passed5/5npm

Best

Auth method

Passed25/25env var

Best

Passed25/25env var

Best

Non-interactive

Passed25/25auto detect

Best

Failed0/25none

JSON output

Passed15/15full

Best

Failed0/15none

llms.txt

Failed0/5

Passed5/5

Best

Agent skill

Passed20/20

Best

Passed20/20

Best

Eval Details

Integration task

# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...

# Task Using the openrouter CLI (OpenRouter), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Ser...

Generated files

Trace tool calls

Completed

9/27/2026

Stderr

None captured

Metric	Deepgram Voice STT	OpenRouter Inference	Fireworks AI Inference
Summary
Compare status	Comparable	Comparable	Comparable
Overall score	88 Best	30	25
Grade	B	D	D
Category	Voice STT	Inference	Inference
Overall rank	#4 Best	#49	#51
Category rank	#1 Best	#2	#3
Full eval	Open report	Open report	Open report
CLI Task Scoring
Eval score	85% Best	17%	15%
Task Completed	Passed+30The agent successfully completed all required steps: (1) downloaded the sample audio file using curl, (2) installed the... Best	Failed+0The agent successfully extracted structured invoice data and produced valid JSON output with all required fields (invoic...	Failed+0The agent successfully extracted the structured JSON data with all required fields (invoice_number, vendor, date, line_i...
Used CLI Correctly	Passed+20Agent used CLI commands correctly Best	Failed+0CLI usage issues: Used @mrgoonie/openrouter-cli instead of the official @openrouter/cli from the task specification, The...	Failed+0CLI usage issues: Used curl with raw HTTP API calls instead of firectl CLI for the final inference call, firectl session...
Partial Credit	Passed+10Partial credit: 100% Best	Partial+6Partial credit: 60%	Partial+3Partial credit: 30%
Zero Errors	Passed+10No errors Best	Failed+0Failed tool calls: 2	Failed+0Failed tool calls: 4
Efficiency	Partial+3.511 tool calls (<=15, good) Best	Partial+2.019 tool calls (<=20, acceptable)	Failed+029 tool calls (>25, inefficient)
Time	Partial+6103s (1-2min, great) Best	Partial+3.5147s (2-3min, good)	Partial+6106s (1-2min, great) Best
Cost	Partial+6$0.11 ($0.10-0.20, great) Best	Partial+6$0.19 ($0.10-0.20, great) Best	Partial+6$0.20 ($0.10-0.20, great) Best
Runtime (actual)	1m 42s Best	2m 26s	1m 46s
Cost (actual)	$0.1075 Best	$0.1949	$0.1953
Tool calls	11 Best	19	29
Errors	0 Best	2	4
Tokens used	374,221 Best	817,040	730,987
CLI Discovery
Discovery score	95% Best	60%	50%
CLI exists	Passed5/5dg Best	Passed5/5openrouter Best	Passed5/5firectl Best
Install docs	Passed5/5pip Best	Passed5/5npm Best	Passed5/5binary download Best
Auth method	Passed25/25env var Best	Passed25/25env var Best	Passed25/25env var Best
Non-interactive	Passed25/25auto detect Best	Failed0/25none	Failed0/25none
JSON output	Passed15/15full Best	Failed0/15none	Passed15/15full Best
llms.txt	Failed0/5	Passed5/5 Best	Failed0/5
Agent skill	Passed20/20 Best	Passed20/20 Best	Failed0/20
Eval Details
Integration task	# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...	# Task Using the openrouter CLI (OpenRouter), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Ser...	# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...
Generated files	1	1	1
Trace tool calls	11	19	29
Completed	9/27/2026	9/27/2026	9/27/2026
Stderr	None captured	None captured	None captured