Compare companies

Side-by-side CLI results for companies evaluated by Codex. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Showing the first two companies on mobile. Use a wider screen to compare 3 companies.

Selected

Company 1

Company 2

Company 3

Mobile compare is limited to two companies. Use a wider screen to compare all 3.

AssemblyAI

Voice STT

Fireworks AI

Inference

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Voice STT

Inference

Overall rank

#15

Best

#51

Category rank

Best

Full eval

Open report

CLI Task Scoring

Eval score

79%

Best

15%

Task Completed

Passed+30The agent successfully completed all required steps: (1) downloaded the sample audio file using curl, (2) installed the...

Best

Failed+0The agent successfully extracted the structured JSON data with all required fields (invoice_number, vendor, date, line_i...

Used CLI Correctly

Passed+20Agent used CLI commands correctly

Best

Failed+0CLI usage issues: Used curl with raw HTTP API calls instead of firectl CLI for the final inference call, firectl session...

Partial Credit

Passed+10Partial credit: 100%

Best

Partial+3Partial credit: 30%

Zero Errors

Failed+0Failed tool calls: 2

Best

Failed+0Failed tool calls: 4

Best

Efficiency

Partial+3.512 tool calls (<=15, good)

Best

Failed+029 tool calls (>25, inefficient)

Time

Partial+6105s (1-2min, great)

Best

Partial+6106s (1-2min, great)

Best

Cost

Passed+10$0.10 (<$0.10, excellent)

Best

Partial+6$0.20 ($0.10-0.20, great)

Runtime (actual)

1m 44s

Best

1m 46s

Cost (actual)

$0.0972

Best

$0.1953

Tool calls

Best

Errors

Best

Tokens used

381,257

Best

730,987

CLI Discovery

Discovery score

70%

Best

50%

CLI exists

Passed5/5assemblyai

Best

Passed5/5firectl

Best

Install docs

Passed5/5curl script

Best

Passed5/5binary download

Best

Auth method

Passed25/25api key cmd

Best

Passed25/25env var

Best

Non-interactive

Failed0/25none

Best

Failed0/25none

Best

JSON output

Passed15/15full

Best

Passed15/15full

Best

llms.txt

Failed0/5

Best

Failed0/5

Best

Agent skill

Passed20/20

Best

Failed0/20

Eval Details

Integration task

# Task Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sport...

# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...

Generated files

Trace tool calls

Completed

9/27/2026

Stderr

None captured

Metric	AssemblyAI Voice STT	Fireworks AI Inference	Deepgram Voice STT
Summary
Compare status	Comparable	Comparable	Comparable
Overall score	76	25	88 Best
Grade	B	D	B
Category	Voice STT	Inference	Voice STT
Overall rank	#15	#51	#4 Best
Category rank	#2	#3	#1 Best
Full eval	Open report	Open report	Open report
CLI Task Scoring
Eval score	79%	15%	85% Best
Task Completed	Passed+30The agent successfully completed all required steps: (1) downloaded the sample audio file using curl, (2) installed the... Best	Failed+0The agent successfully extracted the structured JSON data with all required fields (invoice_number, vendor, date, line_i...	Passed+30The agent successfully completed all required steps: (1) downloaded the sample audio file using curl, (2) installed the... Best
Used CLI Correctly	Passed+20Agent used CLI commands correctly Best	Failed+0CLI usage issues: Used curl with raw HTTP API calls instead of firectl CLI for the final inference call, firectl session...	Passed+20Agent used CLI commands correctly Best
Partial Credit	Passed+10Partial credit: 100% Best	Partial+3Partial credit: 30%	Passed+10Partial credit: 100% Best
Zero Errors	Failed+0Failed tool calls: 2	Failed+0Failed tool calls: 4	Passed+10No errors Best
Efficiency	Partial+3.512 tool calls (<=15, good) Best	Failed+029 tool calls (>25, inefficient)	Partial+3.511 tool calls (<=15, good) Best
Time	Partial+6105s (1-2min, great) Best	Partial+6106s (1-2min, great) Best	Partial+6103s (1-2min, great) Best
Cost	Passed+10$0.10 (<$0.10, excellent) Best	Partial+6$0.20 ($0.10-0.20, great)	Partial+6$0.11 ($0.10-0.20, great)
Runtime (actual)	1m 44s	1m 46s	1m 42s Best
Cost (actual)	$0.0972 Best	$0.1953	$0.1075
Tool calls	12	29	11 Best
Errors	2	4	0 Best
Tokens used	381,257	730,987	374,221 Best
CLI Discovery
Discovery score	70%	50%	95% Best
CLI exists	Passed5/5assemblyai Best	Passed5/5firectl Best	Passed5/5dg Best
Install docs	Passed5/5curl script Best	Passed5/5binary download Best	Passed5/5pip Best
Auth method	Passed25/25api key cmd Best	Passed25/25env var Best	Passed25/25env var Best
Non-interactive	Failed0/25none	Failed0/25none	Passed25/25auto detect Best
JSON output	Passed15/15full Best	Passed15/15full Best	Passed15/15full Best
llms.txt	Failed0/5 Best	Failed0/5 Best	Failed0/5 Best
Agent skill	Passed20/20 Best	Failed0/20	Passed20/20 Best
Eval Details
Integration task	# Task Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sport...	# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...	# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...
Generated files	1	1	1
Trace tool calls	12	29	11
Completed	9/27/2026	9/27/2026	9/27/2026
Stderr	None captured	None captured	None captured