Compare companies

Side-by-side CLI results for companies evaluated by Claude Code. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Showing the first two companies on mobile. Use a wider screen to compare 3 companies.

Selected

Company 1

Company 2

Company 3

Mobile compare is limited to two companies. Use a wider screen to compare all 3.

AssemblyAI

Voice STT

Fireworks AI

Inference

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Voice STT

Inference

Overall rank

#17

Best

#51

Category rank

Best

Full eval

Open report

CLI Task Scoring

Eval score

78%

Best

14%

Task Completed

Passed+30The agent successfully completed all required steps: (1) installed the official AssemblyAI CLI from the correct source,...

Best

Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...

Used CLI Correctly

Passed+20Agent used CLI commands correctly

Best

Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...

Partial Credit

Passed+10Partial credit: 100%

Best

Partial+4.0Partial credit: 40%

Zero Errors

Failed+0Failed tool calls: 2

Best

Failed+0Failed tool calls: 2

Best

Efficiency

Partial+68 tool calls (<=10, great)

Best

Partial+3.515 tool calls (<=15, good)

Time

Partial+687s (1-2min, great)

Best

Partial+3.5138s (2-3min, good)

Cost

Partial+6$0.13 ($0.10-0.20, great)

Best

Partial+3.5$0.22 ($0.20-0.40, good)

Runtime (actual)

1m 27s

Best

2m 18s

Cost (actual)

$0.1278

Best

$0.2219

Tool calls

Best

Errors

Best

Tokens used

212,720

Best

361,995

CLI Discovery

Discovery score

70%

Best

50%

CLI exists

Passed5/5assemblyai

Best

Passed5/5firectl

Best

Install docs

Passed5/5curl script

Best

Passed5/5binary download

Best

Auth method

Passed25/25api key cmd

Best

Passed25/25env var

Best

Non-interactive

Failed0/25none

Best

Failed0/25none

Best

JSON output

Passed15/15full

Best

Passed15/15full

Best

llms.txt

Failed0/5

Best

Failed0/5

Best

Agent skill

Passed20/20

Best

Failed0/20

Eval Details

Integration task

# Task Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sport...

# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...

Generated files

Trace tool calls

Completed

9/27/2026

Stderr

None captured

Metric	AssemblyAI Voice STT	Fireworks AI Inference	Deepgram Voice STT
Summary
Compare status	Comparable	Comparable	Comparable
Overall score	75	25	92 Best
Grade	B	D	A
Category	Voice STT	Inference	Voice STT
Overall rank	#17	#51	#1 Best
Category rank	#2	#3	#1 Best
Full eval	Open report	Open report	Open report
CLI Task Scoring
Eval score	78%	14%	92% Best
Task Completed	Passed+30The agent successfully completed all required steps: (1) installed the official AssemblyAI CLI from the correct source,... Best	Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...	Passed+30The agent successfully completed all task requirements: (1) installed the official `dg` CLI via `pip install deepctl`, (... Best
Used CLI Correctly	Passed+20Agent used CLI commands correctly Best	Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...	Passed+20Agent used CLI commands correctly Best
Partial Credit	Passed+10Partial credit: 100% Best	Partial+4.0Partial credit: 40%	Passed+10Partial credit: 100% Best
Zero Errors	Failed+0Failed tool calls: 2	Failed+0Failed tool calls: 2	Passed+10No errors Best
Efficiency	Partial+68 tool calls (<=10, great) Best	Partial+3.515 tool calls (<=15, good)	Partial+67 tool calls (<=10, great) Best
Time	Partial+687s (1-2min, great) Best	Partial+3.5138s (2-3min, good)	Partial+6109s (1-2min, great) Best
Cost	Partial+6$0.13 ($0.10-0.20, great)	Partial+3.5$0.22 ($0.20-0.40, good)	Passed+10$0.10 (<$0.10, excellent) Best
Runtime (actual)	1m 27s Best	2m 18s	1m 49s
Cost (actual)	$0.1278	$0.2219	$0.0959 Best
Tool calls	8	15	7 Best
Errors	2	2	0 Best
Tokens used	212,720	361,995	181,457 Best
CLI Discovery
Discovery score	70%	50%	95% Best
CLI exists	Passed5/5assemblyai Best	Passed5/5firectl Best	Passed5/5dg Best
Install docs	Passed5/5curl script Best	Passed5/5binary download Best	Passed5/5pip Best
Auth method	Passed25/25api key cmd Best	Passed25/25env var Best	Passed25/25env var Best
Non-interactive	Failed0/25none	Failed0/25none	Passed25/25auto detect Best
JSON output	Passed15/15full Best	Passed15/15full Best	Passed15/15full Best
llms.txt	Failed0/5 Best	Failed0/5 Best	Failed0/5 Best
Agent skill	Passed20/20 Best	Failed0/20	Passed20/20 Best
Eval Details
Integration task	# Task Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sport...	# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...	# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...
Generated files	1	1	1
Trace tool calls	8	15	7
Completed	9/27/2026	9/27/2026	9/27/2026
Stderr	None captured	None captured	None captured