Compare companies

Side-by-side CLI results for companies evaluated by Claude Code. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Selected

Company 1

Company 2

Stripe

Payment

Fireworks AI

Inference

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Payment

Inference

Overall rank

Best

#51

Category rank

Best

Full eval

Open report

CLI Task Scoring

Eval score

75%

Best

14%

Task Completed

Passed+30The agent successfully completed all required steps using the official Stripe CLI. Step [4] installed the stripe CLI via...

Best

Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...

Used CLI Correctly

Passed+20Agent used CLI commands correctly

Best

Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...

Partial Credit

Passed+10Partial credit: 100%

Best

Partial+4.0Partial credit: 40%

Zero Errors

Failed+0Failed tool calls: 1

Best

Failed+0Failed tool calls: 2

Best

Efficiency

Partial+3.511 tool calls (<=15, good)

Best

Partial+3.515 tool calls (<=15, good)

Best

Time

Partial+683s (1-2min, great)

Best

Partial+3.5138s (2-3min, good)

Cost

Partial+6$0.17 ($0.10-0.20, great)

Best

Partial+3.5$0.22 ($0.20-0.40, good)

Runtime (actual)

1m 23s

Best

2m 18s

Cost (actual)

$0.1656

Best

$0.2219

Tool calls

Best

Errors

Best

Tokens used

293,058

Best

361,995

CLI Discovery

Discovery score

93%

Best

50%

CLI exists

Passed5/5stripe

Best

Passed5/5firectl

Best

Install docs

Passed5/5binary download

Best

Passed5/5binary download

Best

Auth method

Passed25/25env var

Best

Passed25/25env var

Best

Non-interactive

Partial18/25flag

Best

Failed0/25none

JSON output

Passed15/15default json

Best

Passed15/15full

Best

llms.txt

Passed5/5

Best

Failed0/5

Agent skill

Passed20/20

Best

Failed0/20

Eval Details

Integration task

# Task Using the stripe CLI (Stripe), create payment resources: 1. Create a product called "Pro Plan" using the CLI 2. Create a price of $29/month (recurring) for that product 3....

# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...

Generated files

Trace tool calls

Completed

9/27/2026

Stderr

None captured

Metric	Stripe Payment	Fireworks AI Inference
Summary
Compare status	Comparable	Comparable
Overall score	80 Best	25
Grade	B	D
Category	Payment	Inference
Overall rank	#9 Best	#51
Category rank	#1 Best	#3
Full eval	Open report	Open report
CLI Task Scoring
Eval score	75% Best	14%
Task Completed	Passed+30The agent successfully completed all required steps using the official Stripe CLI. Step [4] installed the stripe CLI via... Best	Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...
Used CLI Correctly	Passed+20Agent used CLI commands correctly Best	Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...
Partial Credit	Passed+10Partial credit: 100% Best	Partial+4.0Partial credit: 40%
Zero Errors	Failed+0Failed tool calls: 1 Best	Failed+0Failed tool calls: 2 Best
Efficiency	Partial+3.511 tool calls (<=15, good) Best	Partial+3.515 tool calls (<=15, good) Best
Time	Partial+683s (1-2min, great) Best	Partial+3.5138s (2-3min, good)
Cost	Partial+6$0.17 ($0.10-0.20, great) Best	Partial+3.5$0.22 ($0.20-0.40, good)
Runtime (actual)	1m 23s Best	2m 18s
Cost (actual)	$0.1656 Best	$0.2219
Tool calls	11 Best	15
Errors	1 Best	2
Tokens used	293,058 Best	361,995
CLI Discovery
Discovery score	93% Best	50%
CLI exists	Passed5/5stripe Best	Passed5/5firectl Best
Install docs	Passed5/5binary download Best	Passed5/5binary download Best
Auth method	Passed25/25env var Best	Passed25/25env var Best
Non-interactive	Partial18/25flag Best	Failed0/25none
JSON output	Passed15/15default json Best	Passed15/15full Best
llms.txt	Passed5/5 Best	Failed0/5
Agent skill	Passed20/20 Best	Failed0/20
Eval Details
Integration task	# Task Using the stripe CLI (Stripe), create payment resources: 1. Create a product called "Pro Plan" using the CLI 2. Create a price of $29/month (recurring) for that product 3....	# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...
Generated files	1	1
Trace tool calls	11	15
Completed	9/27/2026	9/27/2026
Stderr	None captured	None captured