Compare companies

Side-by-side CLI results for companies evaluated by Codex. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Selected

Company 1

Company 2

Stripe

Payment

Fireworks AI

Inference

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Payment

Inference

Overall rank

#10

Best

#51

Category rank

Best

Full eval

Open report

CLI Task Scoring

Eval score

74%

Best

15%

Task Completed

Passed+30The agent successfully completed all required tasks using the official Stripe CLI. The trace shows: (1) installation of...

Best

Failed+0The agent successfully extracted the structured JSON data with all required fields (invoice_number, vendor, date, line_i...

Used CLI Correctly

Passed+20Agent used CLI commands correctly

Best

Failed+0CLI usage issues: Used curl with raw HTTP API calls instead of firectl CLI for the final inference call, firectl session...

Partial Credit

Passed+10Partial credit: 100%

Best

Partial+3Partial credit: 30%

Zero Errors

Failed+0Failed tool calls: 2

Best

Failed+0Failed tool calls: 4

Best

Efficiency

Partial+2.017 tool calls (<=20, acceptable)

Best

Failed+029 tool calls (>25, inefficient)

Time

Partial+6106s (1-2min, great)

Best

Partial+6106s (1-2min, great)

Best

Cost

Partial+6$0.11 ($0.10-0.20, great)

Best

Partial+6$0.20 ($0.10-0.20, great)

Best

Runtime (actual)

1m 46s

Best

Cost (actual)

$0.1091

Best

$0.1953

Tool calls

Best

Errors

Best

Tokens used

427,110

Best

730,987

CLI Discovery

Discovery score

93%

Best

50%

CLI exists

Passed5/5stripe

Best

Passed5/5firectl

Best

Install docs

Passed5/5binary download

Best

Passed5/5binary download

Best

Auth method

Passed25/25env var

Best

Passed25/25env var

Best

Non-interactive

Partial18/25flag

Best

Failed0/25none

JSON output

Passed15/15default json

Best

Passed15/15full

Best

llms.txt

Passed5/5

Best

Failed0/5

Agent skill

Passed20/20

Best

Failed0/20

Eval Details

Integration task

# Task Using the stripe CLI (Stripe), create payment resources: 1. Create a product called "Pro Plan" using the CLI 2. Create a price of $29/month (recurring) for that product 3....

# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...

Generated files

Trace tool calls

Completed

9/27/2026

Stderr

None captured

Metric	Stripe Payment	Fireworks AI Inference
Summary
Compare status	Comparable	Comparable
Overall score	79 Best	25
Grade	B	D
Category	Payment	Inference
Overall rank	#10 Best	#51
Category rank	#1 Best	#3
Full eval	Open report	Open report
CLI Task Scoring
Eval score	74% Best	15%
Task Completed	Passed+30The agent successfully completed all required tasks using the official Stripe CLI. The trace shows: (1) installation of... Best	Failed+0The agent successfully extracted the structured JSON data with all required fields (invoice_number, vendor, date, line_i...
Used CLI Correctly	Passed+20Agent used CLI commands correctly Best	Failed+0CLI usage issues: Used curl with raw HTTP API calls instead of firectl CLI for the final inference call, firectl session...
Partial Credit	Passed+10Partial credit: 100% Best	Partial+3Partial credit: 30%
Zero Errors	Failed+0Failed tool calls: 2 Best	Failed+0Failed tool calls: 4 Best
Efficiency	Partial+2.017 tool calls (<=20, acceptable) Best	Failed+029 tool calls (>25, inefficient)
Time	Partial+6106s (1-2min, great) Best	Partial+6106s (1-2min, great) Best
Cost	Partial+6$0.11 ($0.10-0.20, great) Best	Partial+6$0.20 ($0.10-0.20, great) Best
Runtime (actual)	1m 46s	1m 46s Best
Cost (actual)	$0.1091 Best	$0.1953
Tool calls	17 Best	29
Errors	2 Best	4
Tokens used	427,110 Best	730,987
CLI Discovery
Discovery score	93% Best	50%
CLI exists	Passed5/5stripe Best	Passed5/5firectl Best
Install docs	Passed5/5binary download Best	Passed5/5binary download Best
Auth method	Passed25/25env var Best	Passed25/25env var Best
Non-interactive	Partial18/25flag Best	Failed0/25none
JSON output	Passed15/15default json Best	Passed15/15full Best
llms.txt	Passed5/5 Best	Failed0/5
Agent skill	Passed20/20 Best	Failed0/20
Eval Details
Integration task	# Task Using the stripe CLI (Stripe), create payment resources: 1. Create a product called "Pro Plan" using the CLI 2. Create a price of $29/month (recurring) for that product 3....	# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...
Generated files	1	1
Trace tool calls	17	29
Completed	9/27/2026	9/27/2026
Stderr	None captured	None captured