Compare companies

Side-by-side MCP results for companies evaluated by Codex. Compare score, rank, discoverability, cost, errors, runtime, and the underlying eval artifacts.

Model

Claude Code Codex

Codex

Tooling

API CLI MCP

Choose 2-4 companies

Selected

Company 1

Company 2

Showing the first two companies on mobile. Use a wider screen to compare 3 companies.

Selected

Company 1

Company 2

Company 3

Mobile compare is limited to two companies. Use a wider screen to compare all 3.

Deepgram

Voice STT

Agentmail

Summary

Compare status

Comparable

Overall score

Best

Grade

Category

Voice STT

Overall rank

#47

#13

Best

Category rank

Best

Full eval

Open report

MCP Task Scoring

Task score

31%

83%

Best

Task

Failed+0The agent correctly attempted to use MCP tools as required (mcp__deepgram__list_mcp_resources, mcp__codex__list_mcp_reso...

Passed+30The agent successfully completed all required tasks using only MCP tools. It: (1) discovered and listed available MCP to...

Best

Used MCP Correctly

Passed+20Agent made 5 MCP tool calls

Best

Passed+20Agent made 4 MCP tool calls

Best

Partial Credit

Failed+0Partial credit: 0%

Passed+10Partial credit: 100%

Best

Zero Errors

Failed+0Tool outputs with errors: 5

Passed+10No errors

Best

Efficiency

Partial+3.515 tool calls (<=15, good)

Partial+66 tool calls (<=10, great)

Best

Time

Partial+2.0242s (3-5min, slow)

Best

Partial+1.0301s (5-10min, very slow)

Cost

Partial+6$0.12 ($0.10-0.20, great)

Best

Partial+6$0.12 ($0.10-0.20, great)

Best

Runtime (actual)

4m 1s

Best

5m 0s

Cost (actual)

$0.1215

Best

$0.1215

Tool calls

Best

Errors

Best

Tokens used

567,330

305,245

Best

MCP Discovery

Discovery score

55%

75%

Best

Official MCP

Passed5/5

Best

Passed5/5

Best

Auth Method

Passed15/15env var

Best

Passed15/15env var

Best

Tool Count

Passed10/103 tools, lean

Best

Partial3/1041 tools, heavy

Tool Description Length

Passed10/1029 tokens, lean

Best

Failed0/1013,234 tokens, bloated

llms.txt

Passed10/10

Best

Passed10/10

Best

Output Schema

Failed0/5

Best

Failed0/5

Best

Tool Stability

Passed5/5

Best

Passed5/5

Best

Security Posture

—

Partial9/10No API keys or tokens visible in descriptions or schemas; destructive operations (delete_inbox, delete_thread, delete_dr...

Tool API Mapping

—

Partial14/15Tools map cleanly 1:1 to API operations with clear, granular operations (list, search, get, create, update, delete patte...

Description Clarity

—

Partial14/15Descriptions are comprehensive and specific about what each tool does, with clear guidance on inputs, outputs, and usage...

Eval Details

Integration task

# Task Using the Deepgram MCP, transcribe audio. 1. Discover the MCP tools relevant to transcription 2. Transcribe this sample audio file: https://storage.googleapis.com/aai-w...

# Task Using the Agentmail MCP, perform email operations: 1. Discover available MCP tools and their capabilities 2. Create a new email inbox (or list available inboxes/domains) 3...

Generated files

Trace tool calls

Completed

9/27/2026

10/14/2026

Stderr

None captured

Metric	Deepgram Voice STT	Agentmail Email	Resend Email
Summary
Compare status	Comparable	Comparable	Comparable
Overall score	38	80 Best	79
Grade	D	B	B
Category	Voice STT	Email	Email
Overall rank	#47	#13 Best	#15
Category rank	#1 Best	#1 Best	#2
Full eval	Open report	Open report	Open report
MCP Task Scoring
Task score	31%	83%	84% Best
Task	Failed+0The agent correctly attempted to use MCP tools as required (mcp__deepgram__list_mcp_resources, mcp__codex__list_mcp_reso...	Passed+30The agent successfully completed all required tasks using only MCP tools. It: (1) discovered and listed available MCP to... Best	Passed+30The agent successfully completed all required tasks using only MCP tools. It discovered MCP capabilities, listed availab... Best
Used MCP Correctly	Passed+20Agent made 5 MCP tool calls Best	Passed+20Agent made 4 MCP tool calls Best	Passed+20Agent made 5 MCP tool calls Best
Partial Credit	Failed+0Partial credit: 0%	Passed+10Partial credit: 100% Best	Passed+10Partial credit: 100% Best
Zero Errors	Failed+0Tool outputs with errors: 5	Passed+10No errors Best	Passed+10No errors Best
Efficiency	Partial+3.515 tool calls (<=15, good)	Partial+66 tool calls (<=10, great) Best	Partial+67 tool calls (<=10, great) Best
Time	Partial+2.0242s (3-5min, slow) Best	Partial+1.0301s (5-10min, very slow)	Partial+2.0215s (3-5min, slow) Best
Cost	Partial+6$0.12 ($0.10-0.20, great) Best	Partial+6$0.12 ($0.10-0.20, great) Best	Partial+6$0.11 ($0.10-0.20, great) Best
Runtime (actual)	4m 1s	5m 0s	3m 34s Best
Cost (actual)	$0.1215	$0.1215	$0.1147 Best
Tool calls	15	6 Best	7
Errors	5	0 Best	0 Best
Tokens used	567,330	305,245 Best	372,658
MCP Discovery
Discovery score	55%	75% Best	70%
Official MCP	Passed5/5 Best	Passed5/5 Best	Passed5/5 Best
Auth Method	Passed15/15env var Best	Passed15/15env var Best	Passed15/15env var Best
Tool Count	Passed10/103 tools, lean Best	Partial3/1041 tools, heavy	Failed0/10106 tools, bloated
Tool Description Length	Passed10/1029 tokens, lean Best	Failed0/1013,234 tokens, bloated	Failed0/1029,709 tokens, bloated
llms.txt	Passed10/10 Best	Passed10/10 Best	Passed10/10 Best
Output Schema	Failed0/5 Best	Failed0/5 Best	Failed0/5 Best
Tool Stability	Passed5/5 Best	Passed5/5 Best	Passed5/5 Best
Security Posture	—	Partial9/10No API keys or tokens visible in descriptions or schemas; destructive operations (delete_inbox, delete_thread, delete_dr... Best	Partial9/10No API keys or tokens visible in tool descriptions or schemas; parameters are clearly typed with validation; minor conce... Best
Tool API Mapping	—	Partial14/15Tools map cleanly 1:1 to API operations with clear, granular operations (list, search, get, create, update, delete patte... Best	Partial12/15Most tools map 1:1 to individual API operations (e.g., create_contact, get_broadcast, add_suppression), but create-autom...
Description Clarity	—	Partial14/15Descriptions are comprehensive and specific about what each tool does, with clear guidance on inputs, outputs, and usage... Best	Partial14/15Tool descriptions are comprehensive and clearly explain purpose, when to use, expected inputs/outputs, and workflows; mi... Best
Eval Details
Integration task	# Task Using the Deepgram MCP, transcribe audio. 1. Discover the MCP tools relevant to transcription 2. Transcribe this sample audio file: https://storage.googleapis.com/aai-w...	# Task Using the Agentmail MCP, perform email operations: 1. Discover available MCP tools and their capabilities 2. Create a new email inbox (or list available inboxes/domains) 3...	# Task Using the Resend MCP, perform email operations: 1. Discover available MCP tools and their capabilities 2. Create a new email inbox (or list available inboxes/domains) 3. S...
Generated files	1	1	1
Trace tool calls	15	6	7
Completed	9/27/2026	10/14/2026	10/14/2026
Stderr	None captured	None captured	None captured