Back to leaderboard

Agentmail vs Resend

Same Email task · View

#1 in Email

92ABest on 2 of 19

#2 in Email

91ABest on 4 of 19

CLI task scoring

Eval score
89%
88%
Task Completed
Passed+30The agent successfully completed all required tasks using the agentmail CLI correctly. Step 1: Listed inboxes (empty) th...
Passed+30The agent successfully completed all required task steps using the official `resend` CLI binary installed via `npm insta...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Passed+10Partial credit: 100%
Passed+10Partial credit: 100%
Zero Errors
Passed+10No errors
Passed+10No errors
Efficiency
Partial+3.512 tool calls (<=15, good)
Partial+69 tool calls (<=10, great)
Time
Passed+1060s (<60s, excellent)
Partial+680s (1-2min, great)
Cost
Partial+6$0.13 ($0.10-0.20, great)
Partial+6$0.12 ($0.10-0.20, great)
Runtime (actual)Lower is better
59s
1m 19s
Cost (actual)Lower is better
$0.1333
$0.1180
Tool callsLower is better
12
9
ErrorsLower is better
0
0
Tokens usedLower is better
340,255
248,805

CLI discovery

Discovery score
100%
100%
CLI exists
Passed5/5agentmail
Passed5/5resend
Install docs
Passed5/5npm
Passed5/5npm
Auth method
Passed25/25env var
Passed25/25env var
Non-interactive
Passed25/25auto detect
Passed25/25auto detect
JSON output
Passed15/15full
Passed15/15full
llms.txt
Passed5/5
Passed5/5
Agent skill
Passed20/20
Passed20/20

Eval details

Integration task
# Task Using the agentmail CLI (Agentmail), perform email operations: 1. Create a new email inbox (or list available inboxes) 2. Send an email from an inbox to "support@lightsage...
# Task Using the resend CLI (Resend), perform email operations: 1. Create a new email inbox (or list available inboxes) 2. Send an email from an inbox to a ***** email with subje...
Generated files
1
1
Trace tool calls
12
9
Completed
10/15/2026
9/27/2026
Stderr
None captured
None captured