Back to leaderboard

Nylas vs Agentmail

Same Email task · View

#3 in Email

36DBest on 1 of 22

#1 in Email

84BBest on 11 of 22

MCP task scoring

Task score
33%
88%
Task
Failed+0The agent correctly identified and attempted to use MCP tools for all operations as required. However, the task could no...
Passed+30The agent successfully completed all required tasks using only Agentmail MCP tools. It discovered available tools, liste...
Used MCP Correctly
Passed+20Agent made 0 MCP tool calls
Passed+20Agent made 5 MCP tool calls
Partial Credit
Failed+0Partial credit: 0%
Passed+10Partial credit: 100%
Zero Errors
Failed+0Tool outputs with errors: 2
Passed+10No errors
Efficiency
Partial+68 tool calls (<=10, great)
Partial+67 tool calls (<=10, great)
Time
Partial+1.0358s (5-10min, very slow)
Partial+2.0280s (3-5min, slow)
Cost
Partial+6$0.14 ($0.10-0.20, great)
Passed+10$0.10 (<$0.10, excellent)
Runtime (actual)Lower is better
5m 57s
4m 39s
Cost (actual)Lower is better
$0.1440
$0.0965
Tool callsLower is better
8
7
ErrorsLower is better
2
0
Tokens usedLower is better
243,767
180,827

MCP discovery

Discovery score
45%
75%
Official MCP
Passed5/5
Passed5/5
Auth Method
Passed15/15env var
Passed15/15env var
Tool Count
Failed0/100 tools, bloated
Partial3/1041 tools, heavy
Tool Description Length
Passed10/100 tokens, lean
Failed0/1013,234 tokens, bloated
llms.txt
Passed10/10
Passed10/10
Output Schema
Failed0/5
Failed0/5
Tool Stability
Passed5/5
Passed5/5
Security Posture
—
Partial9/10No API keys or tokens are exposed in tool descriptions or schemas; parameters are clearly typed and validated; however,...
Tool API Mapping
—
Partial14/15Tools map cleanly 1:1 to individual API operations (create_inbox, get_message, list_threads, etc.), with rare exceptions...
Description Clarity
—
Partial14/15Descriptions are comprehensive and explain tool purpose, parameters, expected outcomes, and edge cases clearly; agents c...

Eval details

Integration task
# Task Using the Nylas MCP, perform email operations: 1. Discover available MCP tools and their capabilities 2. Create a new email inbox (or list available inboxes/domains) 3. Se...
# Task Using the Agentmail MCP, perform email operations: 1. Discover available MCP tools and their capabilities 2. Create a new email inbox (or list available inboxes/domains) 3...
Generated files
1
1
Trace tool calls
8
7
Completed
9/27/2026
10/14/2026
Stderr
None captured
None captured