Back to leaderboard

Chroma

Same Vector Databases task · View

#3 in Vector Databases

80B

MCP task scoring

Task score
76%
Task
Passed+30The agent successfully completed all required steps using only MCP tools from the Chroma server. Step 1: Tool discovery...
Used MCP Correctly
Passed+20Agent made 5 MCP tool calls
Partial Credit
Passed+10Partial credit: 100%
Zero Errors
Failed+0Failed tool calls: 1
Efficiency
Partial+67 tool calls (<=10, great)
Time
Failed+0612s (>=10min, too slow)
Cost
Passed+10$0.09 (<$0.10, excellent)
Runtime (actual)Lower is better
10m 11s
Cost (actual)Lower is better
$0.0930
Tool callsLower is better
7
ErrorsLower is better
1
Tokens usedLower is better
169,880

MCP discovery

Discovery score
90%
Official MCP
Passed5/5
Auth Method
Passed15/15env var
Tool Count
Passed10/1013 tools, lean
Tool Description Length
Partial7/103,131 tokens, moderate
llms.txt
Passed10/10
Output Schema
Failed0/5
Tool Stability
Passed5/5
Security Posture
Partial9/10No API keys, tokens, or credentials visible in descriptions or schemas; parameters are properly typed; however, chroma_d...
Tool API Mapping
Scored15/15All 13 tools map cleanly to individual Chroma API operations (create, read, update, delete, list, fork, peek, query, mod...
Description Clarity
Partial14/15Descriptions are comprehensive and clear, explaining purpose, parameters, return values, and edge cases; the chroma_get_...

Eval details

Integration task
# Task Using the Chroma MCP, perform vector database operations: 1. Discover available MCP tools and their capabilities 2. Create a new collection/index called "faq-*****" 3. Lis...
Generated files
1
Trace tool calls
7
Completed
10/14/2026
Stderr
None captured
Full eval