Chroma
All 1 ran the same Vector Databases task · Same Vector Databases task · View the taskView the task
CompanyClick a name to swap it
#3 in Vector Databases · #22 overall
80B
Compare up to 4. Next in Vector Databases:
MCP task scoring
Task score
76%
Task
Passed+30The agent successfully completed all required steps using only MCP tools from the Chroma server. Step 1: Tool discovery...
Used MCP Correctly
Passed+20Agent made 5 MCP tool calls
Partial Credit
Passed+10Partial credit: 100%
Zero Errors
Failed+0Failed tool calls: 1
Efficiency
Partial+67 tool calls (<=10, great)
Time
Failed+0612s (>=10min, too slow)
Cost
Passed+10$0.09 (<$0.10, excellent)
Runtime (actual)Lower is better
10m 11s
Cost (actual)Lower is better
$0.0930
Tool callsLower is better
7
ErrorsLower is better
1
Tokens usedLower is better
169,880
MCP discovery
Discovery score
90%
Official MCP
Passed5/5
Auth Method
Passed15/15env var
Tool Count
Passed10/1013 tools, lean
Tool Description Length
Partial7/103,131 tokens, moderate
llms.txt
Passed10/10
Output Schema
Failed0/5
Tool Stability
Passed5/5
Security Posture
Partial9/10No API keys, tokens, or credentials visible in descriptions or schemas; parameters are properly typed; however, chroma_d...
Tool API Mapping
Scored15/15All 13 tools map cleanly to individual Chroma API operations (create, read, update, delete, list, fork, peek, query, mod...
Description Clarity
Partial14/15Descriptions are comprehensive and clear, explaining purpose, parameters, return values, and edge cases; the chroma_get_...
Eval details
Integration task
# Task
Using the Chroma MCP, perform vector database operations:
1. Discover available MCP tools and their capabilities
2. Create a new collection/index called "faq-*****"
3. Lis...
Generated files
1
Trace tool calls
7
Completed
10/14/2026
Stderr
None captured
Full eval
#3 in Vector Databases
80B
MCP task scoring
Task score
76%
Task
Passed+30The agent successfully completed all required steps using only MCP tools from the Chroma server. Step 1: Tool discovery...
Used MCP Correctly
Passed+20Agent made 5 MCP tool calls
Partial Credit
Passed+10Partial credit: 100%
Zero Errors
Failed+0Failed tool calls: 1
Efficiency
Partial+67 tool calls (<=10, great)
Time
Failed+0612s (>=10min, too slow)
Cost
Passed+10$0.09 (<$0.10, excellent)
Runtime (actual)Lower is better
10m 11s
Cost (actual)Lower is better
$0.0930
Tool callsLower is better
7
ErrorsLower is better
1
Tokens usedLower is better
169,880
MCP discovery
Discovery score
90%
Official MCP
Passed5/5
Auth Method
Passed15/15env var
Tool Count
Passed10/1013 tools, lean
Tool Description Length
Partial7/103,131 tokens, moderate
llms.txt
Passed10/10
Output Schema
Failed0/5
Tool Stability
Passed5/5
Security Posture
Partial9/10No API keys, tokens, or credentials visible in descriptions or schemas; parameters are properly typed; however, chroma_d...
Tool API Mapping
Scored15/15All 13 tools map cleanly to individual Chroma API operations (create, read, update, delete, list, fork, peek, query, mod...
Description Clarity
Partial14/15Descriptions are comprehensive and clear, explaining purpose, parameters, return values, and edge cases; the chroma_get_...
Eval details
Integration task
# Task
Using the Chroma MCP, perform vector database operations:
1. Discover available MCP tools and their capabilities
2. Create a new collection/index called "faq-*****"
3. Lis...
Generated files
1
Trace tool calls
7
Completed
10/14/2026
Stderr
None captured
Full eval