Chroma
All 1 ran the same Vector Databases task · Same Vector Databases task · View the taskView the task
CompanyClick a name to swap it
#2 in Vector Databases · #44 overall
35D
Compare up to 4. Next in Vector Databases:
CLI task scoring
Eval score
24%
Task Completed
Failed+0The agent correctly installed and used the official `chroma` CLI from the `chromadb` package (verified in trace [16] as...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Partial+1.5Partial credit: 15%
Zero Errors
Failed+0Failed tool calls: 12
Efficiency
Failed+036 tool calls (>25, inefficient)
Time
Partial+1.0474s (5-10min, very slow)
Cost
Partial+2.0$0.60 ($0.40-0.70, expensive)
Runtime (actual)Lower is better
7m 53s
Cost (actual)Lower is better
$0.5970
Tool callsLower is better
36
ErrorsLower is better
12
Tokens usedLower is better
1,322,436
CLI discovery
Discovery score
60%
CLI exists
Passed5/5chroma
Install docs
Passed5/5pip
Auth method
Passed25/25api key cmd
Non-interactive
Failed0/25none
JSON output
Failed0/15none
llms.txt
Passed5/5
Agent skill
Passed20/20
Eval details
Integration task
# Task
Using the chroma CLI (Chroma), perform vector database operations:
1. Create a new collection/index called "faq-*****"
2. List collections to verify it was created
3. Prin...
Generated files
1
Trace tool calls
36
Completed
9/27/2026
Stderr
None captured
Full eval
#2 in Vector Databases
35D
CLI task scoring
Eval score
24%
Task Completed
Failed+0The agent correctly installed and used the official `chroma` CLI from the `chromadb` package (verified in trace [16] as...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Partial+1.5Partial credit: 15%
Zero Errors
Failed+0Failed tool calls: 12
Efficiency
Failed+036 tool calls (>25, inefficient)
Time
Partial+1.0474s (5-10min, very slow)
Cost
Partial+2.0$0.60 ($0.40-0.70, expensive)
Runtime (actual)Lower is better
7m 53s
Cost (actual)Lower is better
$0.5970
Tool callsLower is better
36
ErrorsLower is better
12
Tokens usedLower is better
1,322,436
CLI discovery
Discovery score
60%
CLI exists
Passed5/5chroma
Install docs
Passed5/5pip
Auth method
Passed25/25api key cmd
Non-interactive
Failed0/25none
JSON output
Failed0/15none
llms.txt
Passed5/5
Agent skill
Passed20/20
Eval details
Integration task
# Task
Using the chroma CLI (Chroma), perform vector database operations:
1. Create a new collection/index called "faq-*****"
2. List collections to verify it was created
3. Prin...
Generated files
1
Trace tool calls
36
Completed
9/27/2026
Stderr
None captured
Full eval