Back to leaderboard

Chroma

Same Vector Databases task · View

#2 in Vector Databases

35D

CLI task scoring

Eval score
24%
Task Completed
Failed+0The agent correctly installed and used the official `chroma` CLI from the `chromadb` package (verified in trace [16] as...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Partial+1.5Partial credit: 15%
Zero Errors
Failed+0Failed tool calls: 12
Efficiency
Failed+036 tool calls (>25, inefficient)
Time
Partial+1.0474s (5-10min, very slow)
Cost
Partial+2.0$0.60 ($0.40-0.70, expensive)
Runtime (actual)Lower is better
7m 53s
Cost (actual)Lower is better
$0.5970
Tool callsLower is better
36
ErrorsLower is better
12
Tokens usedLower is better
1,322,436

CLI discovery

Discovery score
60%
CLI exists
Passed5/5chroma
Install docs
Passed5/5pip
Auth method
Passed25/25api key cmd
Non-interactive
Failed0/25none
JSON output
Failed0/15none
llms.txt
Passed5/5
Agent skill
Passed20/20

Eval details

Integration task
# Task Using the chroma CLI (Chroma), perform vector database operations: 1. Create a new collection/index called "faq-*****" 2. List collections to verify it was created 3. Prin...
Generated files
1
Trace tool calls
36
Completed
9/27/2026
Stderr
None captured
Full eval