Back to leaderboard

Scalekit

Same Auth task · View

#4 in Auth

32D

CLI task scoring

Eval score
32%
Task Completed
Failed+0The agent successfully installed and ran the official @scalekit-sdk/dryrun CLI tool with correct credentials, and verifi...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Partial+6.5Partial credit: 65%
Zero Errors
Failed+0Failed tool calls: 1
Efficiency
Partial+1.025 tool calls (<=25, below average)
Time
Partial+1.0328s (5-10min, very slow)
Cost
Partial+3.5$0.38 ($0.20-0.40, good)
Runtime (actual)Lower is better
5m 27s
Cost (actual)Lower is better
$0.3789
Tool callsLower is better
25
ErrorsLower is better
1
Tokens usedLower is better
825,747

CLI discovery

Discovery score
33%
CLI exists
Passed5/5npx @scalekit-sdk/dryrun
Install docs
Passed5/5npx
Auth method
Partial3/25oauth
Non-interactive
Failed0/25none
JSON output
Failed0/15none
llms.txt
Failed0/5
Agent skill
Passed20/20

Eval details

Integration task
# Task Using the @scalekit-sdk/dryrun CLI (Scalekit), ***** an SSO/OIDC authentication flow: 1. Run `npx @scalekit-sdk/dryrun` with the Scalekit environment credentials (SCALEKIT...
Generated files
1
Trace tool calls
25
Completed
9/27/2026
Stderr
None captured
Full eval