Back to leaderboard

Fireworks AI

Same Inference task · View

#3 in Inference

25D

CLI task scoring

Eval score
14%
Task Completed
Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...
Used CLI Correctly
Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...
Partial Credit
Partial+4.0Partial credit: 40%
Zero Errors
Failed+0Failed tool calls: 2
Efficiency
Partial+3.515 tool calls (<=15, good)
Time
Partial+3.5138s (2-3min, good)
Cost
Partial+3.5$0.22 ($0.20-0.40, good)
Runtime (actual)Lower is better
2m 18s
Cost (actual)Lower is better
$0.2219
Tool callsLower is better
15
ErrorsLower is better
2
Tokens usedLower is better
361,995

CLI discovery

Discovery score
50%
CLI exists
Passed5/5firectl
Install docs
Passed5/5binary download
Auth method
Passed25/25env var
Non-interactive
Failed0/25none
JSON output
Passed15/15full
llms.txt
Failed0/5
Agent skill
Failed0/20

Eval details

Integration task
# Task Using the firectl CLI (Fireworks AI), extract structured data from this invoice text: --- INVOICE #INV-2024-0042 From: Acme Corp Date: 2024-03-15 Items: - Consulting Serv...
Generated files
1
Trace tool calls
15
Completed
9/27/2026
Stderr
None captured
Full eval