Groq vs Fireworks AI
Both ran the same Inference task · Same Inference task · View the taskView the task
CompanyClick a name to swap it
#1 in Inference · #40 overall
38DBest on 4 of 19
#3 in Inference · #52 overall
25DBest on 9 of 19
Compare up to 4. Next in Inference:
CLI task scoring
Eval score
31%
14%
Task Completed
Failed+0The agent correctly installed and used the official `groq` CLI (groq-code-cli via npm), and attempted to invoke it for i...
Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...
Partial Credit
Partial+1.5Partial credit: 15%
Partial+4.0Partial credit: 40%
Zero Errors
Passed+10No errors
Failed+0Failed tool calls: 2
Efficiency
Failed+060 tool calls (>25, inefficient)
Partial+3.515 tool calls (<=15, good)
Time
Failed+0823s (>=10min, too slow)
Partial+3.5138s (2-3min, good)
Cost
Failed+0$1.07 (>=$1.00, too expensive)
Partial+3.5$0.22 ($0.20-0.40, good)
Runtime (actual)Lower is better
13m 42s
2m 18s
Cost (actual)Lower is better
$1.07
$0.2219
Tool callsLower is better
60
15
ErrorsLower is better
0
2
Tokens usedLower is better
3,010,847
361,995
CLI discovery
Discovery score
55%
50%
CLI exists
Passed5/5groq
Passed5/5firectl
Install docs
Passed5/5npm
Passed5/5binary download
Auth method
Passed25/25env var
Passed25/25env var
Non-interactive
Failed0/25none
Failed0/25none
JSON output
Failed0/15none
Passed15/15full
llms.txt
Failed0/5
Failed0/5
Agent skill
Passed20/20
Failed0/20
Eval details
Integration task
# Task
Using the groq CLI (Groq), extract structured data from this invoice text:
---
INVOICE #INV-2024-0042
From: Acme Corp
Date: 2024-03-15
Items:
- Consulting Services: $500....
# Task
Using the firectl CLI (Fireworks AI), extract structured data from this invoice text:
---
INVOICE #INV-2024-0042
From: Acme Corp
Date: 2024-03-15
Items:
- Consulting Serv...
Generated files
4
1
Trace tool calls
60
15
Completed
10/13/2026
9/27/2026
Stderr
None captured
None captured
#1 in Inference
38DBest on 4 of 19
#3 in Inference
25DBest on 9 of 19
CLI task scoring
Eval score
31%
14%
Task Completed
Failed+0The agent correctly installed and used the official `groq` CLI (groq-code-cli via npm), and attempted to invoke it for i...
Failed+0The agent installed firectl correctly and used it to authenticate and list serverless models, demonstrating proper CLI u...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Failed+0CLI usage issues: Did not use firectl CLI for the inference/chat completion call as required by the task, Switched to cu...
Partial Credit
Partial+1.5Partial credit: 15%
Partial+4.0Partial credit: 40%
Zero Errors
Passed+10No errors
Failed+0Failed tool calls: 2
Efficiency
Failed+060 tool calls (>25, inefficient)
Partial+3.515 tool calls (<=15, good)
Time
Failed+0823s (>=10min, too slow)
Partial+3.5138s (2-3min, good)
Cost
Failed+0$1.07 (>=$1.00, too expensive)
Partial+3.5$0.22 ($0.20-0.40, good)
Runtime (actual)Lower is better
13m 42s
2m 18s
Cost (actual)Lower is better
$1.07
$0.2219
Tool callsLower is better
60
15
ErrorsLower is better
0
2
Tokens usedLower is better
3,010,847
361,995
CLI discovery
Discovery score
55%
50%
CLI exists
Passed5/5groq
Passed5/5firectl
Install docs
Passed5/5npm
Passed5/5binary download
Auth method
Passed25/25env var
Passed25/25env var
Non-interactive
Failed0/25none
Failed0/25none
JSON output
Failed0/15none
Passed15/15full
llms.txt
Failed0/5
Failed0/5
Agent skill
Passed20/20
Failed0/20
Eval details
Integration task
# Task
Using the groq CLI (Groq), extract structured data from this invoice text:
---
INVOICE #INV-2024-0042
From: Acme Corp
Date: 2024-03-15
Items:
- Consulting Services: $500....
# Task
Using the firectl CLI (Fireworks AI), extract structured data from this invoice text:
---
INVOICE #INV-2024-0042
From: Acme Corp
Date: 2024-03-15
Items:
- Consulting Serv...
Generated files
4
1
Trace tool calls
60
15
Completed
10/13/2026
9/27/2026
Stderr
None captured
None captured
Full eval