Back to leaderboard

Deepgram vs AssemblyAI

Same Voice STT task · View

#1 in Voice STT

92ABest on 7 of 19

#2 in Voice STT

75BBest on 1 of 19

CLI task scoring

Eval score
92%
78%
Task Completed
Passed+30The agent successfully completed all task requirements: (1) installed the official `dg` CLI via `pip install deepctl`, (...
Passed+30The agent successfully completed all required steps: (1) installed the official AssemblyAI CLI from the correct source,...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Passed+10Partial credit: 100%
Passed+10Partial credit: 100%
Zero Errors
Passed+10No errors
Failed+0Failed tool calls: 2
Efficiency
Partial+67 tool calls (<=10, great)
Partial+68 tool calls (<=10, great)
Time
Partial+6109s (1-2min, great)
Partial+687s (1-2min, great)
Cost
Passed+10$0.10 (<$0.10, excellent)
Partial+6$0.13 ($0.10-0.20, great)
Runtime (actual)Lower is better
1m 49s
1m 27s
Cost (actual)Lower is better
$0.0959
$0.1278
Tool callsLower is better
7
8
ErrorsLower is better
0
2
Tokens usedLower is better
181,457
212,720

CLI discovery

Discovery score
95%
70%
CLI exists
Passed5/5dg
Passed5/5assemblyai
Install docs
Passed5/5pip
Passed5/5curl script
Auth method
Passed25/25env var
Passed25/25api key cmd
Non-interactive
Passed25/25auto detect
Failed0/25none
JSON output
Passed15/15full
Passed15/15full
llms.txt
Failed0/5
Failed0/5
Agent skill
Passed20/20
Passed20/20

Eval details

Integration task
# Task Using the dg CLI (Deepgram), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...
# Task Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file: 1. Download this sample audio file: https://storage.googleapis.com/aai-web-samples/5_common_sport...
Generated files
1
1
Trace tool calls
7
8
Completed
9/27/2026
9/27/2026
Stderr
None captured
None captured