AssemblyAI vs Deepgram
Both ran the same Voice STT task · Same Voice STT task · View the taskView the task
CompanyClick a name to swap it
#2 in Voice STT · #17 overall
75BBest on 1 of 19
#1 in Voice STT · tied #1 overall
92ABest on 7 of 19
Compare up to 4.
CLI task scoring
Eval score
78%
92%
Task Completed
Passed+30The agent successfully completed all required steps: (1) installed the official AssemblyAI CLI from the correct source,...
Passed+30The agent successfully completed all task requirements: (1) installed the official `dg` CLI via `pip install deepctl`, (...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Passed+10Partial credit: 100%
Passed+10Partial credit: 100%
Zero Errors
Failed+0Failed tool calls: 2
Passed+10No errors
Efficiency
Partial+68 tool calls (<=10, great)
Partial+67 tool calls (<=10, great)
Time
Partial+687s (1-2min, great)
Partial+6109s (1-2min, great)
Cost
Partial+6$0.13 ($0.10-0.20, great)
Passed+10$0.10 (<$0.10, excellent)
Runtime (actual)Lower is better
1m 27s
1m 49s
Cost (actual)Lower is better
$0.1278
$0.0959
Tool callsLower is better
8
7
ErrorsLower is better
2
0
Tokens usedLower is better
212,720
181,457
CLI discovery
Discovery score
70%
95%
CLI exists
Passed5/5assemblyai
Passed5/5dg
Install docs
Passed5/5curl script
Passed5/5pip
Auth method
Passed25/25api key cmd
Passed25/25env var
Non-interactive
Failed0/25none
Passed25/25auto detect
JSON output
Passed15/15full
Passed15/15full
llms.txt
Failed0/5
Failed0/5
Agent skill
Passed20/20
Passed20/20
Eval details
Integration task
# Task
Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file:
1. Download this sample audio file:
https://storage.googleapis.com/aai-web-samples/5_common_sport...
# Task
Using the dg CLI (Deepgram), transcribe a sample audio file:
1. Download this sample audio file:
https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...
Generated files
1
1
Trace tool calls
8
7
Completed
9/27/2026
9/27/2026
Stderr
None captured
None captured
#2 in Voice STT
75BBest on 1 of 19
#1 in Voice STT
92ABest on 7 of 19
CLI task scoring
Eval score
78%
92%
Task Completed
Passed+30The agent successfully completed all required steps: (1) installed the official AssemblyAI CLI from the correct source,...
Passed+30The agent successfully completed all task requirements: (1) installed the official `dg` CLI via `pip install deepctl`, (...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Passed+10Partial credit: 100%
Passed+10Partial credit: 100%
Zero Errors
Failed+0Failed tool calls: 2
Passed+10No errors
Efficiency
Partial+68 tool calls (<=10, great)
Partial+67 tool calls (<=10, great)
Time
Partial+687s (1-2min, great)
Partial+6109s (1-2min, great)
Cost
Partial+6$0.13 ($0.10-0.20, great)
Passed+10$0.10 (<$0.10, excellent)
Runtime (actual)Lower is better
1m 27s
1m 49s
Cost (actual)Lower is better
$0.1278
$0.0959
Tool callsLower is better
8
7
ErrorsLower is better
2
0
Tokens usedLower is better
212,720
181,457
CLI discovery
Discovery score
70%
95%
CLI exists
Passed5/5assemblyai
Passed5/5dg
Install docs
Passed5/5curl script
Passed5/5pip
Auth method
Passed25/25api key cmd
Passed25/25env var
Non-interactive
Failed0/25none
Passed25/25auto detect
JSON output
Passed15/15full
Passed15/15full
llms.txt
Failed0/5
Failed0/5
Agent skill
Passed20/20
Passed20/20
Eval details
Integration task
# Task
Using the assemblyai CLI (AssemblyAI), transcribe a sample audio file:
1. Download this sample audio file:
https://storage.googleapis.com/aai-web-samples/5_common_sport...
# Task
Using the dg CLI (Deepgram), transcribe a sample audio file:
1. Download this sample audio file:
https://storage.googleapis.com/aai-web-samples/5_common_sports_injuries...
Generated files
1
1
Trace tool calls
8
7
Completed
9/27/2026
9/27/2026
Stderr
None captured
None captured
Full eval