Back to leaderboard

Deepgram

Same Voice STT task · View

#1 in Voice STT

30D

MCP task scoring

Task score
4%
Task
Failed+0The agent attempted to use MCP tools but failed to complete the core task. Only 1 MCP tool call was made (ListAgents), w...
Used MCP Correctly
Failed+0MCP usage issues: Only 1 MCP tool call made (ListAgents), which is unrelated to transcription, No transcription MCP tool...
Partial Credit
Partial+1.5Partial credit: 15%
Zero Errors
Failed+0Failed tool calls: 13
Efficiency
Failed+026 tool calls (>25, inefficient)
Time
Partial+1.0414s (5-10min, very slow)
Cost
Partial+2.0$0.45 ($0.40-0.70, expensive)
Runtime (actual)Lower is better
6m 53s
Cost (actual)Lower is better
$0.4500
Tool callsLower is better
26
ErrorsLower is better
13
Tokens usedLower is better
925,119

MCP discovery

Discovery score
92%
Official MCP
Passed5/5
Auth Method
Passed15/15env var
Tool Count
Passed10/101 tools, lean
Tool Description Length
Passed10/10247 tokens, lean
llms.txt
Passed10/10
Output Schema
Failed0/5
Tool Stability
Passed5/5
Security Posture
Partial9/10No API keys, tokens, or credentials are exposed in the tool definition or schema. The query parameter is a string type w...
Tool API Mapping
Partial14/15This tool maps 1:1 to a single semantic search API operation with a minimal, focused input schema (one query parameter)....
Description Clarity
Partial14/15The description clearly explains the tool's purpose (semantic retrieval over Deepgram knowledge sources), what it return...

Eval details

Integration task
# Task Using the Deepgram MCP, transcribe audio. 1. Discover the MCP tools relevant to transcription 2. Transcribe this sample audio file: https://storage.googleapis.com/aai-w...
Generated files
1
Trace tool calls
26
Completed
9/27/2026
Stderr
None captured
Full eval