Back to leaderboard

Railway vs Render

Same Cloud Hosting task · View

#4 in Cloud Hosting

46DBest on 2 of 19

#2 in Cloud Hosting

74CBest on 6 of 19

CLI task scoring

Eval score
27%
63%
Task Completed
Failed+0The agent correctly installed the Railway CLI (@railway/cli via npm) and cloned the repository as required. However, the...
Passed+30The agent successfully completed all required task steps: (1) cloned the nextjs-starter repository from GitHub, (2) used...
Used CLI Correctly
Passed+20Agent used CLI commands correctly
Passed+20Agent used CLI commands correctly
Partial Credit
Partial+4.0Partial credit: 40%
Passed+10Partial credit: 100%
Zero Errors
Failed+0Failed tool calls: 6
Failed+0Failed tool calls: 7
Efficiency
Failed+033 tool calls (>25, inefficient)
Failed+030 tool calls (>25, inefficient)
Time
Partial+1.0331s (5-10min, very slow)
Partial+1.0312s (5-10min, very slow)
Cost
Partial+2.0$0.52 ($0.40-0.70, expensive)
Partial+2.0$0.49 ($0.40-0.70, expensive)
Runtime (actual)Lower is better
5m 30s
5m 11s
Cost (actual)Lower is better
$0.5193
$0.4943
Tool callsLower is better
33
30
ErrorsLower is better
6
7
Tokens usedLower is better
1,222,643
1,280,145

CLI discovery

Discovery score
93%
100%
CLI exists
Passed5/5railway
Passed5/5render
Install docs
Passed5/5npm
Passed5/5install script
Auth method
Passed25/25env var
Passed25/25env var
Non-interactive
Partial18/25flag
Passed25/25auto detect
JSON output
Passed15/15full
Passed15/15full
llms.txt
Passed5/5
Passed5/5
Agent skill
Passed20/20
Passed20/20

Eval details

Integration task
# Task Using the railway CLI (Railway), deploy this application https://github.com/[REDACTED:NAME]/nextjs-starter to Railway: 1. Clone the repository: `git clone https://github.c...
# Task Using the render CLI (Render), deploy this application https://github.com/[REDACTED:NAME]/nextjs-starter to Render: 1. Clone the repository: `git clone https://github.com/...
Generated files
1
1
Trace tool calls
33
30
Completed
10/4/2026
9/27/2026
Stderr
None captured
None captured