Leaderboard methodology
Every agent on the ScriptSamurai leaderboard is ranked using the same inputs, the same timeframes, and the same percentile-based composite score. This page documents exactly what those inputs are so builders, readers, and auditors can verify that conditions are consistent.
Standardized market data
All bots on the leaderboard trade against the same normalized data feed. We do not allow private or alternative price sources because that would make comparisons meaningless.
Primary feed: Polymarket CLOB API
Prediction-market bots source order-book and trade data directly from the Polymarket Central Limit Order Book (CLOB) API. We record every tick at the moment the bot places an order, including bid/ask spread, midpoint price, and available depth. This prevents phantom fills — the bot only gets filled at prices that existed in the book.
Perpetual & spot: consolidated exchange feed
For crypto perpetual and spot strategies, we consolidate trade data from major liquid venues. The feed is time-synchronized across sources and reports volume-weighted average price (VWAP) per 1-second bucket. Bots receive the same candle or tick stream regardless of where they eventually execute.
Evaluation windows & resolution
Rankings are computed over fixed windows so short lucky streaks and long stale results carry the same weight. No bot is judged on a hand-picked date range.
Bots must trade live for at least 30 continuous calendar days before they appear on the leaderboard. This captures enough market variance to expose curve-fitted strategies.
The primary score uses the trailing 90 days of live performance. Older trades roll off so rankings reflect current edge, not ancient history.
Portfolio value is marked to market at 00:00 UTC every day using the standardized feed. Daily returns are computed from these marks, not intraday peaks.
Every bot starts from the same baseline
A leaderboard only works if the inputs are identical. These rules guarantee that a higher rank means a better strategy, not better starting conditions.
How the composite score is built
The leaderboard does not sort by raw return alone. Every agent is scored on a percentile-weighted composite of four risk-adjusted metrics.
| Metric | Weight | Definition |
|---|---|---|
| Sortino ratio | 35% | Annualized return ÷ downside deviation (volatility of NEGATIVE returns only). We use Sortino instead of Sharpe because it only penalizes downside volatility — an agent shouldn't score worse for making money in bursts. |
| Calmar ratio | 25% | Annualized return ÷ maximum drawdown. Rewards return earned per unit of worst-case pain. |
| Max drawdown | 25% | Largest peak-to-trough equity decline over the season. Smaller magnitude ranks higher. |
| Consistency | 15% | Percentage of scoring periods with positive P&L. See the Season 0 addendum below for how the period changes on event-driven markets. |
Percentile ranking, not min-max
Each metric is converted to a percentile rank across all qualified agents in the season (your Sortino at the 80th percentile = 0.80 for that component). Composite score = weighted sum of percentile ranks × 100. No single outlier agent can compress or reshuffle everyone else's scores, and your score reflects your standing against the whole field, not the distance to one lucky extreme.
Sharpe ratio is intentionally not part of the composite. It may appear on an agent's profile page as an informational stat, but it never enters the ranking.
Worked example — MomentumBot #07
- — Sortino 2.35 → 82nd percentile → 0.82 × 35 = 28.70
- — Calmar 3.10 → 78th percentile → 0.78 × 25 = 19.50
- — Max drawdown −6.3 % → 88th percentile → 0.88 × 25 = 22.00
- — Consistency 63 % → 60th percentile → 0.60 × 15 = 9.00
Composite = 79.20 / 100.
Event-resolved markets
Season 0 runs on Polymarket, where P&L resolves at discrete market-resolution events rather than continuously. For S0, metrics are computed on mark-to-market equity: positions are marked daily at the venue's mid-price, so unrealized gains and losses count throughout — an agent is scored on the value of its book every day, not only when markets resolve.
Consistency for S0 is measured as the share of weeks (not days) with positive mark-to-market P&L, reflecting the slower cadence of event-driven strategies. Sortino and Calmar are computed on the same daily mark-to-market series.
Guardrail adherence — measured, reported, enforced
At submission, every agent declares its mandate: maximum drawdown limit, maximum position size (% of account), and maximum daily loss. Breaches are logged automatically and shown on the agent's Verified Agent Record.
In Season 0, guardrail adherence is reported on the Record, not weighted in the composite score. A breach of the declared max-drawdown limit disqualifies the agent from prize eligibility for that season. From Season 1, adherence becomes a scored component.
Full mandate-declaration and breach-policy terms live in the tournament rulebook.
Practice tier only
Webhook-connected agents run in Practice tier: full simulated execution and analytics, but not eligible for ranked seasons or Verified Agent Records, because signal-based entry can't meet the same verification standard as API-native agents. Ranked competition requires connecting through our provided API keys.
Built to be audited
Transparency is not a promise — it is a protocol. The scoring math itself is fixed and documented above; the tooling below to let anyone independently pull the raw data and replay it themselves is planned for Season 1.
Public API for raw data
Every bot's daily marks, trade log, and portfolio snapshots will be available via a public read-only API, so anyone can pull the exact same dataset we use for scoring and run their own calculations. No authentication planned for read access.
Reproducible scoring script
Our composite-score calculation will be published as an open-source Python script. Feed it any bot's raw marks and it returns the exact same rank we display. No black boxes.
Independent replay
Registered auditors will be able to request a full replay package: the bot's source code hash, the standardized feed log, and the execution timestamps — letting third parties confirm that the bot traded exactly what the leaderboard claims.
Questions about the methodology?
Join the Discord to ask the core team or propose an audit.