# OpenRouter public datasets

Daily CSV and JSON snapshots of the benchmark results and model rankings shown on https://openrouter.ai. Schema version v1.

## License and attribution

All files are licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). See https://openrouter.ai/api/v1/datasets/exports/LICENSE.

Attribution text: Source: OpenRouter (openrouter.ai), licensed under CC BY 4.0: reuse and republish with attribution to OpenRouter.

Suggested citation: OpenRouter. "<dataset title>", snapshot <snapshot_date>. Retrieved <retrieved_at>. <snapshot URL>. CC BY 4.0.

## Update cadence and URLs

A snapshot is published once per UTC day. Dated files are immutable once written. The `latest` alias is rewritten to point at the newest snapshot.

The `latest` files are convenience aliases, not a release contract. Each dataset's alias moves independently, and the aliases are served through a short public cache, so for up to about an hour after a publish the `latest.csv` and `latest.json` of one dataset may still come from different snapshot dates. A reproducible pipeline reads `latest.manifest.json` (or `index.json`), downloads the dated files it names, and verifies their checksums.

- Dated file: `https://openrouter.ai/api/v1/datasets/exports/<dataset>/<YYYY-MM-DD>.csv` (also `.json` and `.manifest.json`)
- Latest alias: `https://openrouter.ai/api/v1/datasets/exports/<dataset>/latest.csv` (also `.json` and `.manifest.json`)
- Catalog: https://openrouter.ai/api/v1/datasets/exports/index.json

Each `.manifest.json` lists the snapshot date, retrieval timestamp, row count, and for the CSV and JSON files their byte size and SHA-256 checksum. Verify a download with `sha256sum <file>` and compare against `files.<format>.sha256`.

The JSON file wraps the rows in an envelope that repeats the license, attribution, retrieval timestamp, methodology, harness (`null` for datasets not produced by a benchmark run), and column definitions so the file is self-describing.

## Harness

Rows in the benchmark datasets come from runs executed by the OpenRouter benchmark harness (https://github.com/OpenRouterTeam/benchmark-harness). The harness does not record a version per run in the public results table, so no per-row harness version column exists. The harness repository history is the version record for a given run timestamp. The rankings dataset is derived from platform usage and involves no harness.

## What is never exported

- Only values already displayed on public openrouter.ai pages are exported. A column is added only when the page it mirrors shows the value.
- Benchmark rows come from the same query the public leaderboard pages use: successful runs, visible runs (hidden = false), no epoch sub-runs, and the page minimum sample floor. Hiding a run on the site removes it from the next snapshot.
- Private endpoints, partner-restricted endpoints, and non-public models are excluded by run visibility: benchmark runs are created hidden and reach the public pages, and therefore the export, only after an operator publishes them. Rankings count only the public model variants the rankings page shows.
- Upstream benchmark question sets, prompts, model outputs, and per-question grades are never published. Rows carry aggregate scores only.
- Rankings export tokens per public model variant per day. There is no per-app, per-user, per-organization, or per-request data.
- Internal identifiers other than the benchmark run id (workflow id) and model and provider slugs are omitted, including endpoint ids and run configuration blobs.

## Datasets

### OpenRouter benchmark runs (`benchmark-runs`)

One row per public OpenRouter-run benchmark run (GPQA Diamond, tau2-bench airline). The leaderboard pages aggregate these rows per model and provider.

Source pages: https://openrouter.ai/benchmarks/gpqa-diamond, https://openrouter.ai/benchmarks/tau2-bench-airline

Methodology: Runs are executed by the OpenRouter benchmark harness against public OpenRouter endpoints. A source row is read only when the run succeeded, is visible on openrouter.ai, is not an epoch sub-run, and meets the minimum sample floor. Each benchmark reads its newest 10,000 such runs, the same window the leaderboard page aggregates, and then excludes rows that scored zero, so a snapshot can hold fewer rows than the window. Older source rows roll out of the snapshot as new runs land. The leaderboard accuracy for a model and provider is the task-weighted mean of accuracy over these rows.

Latest: https://openrouter.ai/api/v1/datasets/exports/benchmark-runs/latest.csv

| Column | Type | Nullable | Description |
| --- | --- | --- | --- |
| `benchmark` | string | no | Benchmark identifier. |
| `benchmark_page_url` | string | no | openrouter.ai leaderboard page that displays this run. |
| `run_id` | string | no | Harness workflow id of the run. |
| `run_timestamp` | string | no | ISO 8601 UTC timestamp the run finished. |
| `model_permaslug` | string | no | Immutable OpenRouter model version identifier. |
| `model_slug` | string | yes | Current OpenRouter model slug, null when the version is no longer listed. |
| `model_name` | string | yes | Display name, null when the version is no longer listed. |
| `provider_name` | string | no | Inference provider that served the run. |
| `accuracy` | number | no | Fraction of tasks solved, 0 to 1. |
| `total_questions` | integer | no | Tasks attempted in the run (sample size). |
| `total_cost_usd` | number | yes | Total inference spend for the run in USD. |
| `duration_ms` | number | yes | Wall-clock duration of the run in milliseconds. |
| `generation_time_ms` | number | yes | Summed model generation time in milliseconds. |
| `output_tokens` | number | yes | Summed completion tokens across the run. |

### OpenRouter benchmark leaderboard (`benchmark-leaderboard`)

One row per benchmark, model version, and provider: the aggregate values displayed on the OpenRouter leaderboard pages (GPQA Diamond, tau2-bench airline). Derived from the benchmark-runs dataset.

Source pages: https://openrouter.ai/benchmarks/gpqa-diamond, https://openrouter.ai/benchmarks/tau2-bench-airline

Methodology: Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's newest 10,000 non-zero-score runs: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.

Latest: https://openrouter.ai/api/v1/datasets/exports/benchmark-leaderboard/latest.csv

| Column | Type | Nullable | Description |
| --- | --- | --- | --- |
| `benchmark` | string | no | Benchmark identifier. |
| `benchmark_page_url` | string | no | openrouter.ai leaderboard page that displays this row. |
| `model_permaslug` | string | no | Immutable OpenRouter model version identifier. |
| `model_slug` | string | yes | Current OpenRouter model slug, null when the version is no longer listed. |
| `model_name` | string | no | Display name as shown on the leaderboard page. |
| `provider_name` | string | no | Inference provider that served the aggregated runs. |
| `accuracy` | number | no | Task-weighted mean accuracy across runs, 0 to 1. |
| `accuracy_std_dev` | number | yes | Population standard deviation of per-run accuracy; null when only one run exists. |
| `run_count` | integer | no | Number of runs aggregated. |
| `total_questions` | integer | no | Tasks attempted across all aggregated runs (sample size). |
| `avg_cost_per_task_usd` | number | yes | Summed run cost divided by summed tasks, in USD. |
| `avg_duration_per_task_ms` | number | yes | Summed generation time divided by summed tasks, in milliseconds. |
| `avg_output_tokens_per_task` | number | yes | Summed completion tokens divided by summed tasks. |
| `last_run_timestamp` | string | no | ISO 8601 UTC timestamp of the most recent aggregated run. |

### OpenRouter web search benchmark lanes (`search-benchmark-lanes`)

One row per lane (suite, model, search engine, surface, turn budget, reasoning effort) on the OpenRouter web search benchmark pages (BrowseComp, HLE, DeepSearchQA, WideSearch), pooled across the independent runs the page displays.

Source pages: https://openrouter.ai/benchmarks/browsecomp, https://openrouter.ai/benchmarks/hle, https://openrouter.ai/benchmarks/deepsearchqa, https://openrouter.ai/benchmarks/widesearch

Methodology: Runs pair a model with a web search engine through the OpenRouter server-tool or plugin surface. A run contributes only when it succeeded, is visible on openrouter.ai, has no epoch marker, ran through a public search surface, answered at least 45 questions, and is the latest run of its resume lineage; within a lane only runs sharing the most recent configuration are pooled. Score is pooled correct answers over pooled questions, which is the value the suite pages plot.

Latest: https://openrouter.ai/api/v1/datasets/exports/search-benchmark-lanes/latest.csv

| Column | Type | Nullable | Description |
| --- | --- | --- | --- |
| `benchmark` | string | no | Search suite identifier. |
| `benchmark_page_url` | string | no | openrouter.ai suite page that displays this lane. |
| `model_permaslug` | string | no | Immutable OpenRouter model version identifier. |
| `model_name` | string | yes | Display name, null when the version is no longer listed. |
| `search_engine` | string | no | Web search engine used. |
| `surface` | string | no | How search was invoked: server-tool or plugin. |
| `max_agent_turns` | integer | yes | Agent turn budget, null when the lane ran with the server default. |
| `reasoning_effort` | string | yes | Reasoning effort requested from the model, null when unset. |
| `score` | number | no | Pooled correct answers divided by pooled questions, 0 to 1. |
| `item_f1` | number | no | Sample-weighted partial-credit metric (item F1 for WideSearch, accuracy otherwise). |
| `runs` | integer | no | Independent runs pooled into the lane. |
| `total_questions` | integer | no | Questions pooled across the runs (sample size). |
| `total_correct` | integer | no | Correct answers pooled across the runs. |
| `cost_per_task_usd` | number | yes | Mean inference and search spend per question in USD. |
| `latency_mean_ms` | number | yes | Sample-weighted mean generation time per question in milliseconds, null when no run recorded timing. |
| `last_run_at` | string | no | ISO 8601 UTC timestamp of the most recent pooled run. |

### OpenRouter model rankings, daily token share (`rankings-daily`)

One row per UTC day and public model variant for the top 50 models by tokens processed on OpenRouter, plus one "other" row per day aggregating every remaining public model. Covers the trailing 30 complete days before the snapshot date.

Source pages: https://openrouter.ai/rankings

Methodology: Tokens are prompt plus completion tokens from the same public activity view that draws the chart on openrouter.ai/rankings; private models and private endpoints are excluded at the source. Each model variant (for example ":free") ranks as its own entry. Within a day, models are ranked by total tokens descending with an alphabetical tiebreaker; the top 50 appear individually and the rest collapse into "other". Models with zero text tokens on a day (audio, image, and embedding endpoints) are omitted.

Latest: https://openrouter.ai/api/v1/datasets/exports/rankings-daily/latest.csv

| Column | Type | Nullable | Description |
| --- | --- | --- | --- |
| `date` | string | no | UTC day, YYYY-MM-DD. |
| `model_permaslug` | string | no | Model variant permaslug, or "other" for the aggregated long tail. |
| `rank` | integer | yes | 1-based rank by total tokens within the day; null on the "other" row. |
| `total_tokens` | integer | no | Prompt plus completion tokens processed for the model on that day. |
| `share_of_daily_tokens` | number | no | Fraction of the day's tokens across every row for that day (including "other"), 0 to 1. |