OpenRouter Benchmarks
WideSearch asks the model to fill in an entire table: given a query and a required column schema, it has to find every entity and every attribute, like all launches of a rocket family with dates and outcomes. Its 200 tasks (100 English, 100 Chinese) are graded strictly — a task counts only when every row and every cell matches the reference table. We run it with the model held fixed and vary the search lane (engine, request surface, and agent turn budget) through OpenRouter's web-search tooling, so a score reflects the search configuration rather than the model. Where BrowseComp goes deep on one hidden fact, WideSearch goes wide across many.
Last benchmark run Jul 21, 2026, 7:32 AM UTC
Score by engine and turn budget. A complete table takes many searches to assemble, so this suite is highly budget-sensitive — and strict grading means one missed cell zeroes the whole task.
| Engine | default |
|---|---|
| native | 2.5% |
| Engine | plugin | 1-turn | 5-turn | 25-turn |
|---|---|---|---|---|
| perplexity | 1.0% | 0.5% | 2.5% | 1.5% |
| exa | 0.0% | 1.5% | 1.7% | — |
| parallel | 0.5% | 0.0% | — | — |
| Engine | default |
|---|---|
| native | 1.5% |
Search depth costs money. Each point is one lane, and the frontier shows which configurations nobody beats on both price and accuracy.
One row per verified lane; sort by score, cost, or speed.
| # | Engine | Budget | 95% CI | ||||
|---|---|---|---|---|---|---|---|
| 1 | native | default | 2.5% | 1.1%–5.7% | $0.040 | 0s | 200 |
| 2 | perplexity | 5-turn | 2.5% | 1.1%–5.7% | $0.020 | 0s | 200 |
| 3 | exa | 5-turn | 1.7% | 0.6%–4.9% | $0.028 | 0s | 175 |
| 4 | native | default | 1.5% | 0.5%–4.3% | $0.15 | 0s | 200 |
| 5 | exa | 1-turn | 1.5% | 0.5%–4.3% | $0.007 | 0s | 200 |
| 6 | perplexity | 25-turn | 1.5% | 0.5%–4.3% | $0.031 | 0s | 200 |
| 7 | perplexity | plugin | 1.0% | 0.3%–3.6% | $0.035 | 0s | 200 |
| 8 | perplexity | 1-turn | 0.5% | 0.1%–2.8% | $0.006 | 0s | 200 |
| 9 | parallel | plugin | 0.5% | 0.1%–2.8% | $0.042 | 0s | 200 |
| 10 | parallel | 1-turn | 0.0% | 0.0%–1.9% | $0.003 | 0s | 200 |
| 11 | exa | plugin | 0.0% | 0.0%–1.9% | $0.021 | 0s | 200 |
WideSearch measures breadth. The agent has to enumerate a full entity set, chase down each attribute, and emit a structured table — closer to real research workflows (market scans, literature surveys, competitive tables) than single-answer trivia. Half the tasks are in Chinese, so it also exercises engines outside English-language results.
We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.
These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Read the deltas between lanes, not the raw levels.
The headline score is strict task success: one wrong cell in a fifty-row table zeroes the task, so scores run very low and near-misses are invisible here. The harness also records row- and cell-level precision, recall, and F1 for finer-grained comparison, but this page charts the strict rate. Engine gaps smaller than the confidence intervals read as ties, and a lane's cost is as real as its score — the pareto chart is the honest summary.
The questions are public, which means memorization can inflate absolute scores over time; search-lane deltas remain meaningful because every lane shares the same model.
Each task is one query with a reference table. The model's table is parsed, its rows and columns are aligned to the reference by a judge model, and each cell is compared — exact match where possible, judge-graded equivalence otherwise. A task succeeds only when every reference row is found and every cell matches, with no missing or extra rows.
reward = all_rows_aligned ∧ every_cell_correct // ∈ {0, 1}
judge = gpt-4.1 at temperature 0 for alignment and fuzzy cells
unparseable or empty tables score 0A lane is one (engine, surface, turn budget) configuration. Lane scores aggregate task-weighted across verified runs, so a 100-task run counts twice as much as a 50-task run. Runs below the sample floor (smoke tests) are excluded entirely.
Runs execute on OpenRouter's native bench harness against production endpoints. Each rung of the ladder sets the request-level tool-call budget (max_tool_calls) on the web search server tool, or uses the pluginfor one pre-inference search of the user's question verbatim. Engines run with their default settings, the same configurations production traffic hits.
Every run persists its full configuration, so each lane on this page is filterable down to the exact request parameters that produced it. You can reproduce any lane with the same knobs on your own requests. See the web search docs for the parameters, or browse all models to try one.