OpenRouter Benchmarks
DeepSearchQA asks questions whose answers are lists: every member of a category, every event matching a set of constraints. Its 900 questions each carry a reference list of answer parts, and a response only counts when it finds all of them without padding the list with extras. We run it with the model held fixed and vary the search lane (engine, request surface, and agent turn budget) through OpenRouter's web-search tooling, so a score reflects the search configuration rather than the model. Where BrowseComp rewards locating one hidden fact, DeepSearchQA rewards exhaustive retrieval.
Last benchmark run Jul 21, 2026, 7:08 AM UTC
Score by engine and turn budget. Complete-list answers usually need several searches, so accuracy climbs with budget — an engine that surfaces broad, deduplicated results early needs fewer turns to close the list.
| Engine | default |
|---|---|
| native | 75.0% |
| Engine | 1-turn | 5-turn | 25-turn |
|---|---|---|---|
| exa | 29.0% | 58.5% | 56.0% |
| perplexity | 27.0% | 53.5% | 55.5% |
| parallel | 15.0% | 33.0% | 37.0% |
| Engine | default |
|---|---|
| native | 45.5% |
Search depth costs money. Each point is one lane, and the frontier shows which configurations nobody beats on both price and accuracy.
One row per verified lane; sort by score, cost, or speed.
| # | Engine | Budget | 95% CI | ||||
|---|---|---|---|---|---|---|---|
| 1 | native | default | 75.0% | 68.6%–80.5% | $0.33 | 0s | 200 |
| 2 | exa | 5-turn | 58.5% | 51.6%–65.1% | $0.10 | 0s | 200 |
| 3 | exa | 25-turn | 56.0% | 49.1%–62.7% | $0.16 | 0s | 200 |
| 4 | perplexity | 25-turn | 55.5% | 48.6%–62.2% | $0.13 | 0s | 200 |
| 5 | perplexity | 5-turn | 53.5% | 46.6%–60.3% | $0.089 | 0s | 200 |
| 6 | native | default | 45.5% | 38.7%–52.4% | $0.68 | 0s | 200 |
| 7 | parallel | 25-turn | 37.0% | 30.6%–43.9% | $0.13 | 0s | 200 |
| 8 | parallel | 5-turn | 33.0% | 26.9%–39.8% | $0.033 | 0s | 200 |
| 9 | exa | 1-turn | 29.0% | 23.2%–35.6% | $0.027 | 0s | 200 |
| 10 | perplexity | 1-turn | 27.0% | 21.3%–33.5% | $0.022 | 0s | 200 |
| 11 | parallel | 1-turn | 15.0% | 10.7%–20.6% | $0.012 | 0s | 200 |
Finding one fact and finding every fact are different search problems. A single lucky query can land a BrowseComp answer; a DeepSearchQA answer has to be assembled across searches, with the agent tracking what it has found and what is still missing. That coverage pressure exercises a different part of the search stack: result breadth and deduplication matter as much as ranking.
We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.
These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Read the deltas between lanes, not the raw levels.
The all-or-nothing grade makes this suite strict: missing one list member scores the same as missing them all, so scores run low and small engine gaps are noisy. Engine gaps smaller than the confidence intervals read as ties, and a lane's cost is as real as its score — the pareto chart is the honest summary.
The questions are public, which means memorization can inflate absolute scores over time; search-lane deltas remain meaningful because every lane shares the same model.
Each task is one question with a reference list of expected answer parts. A judge model checks the response against every expected part and also flags excessive answers — parts the response asserts that are not in the reference. A task is correct only when the judge finds every expected part and finds no excessive answers. The grade is binary with no partial credit, and failed or refused tasks score zero.
reward = all_expected_parts_found ∧ no_excessive_answers // ∈ {0, 1}
judge = gemini-2.5-flash, strict json_schema completeness verdict
empty or refused answers skip the judge and score 0A lane is one (engine, surface, turn budget) configuration. Lane scores aggregate task-weighted across verified runs, so a 100-task run counts twice as much as a 50-task run. Runs below the sample floor (smoke tests) are excluded entirely.
Runs execute on OpenRouter's native bench harness against production endpoints. Each rung of the ladder sets the request-level tool-call budget (max_tool_calls) on the web search server tool, or uses the pluginfor one pre-inference search of the user's question verbatim. Engines run with their default settings, the same configurations production traffic hits.
Every run persists its full configuration, so each lane on this page is filterable down to the exact request parameters that produced it. You can reproduce any lane with the same knobs on your own requests. See the web search docs for the parameters, or browse all models to try one.