OpenRouter Benchmarks
BrowseComp tests whether a model can find hard-to-locate facts on the live web. Its 1,266 questions are built to be unfindable in one search, so scoring rewards persistent, multi-step research. The model doesn't need stored knowledge; it's scored entirely on whether its final answer matches a reference after searching. We run it with the model held fixed and vary the search lane (engine, request surface, and agent turn budget) through OpenRouter's web-search tooling, so a score reflects the search configuration rather than the model. We use this benchmark because it is the most search-depth-sensitive suite we run, so we can assess what each engine and turn budget contributes.
Last benchmark run Jul 21, 2026, 7:54 AM UTC
Score by engine and turn budget. BrowseComp questions are engineered to defeat single searches, which makes this the most search-depth-sensitive suite we run: accuracy climbs with budget, and engines converge as the agent gets more searches.
| Engine | default |
|---|---|
| native | 72.5% |
| Engine | plugin | 1-turn | 5-turn | 25-turn |
|---|---|---|---|---|
| perplexity | 13.0% | 1.0% | 37.3% | 51.2% |
| exa | 7.8% | 10.0% | 36.0% | 47.0% |
| parallel | — | 6.0% | 21.7% | 40.0% |
| Engine | default |
|---|---|
| native | 40.0% |
Search depth costs money. Each point is one lane, and the frontier shows which configurations nobody beats on both price and accuracy.
One row per verified lane; sort by score, cost, or speed.
| # | Engine | Budget | 95% CI | ||||
|---|---|---|---|---|---|---|---|
| 1 | native | default | 72.5% | 65.9%–78.2% | $0.36 | 0s | 200 |
| 2 | perplexity | 25-turn | 51.2% | 46.4%–56.1% | $0.22 | 0s | 400 |
| 3 | exa | 25-turn | 47.0% | 40.2%–53.9% | $0.29 | 0s | 200 |
| 4 | native | default | 40.0% | 33.5%–46.9% | $0.87 | 0s | 200 |
| 5 | parallel | 25-turn | 40.0% | 32.5%–48.0% | $0.18 | 0s | 150 |
| 6 | perplexity | 5-turn | 37.3% | 32.1%–42.9% | $0.088 | 0s | 300 |
| 7 | exa | 5-turn | 36.0% | 30.8%–41.6% | $0.12 | 0s | 300 |
| 8 | parallel | 5-turn | 21.7% | 17.4%–26.7% | $0.052 | 0s | 300 |
| 9 | perplexity | plugin | 13.0% | 7.8%–21.0% | $0.28 | 0s | 100 |
| 10 | exa | 1-turn | 10.0% | 7.1%–13.9% | $0.043 | 0s | 300 |
| 11 | exa | plugin | 7.8% | 3.8%–15.2% | $0.34 | 0s | 90 |
| 12 | parallel | 1-turn | 6.0% | 3.8%–9.3% | $0.033 | 0s | 300 |
| 13 | perplexity | 1-turn | 1.0% | 0.2%–5.4% | $0.013 | 0s | 100 |
BrowseComp questions pin down a single, verifiable answer behind several layers of indirection, like a person described by career fragments or an event located by intersecting constraints. One search rarely lands it; the agent has to form hypotheses, search, discard, and pivot. That makes it the sharpest tool we have for measuring what a search configuration contributes: the same model scores several times higher at a full agentic budget than through a single pre-inference search.
We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.
These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Read the deltas between lanes, not the raw levels.
Engine gaps smaller than the confidence intervals read as ties. And a lane's cost is as real as its score: the pareto chart is the honest summary, since a configuration that is slightly better and much pricier is rarely the right default.
The dataset ships encrypted with a canary string to keep it out of training corpora, but the questions are public; grading depends on live multi-step search, which is hard to shortcut through memorization.
Each task is one question with a short reference answer. The model answers in a fixed format (explanation, exact answer, stated confidence), and a judge model grades whether the extracted answer is semantically equivalent to the reference. "1988 to 1996" matches "1988-96"; a different entity fails. The grade is binary with no partial credit, and failed or refused tasks score zero.
reward = judge(extracted_answer ≡ reference_answer) // ∈ {0, 1}
judge = gpt-4.1 at temperature 0, strict json_schema verdict
empty or refused answers skip the judge and score 0A lane is one (engine, surface, turn budget) configuration. Lane scores aggregate task-weighted across verified runs, so a 100-task run counts twice as much as a 50-task run. Runs below the sample floor (smoke tests) are excluded entirely.
Runs execute on OpenRouter's native bench harness against production endpoints. Each rung of the ladder sets the request-level tool-call budget (max_tool_calls) on the web search server tool, or uses the pluginfor one pre-inference search of the user's question verbatim. Engines run with their default settings, the same configurations production traffic hits.
Every run persists its full configuration, so each lane on this page is filterable down to the exact request parameters that produced it. You can reproduce any lane with the same knobs on your own requests. See the web search docs for the parameters, or browse all models to try one.