BrowseComp

BrowseComp questions pin down a single, verifiable answer behind several layers of indirection, like a person described by career fragments or an event located by intersecting constraints. One search rarely lands it; the agent has to form hypotheses, search, discard, and pivot. That makes it the sharpest tool we have for measuring what a search configuration contributes: the same model scores several times higher at a full agentic budget than through a single pre-inference search.

We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.

These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Compare configurations rather than raw levels.

Overlapping confidence ranges are treated as unresolved here, not as proof of equality. Cost should be read alongside quality when one configuration is slightly better and much pricier.

The dataset ships encrypted with a canary string to keep it out of training corpora, but the questions are public; grading depends on live multi-step search, which is hard to shortcut through memorization.

Each task is one question with a short reference answer. The model answers in a fixed format (explanation, exact answer, stated confidence), and a judge model grades whether the extracted answer is semantically equivalent to the reference. "1988 to 1996" matches "1988-96"; a different entity fails. The grade is binary with no partial credit, and failed or refused tasks score zero.

reward = judge(extracted_answer ≡ reference_answer)   // ∈ {0, 1}
judge  = gpt-4.1 at temperature 0, strict json_schema verdict
empty or refused answers skip the judge and score 0

A search configuration is one model, search engine, request format, and maximum search budget. Scores are weighted by the number of questions in each verified run; small smoke tests are excluded.

Runs execute on OpenRouter's native benchmark harness against production endpoints. Server-tool configurations set a request-level tool-call budget on the web search server tool. Plugin configurations make one pre-inference search of the question with the web search plugin. Engines use the same default configurations that serve production traffic.

Every run persists its exact model, engine, request format, search budget, cost, and available timing telemetry. Missing configurations stay missing in the comparison table, and absent or zero telemetry is not treated as free or instantaneous performance.