OpenRouter Benchmarks
Humanity's Last Exam collects expert-written questions at the frontier of human knowledge, spanning mathematics, the sciences, and the humanities. We run its 2,158 text-only questions as a search benchmark: the model answers with live web search rather than from stored knowledge alone, and a judge grades whether its final answer matches the reference. We run it with the model held fixed and vary the search lane (engine, request surface, and agent turn budget) through OpenRouter's web-search tooling, so a score reflects the search configuration rather than the model. Where BrowseComp stresses persistent multi-step browsing, HLE shows what search contributes to questions that are hard on expertise grounds.
No verified runs yet. Check back soon.
HLE sits at the opposite end of the search spectrum from BrowseComp. Its questions were written by subject-matter experts to be unambiguous but extremely hard, so a model's baseline score is mostly a function of what it already knows. Adding web search turns that into a different question: how much expert-level knowledge can a search configuration retrieve on demand?
We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.
These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Read the deltas between lanes, not the raw levels.
Because HLE leans on expertise rather than browsing depth, lane deltas are smaller here than on BrowseComp: many questions are answered (or missed) the same way at every budget. Engine gaps smaller than the confidence intervals read as ties, and a lane's cost is as real as its score — the pareto chart is the honest summary.
We use the text-only subset of the public dataset (multi-modal questions are excluded), so scores are not directly comparable to full-HLE leaderboards. The questions are public, which means memorization can inflate absolute scores over time; search-lane deltas remain meaningful because every lane shares the same model.
Each task is one question with a short reference answer. The model answers in a fixed format (explanation, exact answer, stated confidence), and a judge model grades whether the extracted answer is semantically equivalent to the reference — the same answer-equivalence grading BrowseComp uses. The grade is binary with no partial credit, and failed or refused tasks score zero.
reward = judge(extracted_answer ≡ reference_answer) // ∈ {0, 1}
judge = gpt-4.1 at temperature 0, strict json_schema verdict
empty or refused answers skip the judge and score 0A lane is one (engine, surface, turn budget) configuration. Lane scores aggregate task-weighted across verified runs, so a 100-task run counts twice as much as a 50-task run. Runs below the sample floor (smoke tests) are excluded entirely.
Runs execute on OpenRouter's native bench harness against production endpoints. Each rung of the ladder sets the request-level tool-call budget (max_tool_calls) on the web search server tool, or uses the pluginfor one pre-inference search of the user's question verbatim. Engines run with their default settings, the same configurations production traffic hits.
Every run persists its full configuration, so each lane on this page is filterable down to the exact request parameters that produced it. You can reproduce any lane with the same knobs on your own requests. See the web search docs for the parameters, or browse all models to try one.