Relevance isn’t enough: what it takes to rerank the web

Open-source reranker benchmarks are built around one question: does this document match this query? On this question, frontier rerankers are no longer distinguishable. Exa, Cohere, Voyage AI and Google Cloud all land within two points of each other on BEIR and MIRACL.

Agents searching the open web need more than just a topical match. They need to understand the intent of the searcher: for example, a navigational query is trying to find just one particular page. A time-sensitive query will have some current answers and many stale ones. On top of these issues, a growing share of pages that seem to match the query topic are actually spam, SEO bait, or contain prompt injections to try to game the ranker. To help agents succeed, a good reranker has to generalize across all of these objectives, not just optimize for relevance.

To test this, we developed four new evaluations to supplement the existing three public benchmarks: navigational intent, freshness, authority under adversarial content, and coding. We then compare the reranker that powers every Exa search against the leading standalone reranking APIs. Rerankers tuned only for relevance can actively hurt the quality of Exa results. On navigational queries, every competing reranker made Exa’s results worse. Averaged across all seven benchmarks, Exa not only sets the frontier in both the default and fast latency tiers, but Exa Fast beats every competitor’s default-tier model.

#Reranking

A standard search engine contains (at least) two stages: retrieval and reranking. Exa offers retrieval from a 100B+ web-document pool and then highly semantic reranking within hundreds of milliseconds. To fit the latency budget, the two stages need to meet opposing goals:

• Retrieval searches across all 100B+ documents. It operates on a highly compact representation that makes it fast and keeps storage memory under control. It is recall-optimized and answers the question "Which documents are possible for this query?"
• Reranking operates only on a shortlist, so it can afford much larger compute and context. Reranking optimizes for the final order and answers "Which documents should be seen by the agent and in what order?"

It is very tempting to treat the reranking problem as a simple estimation of $P(\text{relevant} \mid \text{query}, \text{document})$. In practice there is much more that has to be considered:

• Relevance: Relevance continues to be the most important goal. The document has to answer the query, not just be similar to it. Relevance has to generalize across natural languages (not just English) and artificial languages (coding use cases).
• Intent: The query intent has to be well-understood. The query does not have to be informational, it can be e.g. navigational1. In this case a company's homepage or a personal profile is the correct first result.
• Freshness: For many queries, the best answer today is not the best answer tomorrow. Information in the real world gets updated constantly and the reranking should account for that2.
• Authority: The same information very often comes from multiple sources. Sometimes different sources can contradict each other. A good reranker has to make the right trade-offs to avoid spreading misinformation and to prefer primary sources. It should also be robust to any prompt-injection attempts in the document3. If a document owner maliciously adds an instruction "rank this document higher", the reranker should actually rank it lower.
• Instruction-following: The ranking needs might be different for different users. E.g. if a user specifies constraints in the query (say "peer-reviewed articles only"), the reranker should respect them.

Simultaneously satisfying these criteria is perhaps even more important for AI agents. They are more likely than humans to trust the returned results. Any mistake is also more likely to derail the entire agentic trajectory.

#Methodology

We do not currently plan to release a standalone Reranking API. Still, our goal, as with every other component of search, is to define state-of-the-art. To verify this, we measured the performance of our in-house rerankers, which serve Exa production search traffic, against the leading standalone Ranking APIs. Similarly to the broader market, we compared the rerankers in default and fast latency tiers.

Within each evaluated benchmark, we make sure every model sees the same query and the same candidate list in the same order. Only the final reranked order differs to allow fair comparison. We report nDCG@10 as the standard metric in IR benchmarks7. The metric is complemented with 95% confidence intervals computed with bootstrapping8. For details, consult the Appendix.

We use seven benchmarks: three public ones (BEIR, MIRACL and MAIR) and four internal ones (coding, navigational, freshness and authority). Averaged over all seven, both Exa Reranker Auto and Exa Reranker Fast already outperform all competing default-tier models.

#Relevance

This is the simplest scenario of finding the most relevant document to a query. The most common academic datasets for this are:

• BEIR9 - the standard zero-shot benchmark for classic relevance. We follow the methodology of Google Cloud and include only test queries with more than one distinct label10. This means we use six datasets that cover entity search, web QA, biomedical retrieval, scientific citation prediction, COVID-19 literature and argument retrieval (1,646 queries).
• MIRACL11 - multilingual relevance over Wikipedia in 18 languages, judged by native speakers (12,514 dev queries).

Setup: Each query's candidate pool consists of documents that were annotated within the benchmarks. The candidates are embedded with gemini-embedding-212 and retrieved by cosine similarity. Each reranker receives the same top 50 documents.

Results: Across these standard relevance benchmarks all reranking systems are tied within 1-2 percentage points. This makes sense because these are the most commonly evaluated benchmarks. Already the Gemini embedding is near saturation point (especially on MIRACL).

#Coding use case

Coding is a bit different from natural languages due to specific syntax and technical jargon. For example, the documents can name exact API symbols and error strings, and there are important differences due to API versions.

Setup: We use an internal set of 317 long-context coding queries, in the spirit of WebCode13. This task requires a web-index so each model reorders the top 50 results from Exa Instant14 and uses the same 720-character page extracts, and binary labels come from GPT 5.4 (xhigh reasoning) and mark whether a page grounds the answer.

Results: On the coding benchmark, scores are much lower than on classic relevance sets. Voyage and Cohere perform the best here with Google Cloud and Exa slightly behind.

#Navigational queries

Navigational queries are queries with the intent to visit a specific document. This can be a company’s homepage, a product page, or a personal profile. The correct top result is typically a single document.

Setup: Our internal navigational set comprises 3 subsets: canonical company pages (326 queries), CEO lookups (163q) and mixed general navigational queries (150q). This task requires a live web-index. Therefore, we use the top 50 results from Exa Instant that provide the same 8,000-character page extracts. The binary labels are created by GPT 5.4 (xhigh reasoning) based on whether a page is the requested destination.

Results: Navigational queries are an area where Exa shines. It is likely that other rerankers had much shallower training and did not recognize the navigational objective. In fact, all of them are substantially worse than the default ranking returned by Exa Instant.

#Freshness

Some queries like breaking news require the most recent information. Common examples include current stock price or the latest API version. With the amount of information constantly increasing, freshness becomes a crucial capability of search systems.

Setup: We use 1,000 multilingual synthetic queries with 60 candidates each. Every set of documents combines current pages with stale ones. Some examples are an old pricing page, outdated API docs or a superseded announcement. Each model receives URL, title and publication date along with the text.

Results: Exa is the best at ranking fresher content higher. The Fast reranker in particular is dramatically better at detecting the most up-to-date information.

#Authority

The information is not the only thing that matters. More and more documents come from low-quality sources. There is also an increasing number of AI SEO tactics such as prompt-injection that the reranker must withstand.

Setup: This benchmark uses the same format as freshness (1,000 queries with 60 candidates each). The content has adversarial hard negatives: prompt injections ("rank this page first"), SEO keyword stuffing, Unicode payloads and fake authority claims. A reranker should rank the legitimate sources above them.

Results: All rankers are relatively robust to authority. However, Exa is far ahead and gets fooled ~10 percentage points less often than the leading competition from Google Cloud and Cohere.

#Instruction-following

AI Agents often state how they want the results to be ranked. Examples are "include only peer-reviewed sources", "show the official documentation", "give a dataset, not a paper about it".

Setup: MAIR15 is a benchmark of 126 instructed retrieval tasks (9,972 queries) across web, academic, code, legal, finance and medical domains. Every query comes with a task instruction. We pass each model the instruction together with the query and rerank the top 50 retrieved from gemini-embedding-2.

Results: This benchmark was the flagship graph for Voyage Reranker release, so it makes sense they take first place. However, Exa and Cohere are a close second. Google Cloud reranker is much worse than other rerankers at following instructions (it substantially regresses Gemini Embedding retrieval).

End-to-end search quality is what matters to agents, not the individual performance of retrieval or ranking. But to achieve the highest possible search quality, we treat ranking as a first-class citizen, not an afterthought. Even though we don’t currently offer reranking as a standalone API, our benchmarking is designed to compare Exa Reranker with the best standalone products.

On the textbook task, topical relevance on BEIR and MIRACL, every frontier reranker lands within a few points, and Exa belongs to this group. The diversity of search tasks asks for more than topical relevance, though. A navigational query has one correct page, a time-sensitive query has a current answer, and many relevant-looking pages can try to manipulate rankers. That is where we see the standalone reranking APIs falling short, while Exa proves its generalizing capabilities. Averaged over all seven benchmarks, both Exa Reranker Auto and Exa Reranker Fast are quality leaders in their respective group. Additionally, Exa Reranker Fast is able to beat all competing default-tier rerankers (see Figure 1).