Hybrid Search and Reranking for RAG

The problem with using one search method

Most RAG systems begin with vector search. An embedding model maps text to vectors, and passages with similar meanings tend to sit close together, so a question about "cancelling a subscription" can find a passage about "ending your plan" even though the wording differs.

The weakness appears with specifics. An error code, a part number, a function name or a person's surname may carry little distinguishing meaning for an embedding model, so vector search can return passages that are broadly related but not the one containing that exact string.

Keyword search has the opposite profile. It handles exact terms well but has no built-in sense that "ending a plan" and "cancelling a subscription" are related. Hybrid search runs both methods and combines the results, so that each can cover some of the other's gaps.

How the retrieval pipeline fits together

A common arrangement has up to three ranking steps:

Candidate retrieval. Keyword and vector search each return a list of passages. Both should apply the same metadata and permission filters.
Fusion. The two lists are merged into a single ranking, with duplicates combined.
Reranking. Optionally, a slower and more precise model reorders the top of the merged list.

The final few passages then go to the LLM. Each stage narrows the set: first-stage search aims to avoid missing anything relevant across the whole index, and later stages aim to place the best passages at the top.

The vector side of this pipeline depends on your embedding model and store, which are covered in embeddings and choosing a vector database.

Keyword search: details that matter

BM25 is the usual ranking function for keyword search. It gives more weight to terms that are rare across the collection and that appear often in a passage, with an adjustment for passage length. Many search engines and some databases provide it or something similar.

How text is split into tokens affects exact matching. A standard analyser might split ERR-4012 into separate tokens or treat it in a way that hurts exact matching. Test the terms your users really search for, and consider a dedicated field for codes and identifiers. Stemming, stop words and synonyms are also language-dependent, so review those settings if your content is not in English or mixes several languages.

Merging results: fusion methods

You cannot simply add a BM25 score to a vector similarity score, as they are on different scales. A BM25 score is not bounded in the same way as cosine similarity, and its typical range varies with the query. There are two usual responses to this.

Reciprocal rank fusion

Reciprocal rank fusion, or RRF, ignores the scores and uses only each passage's position in each list. A passage's combined score is the sum of 1 / (k + rank) across the lists in which it appears, where k is a constant that dampens the advantage of the very top positions. A passage ranked reasonably well by both methods can therefore overtake one ranked first by only one of them.

# Illustrative only. Each ranking is a list of passage IDs, best first.
# k = 60 is a commonly used starting value, not a rule.
def reciprocal_rank_fusion(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, passage_id in enumerate(ranking, start=1):
            scores[passage_id] = scores.get(passage_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

merged = reciprocal_rank_fusion([keyword_ids, vector_ids])

RRF is a good default because it needs little tuning and is not disturbed by differences in score scale. Some systems add a weight to each list if one method consistently deserves more influence.

Weighted score combination

The alternative is to normalise each set of scores to a common range, then take a weighted sum. This retains information that RRF discards, such as a large gap between the first and second results. In exchange, normalisation can be sensitive to outliers, and the weight has to be tuned on a representative test set. If your questions vary widely, some systems choose the weight per query, for example leaning towards keyword search when a query contains an identifier. That adds another component to test, so it should follow from an observed need.

Reranking: a second, closer look

Search methods designed for large indexes have to be fast, which limits how closely they can compare a query with a passage. A bi-encoder embedding model, for example, represents the query and the passage separately. A reranker, often a cross-encoder, reads the query and passage together, so it can take into account how specific words relate to each other. It usually ranks more accurately, but is too slow to apply to every passage in a large index.

Hence the two-stage design: retrieve a modest number of candidates cheaply, then rerank just those. The numbers below are illustrative starting points to test, not recommendations:

# Illustrative only. The functions and limits are placeholders,
# and both searches return ranked passage IDs.
def retrieve(query, user):
    filters = permissions_for(user)
    keyword_hits = keyword_search(query, filters, limit=50)
    vector_hits = vector_search(embed(query), filters, limit=50)

    candidates = reciprocal_rank_fusion([keyword_hits, vector_hits])[:50]
    passages = load_passages(candidates)

    return rerank(query, passages)[:5]

Reranker trade-offs

Latency. Each query waits for the reranker to score every candidate, so a longer list means a longer wait. Measure this in your own setup.
Cost and hosting. You pay either per call to a hosted service or for the infrastructure, possibly including a GPU, to run a model yourself. Sending content to a third-party service also raises data handling questions.
Input limits. Rerankers often have an input length limit and may truncate long passages silently, so check how yours behaves, particularly if chunks are large.
Language fit. A reranker trained mainly on general English may not suit specialist or non-English content. Test before assuming it helps.
Score meaning. Reranker scores are useful for ordering, but do not assume they are calibrated probabilities. A fixed cut-off for "not relevant" needs testing and may need revisiting if the model changes.

Choosing your settings by measurement

Each of these components adds work, so each should demonstrate a benefit on your own data. Use a labelled question set, as described in how to evaluate a RAG system, and test in this order:

Get a baseline using vector search alone, or keyword search if that is what you have.
Add the second method with RRF. Look at recall at your chosen cut-off, broken down by question type. Exact-term questions are where you would most expect movement.
Set the candidate depth. Increase it until recall stops improving noticeably, since a reranker cannot recover anything outside this set.
Add the reranker. Check whether rank-sensitive measures such as mean reciprocal rank improve, and record the added latency.
Set the final context size. Passing too few passages can omit needed information, whereas too many add cost and distraction. Judge this using final answer quality too.

Then look through the individual failures. A rise in the average can hide questions that became worse, and those examples often reveal more than the summary figure.

Common pitfalls

Inconsistent filters. If permissions or date filters apply to one search method but not the other, restricted or stale content can reach the results. Enforce them in both, ideally through a shared code path.
Filtering after retrieval. Removing passages after taking the top results can leave few or none. Apply filters during the search where your store allows it.
Duplicate passages. Near-identical chunks can fill the final list and crowd out other sources, so consider deduplicating them or limiting the number of results per document.
Index drift. Keyword and vector indexes updated at different times can return different versions of a document. Keep their updates and deletions in step.
Query mismatch. A conversational follow-up needs rewriting into a standalone query before either search method can use it.
Blaming search for a content problem. If the chunk itself is unhelpful, ranking cannot fix it. The guide to choosing chunking strategies covers that side.

When you may not need these

A small collection might work well with one search method, and adding a reranker there may buy little beyond extra latency. Content with almost no exact identifiers may gain less from keyword search, while structured data questions may be better served by a database query. Treat hybrid search and reranking as responses to specific retrieval failures you have seen, rather than as mandatory parts of a RAG system.

If you would like help diagnosing or improving retrieval, see RAG development.

Frequently asked questions

What is hybrid search in RAG?

Hybrid search runs both keyword search and vector search for a query, then merges the results into one ranked list. It aims to catch exact terms that embeddings can miss as well as related meanings that keywords can miss.

What does a reranker do?

A reranker takes a query and a list of candidate passages, then scores each pair directly. The list is reordered by these scores. It is applied to a limited candidate set because it is generally too slow to run across a whole index.

Is hybrid search always better than vector search?

Not always. It often helps when queries contain codes, names or specialist terms, but the benefit depends on your content and questions. Compare both approaches on the same test set before accepting the added complexity.

Can reranking fix poor retrieval?

Only partly. A reranker can only reorder what it receives, so a passage missed by the earlier search stays missed. If the right passage is often absent from the candidates, work on first-stage retrieval, parsing or chunking first.

Hybrid search and reranking for RAG