Reranking and retrieval recall

Learning outcome: Separate finding candidates from ordering them.

A first retrieval stage gathers candidate passages. A reranker then reorders those candidates using another relevance assessment. Reranking cannot promote a correct passage that never entered its candidate set.

Measure the stages separately. Retrieval recall at a chosen cut-off asks whether relevant evidence was found among the candidates. Ranking quality asks how near the top it appears. For a small manual evaluation, record the expected source, whether it was retrieved and where it appeared. If the evidence is missing, investigate indexing, filtering or candidate retrieval before spending time rewriting the answer prompt.

Worked example

In a fictional ten-question test, the correct source appears in the initial candidates for seven questions. A reranker moves six of those seven to the top. The three missing sources remain missing. Reporting only “six excellent answers” conceals the candidate-retrieval problem and gives the wrong team an unclear task to fix.

Try it yourself

Build a ten-row evaluation sheet with expected source, retrieved yes/no, rank and answer supported yes/no. Fill three rows where the correct source is absent.

You can do this exercise on paper. If using an assistant, keep to the fictional material and an approved tool.

Check your understanding

The correct passage is absent from every candidate. What can reranking alone do?

1. It cannot recover that missing passage
2. It can always create the missing evidence
3. It makes the top result authoritative

Workbook

Download the course workbook to record your answer and compare it after the knowledge check.

Further reading

• Elastic: hybrid search
• Elastic: semantic reranking