Improving RAG retrieval quality — chunking and reranking

If the answers are off after adding RAG, the problem is usually not the model but the retrieval stage. If the retrieved documents are wrong, no matter how good the LLM is, it cannot produce the correct answer.

Chunking — vary it by document type

Reranker — high return on cost

Retrieve 20 to 50 items with primary search (vector/BM25), then re-rank the top 5 with a reranker. The reranker scores the question and document together, so the accuracy improvement is significant.

• First-stage retrieval: fast but with semantic loss, so widen the candidate set
• Reranking: slower but more accurate, with a narrow candidate set
• A final top-k of 3–5 is appropriate — too many adds noise

hybrid search

Vector search alone is weak at exact string matching such as model names and part numbers. Using BM25 (keyword) together with vectors and merging the results is especially effective for industrial documents.

There is a case where the accuracy rate noticeably improved with a hybrid + reranking combination in customer manual search. Improving search comes before replacing the model.

No improvement without evaluation

• Label the correct documents for 30–50 questions
• Compare before and after changes using metrics such as Recall@5 and MRR
• Measure with the same set every time parameters are changed