How to Optimize a RAG System: 5 Levers That Actually Move the Needle

You stood up a RAG system in an afternoon — LangChain, a vector store, a few PDFs. It looked great in the demo. Three months later, it's in production with thousands of documents, and users are complaining. Bad answers, missed retrievals, hallucinations on anything remotely complex. Sound familiar?

Optimizing RAG is not a prompt engineering problem. It is an architecture and data engineering problem. Swapping GPT-4 for Claude or tweaking your system prompt will not fix low recall. This article covers the five levers that actually move the needle — in the order you should apply them.

These are not theoretical patterns. They are the interventions we reach for first after auditing production RAG systems. Applied together, we have seen systems move from ~67% to ~89% correct answers on internal benchmarks.

1. Hybrid Search: BM25 + vector retrieval

Vector search via embeddings is powerful, but it is not infallible. Many teams make the mistake of treating it as a complete solution and discarding keyword search entirely. That is a mistake — especially in domains like technical documentation, e-commerce, or legal, where exact matches matter.

Why semantics alone fails

Dense retrieval excels at capturing global meaning and context ("running shoes for trails"). It struggles with out-of-distribution vocabulary and exact matches. If a user queries a specific product reference ("SKU-12345"), a rare acronym, or a proper noun, the embedding can dilute that specific signal into a vector that matches nothing precisely. The information is there; the retrieval just can't find it.

BM25 + vectors: complementary by design

• BM25 (sparse retrieval): An improved TF-IDF. It does not care about meaning — it looks for exact keyword matches and their frequency. If a user types "Error 504 Gateway", BM25 will surface the exact document containing that string.
• Vector search (dense retrieval): Handles synonyms and intent. It finds the answer even if the user says "the server isn't responding" without mentioning the error code.

The real gain is not running both in parallel — it is fusing the results intelligently via RRF (Reciprocal Rank Fusion). RRF normalizes scores across both methods and surfaces documents that rank well on both semantic and lexical relevance. This is typically the highest-ROI change you can make to a RAG system with low recall. Most production vector stores — Qdrant, Weaviate, Elasticsearch — have native hybrid search support. For a detailed breakdown of how these options compare, see our vector database comparison.

For more on hybrid retrieval and how reranking plugs in on top, see our dedicated article on hybrid search and reranking.

2. Document Parsing: Garbage In, Garbage Out

The second pillar that teams consistently underestimate is parsing — everything that happens before chunking. The assumption that the LLM will compensate for poor input quality is wrong. Feed it garbage context and it will either hallucinate or hedge. Chunk quality is bounded by parse quality.

Extracting raw text with a basic Python script and pypdf is not enough for complex documents. PDFs are print formats, not data formats — they have no inherent notion of paragraphs, headings, or logical structure. This is the single biggest source of RAG failures on complex document corpora.

The Visual RAG era: tables, images, layouts

With vision-capable models like GPT-4o, Llama Vision, and specialized OCR models, the parsing problem has become more tractable — but it still requires deliberate engineering decisions.

Three document element types cause the most retrieval failures:

• Tables: Flattening a table to plain text line by line destroys the column/value relationships. The LLM cannot reason over numbers presented as a text dump. Convert tables to Markdown or structured HTML to preserve spatial semantics.
• Images and charts: Technical diagrams and financial charts often contain the answer. Either describe them textually via a VLM (captioning) or embed the image itself (multimodal embeddings). See our full treatment of this in Multimodal RAG: images, PDFs, and tables.
• Complex layouts: Double-column layouts, sidebars, headers and footers — these pollute the text flow and create chunks that mix unrelated content mid-sentence.

Invest in layout-aware parsers before you invest in chunking sophistication. Tools worth evaluating: Unstructured, Docling, Azure Document Intelligence, or vision-based approaches (GPT-4o Vision, DeepSeek OCR). The target output is clean, structured Markdown — get there before you think about chunk boundaries.

3. Chunking: From Fixed-Size to Semantic Boundaries

Chunking is over-indexed in the literature relative to its actual ROI. Teams spend weeks on semantic chunking implementations when their real bottleneck is parsing or hybrid search. That said, chunking strategy does matter — and fixed-size chunking is a meaningful source of retrieval failures.

Fixed chunking is blind. Cutting a sentence mid-phrase, or separating a question from its answer across two chunks, produces broken embedding vectors. The retrieval system is searching for coherent semantic units and finding fragments.

The embedding models themselves are a variable here — for a current comparison of model quality on retrieval tasks, see our embedding models 2026 guide.

Semantic chunking in practice

Semantic chunking replaces character-count splits with meaning-aware segmentation:

• Use a small embedding model to compute similarity between consecutive sentences.
• While topic similarity stays high, keep extending the chunk.
• When similarity drops (topic shift), close the chunk and start a new one.

This guarantees each chunk contains a complete, coherent idea — which directly improves retrieval precision.

Before committing to a full semantic chunking rewrite: add contextual chunk headers first (document title, section heading, date prepended to each chunk), implement Parent Document Retrieval second, measure the delta. In most cases, those two changes are sufficient and the semantic chunking investment does not justify the engineering cost.

4. Query Rewriting: Bridge the Semantic Gap

Even with a solid retrieval stack, the raw user query is often a poor retrieval key. The semantic distance problem: sometimes the answer contains no lexical overlap with the question.

• User query: "I can't log in, it just spins forever."
• Answer document: "OAuth2 authentication server timeout resolution procedure."

These two are semantically related but lexically distant. Embedding the user query and searching directly often fails. Query rewriting uses an intermediate LLM to translate user intent into a retrieval-optimized form.

Three techniques, ordered by implementation complexity:

1. Multi-Query Expansion: The LLM generates 3–4 variants of the question from different angles or with technical synonyms. Run all searches, deduplicate results. Casts a wider net without adding architectural complexity.
2. HyDE (Hypothetical Document Embeddings): Ask the LLM to hallucinate an ideal answer (factually wrong is fine). Embed that hypothetical answer and use it as the retrieval query. You are now searching answer-against-answer rather than question-against-answer. Semantic proximity is typically much better.
3. Query Decomposition: For complex, multi-part questions ("What was the revenue delta between Q2 and Q3 after the pricing change?"), decompose into sub-queries ("Q2 revenue?", "Q3 revenue?", "pricing change date?") and resolve sequentially. Pairs naturally with agentic RAG patterns for multi-hop retrieval.

Query rewriting adds latency (one extra LLM call per query) and cost. Start with Multi-Query Expansion — it is the simplest to implement and debug, and handles the majority of semantic-gap failures.

5. Reranking: Precision Finishing

Reranking is the last lever to reach for — after hybrid search, parsing, and chunking are solid. It is the cherry on top, not the foundation. Teams who add reranking before fixing retrieval fundamentals are polishing a broken pipeline.

Bi-encoder vs. cross-encoder

Your vector store uses bi-encoders: documents are pre-embedded independently, similarity is computed as dot product or cosine. Fast — but approximate. The score does not account for the specific interaction between the query and each document.

A cross-encoder reranker (Cohere Rerank, bge-reranker-v2-m3) reads the query and each candidate document together in a single forward pass and outputs a precise relevance score. It cannot be used for initial retrieval — it is too slow — but it is ideal as a second-stage filter.

The optimized pipeline:

1. Retrieve broad: Pull top-50 candidates from the vector store (fast).
2. Rerank: The cross-encoder scores all 50 against the query.
3. Top-K: Pass only the top-5 to the LLM.

This adds latency — 800ms to 1.5s is common for cross-encoders at query time. Profile it before shipping. If latency is a hard constraint, consider a faster bi-encoder reranker as a compromise, or batch async execution. The payoff: documents that should rank first but ended up at position 15 get correctly surfaced.

Evaluation: How to Know If Any of This Is Working

None of the above matters without measurement. Do not trust intuition. Do not trust spot checks. Build an evaluation pipeline before you optimize.

Use RAGAS, TruLens, or DeepEval to track three core metrics:

• Context Precision: Of the retrieved chunks, how many were actually needed to answer the question?
• Context Recall: Did the system retrieve all the information needed to answer correctly?
• Faithfulness: Does the LLM answer stay within the bounds of the retrieved context, or does it hallucinate?

The iteration loop: experiment (change chunk size, add hybrid search), evaluate (run RAGAS), fix, repeat. Always start by analyzing real failure cases from production logs — understand the errors before reaching for technical solutions.

For teams whose generic RAGAS scores stop correlating with user satisfaction, the next step is building domain-specific evaluators. We cover that in Building custom LLM judges. For a broader view on structured evaluation methodology, see our article on structured outputs in LLM production.

What to Tackle When Optimization Is Not Enough

These five levers cover most production RAG issues. But some failure modes require architectural shifts rather than optimization. When your retrieval pipeline is solid and you still hit ceilings, two directions are worth exploring:

• Agentic RAG: Give an LLM agent a retrieval tool and let it plan multi-step retrieval strategies. Handles complex multi-hop queries that single-shot retrieval cannot. Full breakdown in our Agentic RAG article.
• Fine-tuning the LLM: When the domain vocabulary is so specialized that even good retrieval cannot close the gap. See our comparison of fine-tuning vs. RAG vs. prompting for when each approach makes sense, and our LoRA/QLoRA fine-tuning guide for implementation details.
• Self-hosted architecture: When cost and data sovereignty become constraints at scale, see our guide to self-hosted RAG architecture.
• Multi-agent orchestration: When a single RAG agent is not enough and you need coordinated retrieval across systems — multi-agent orchestration comparison covers the main frameworks.

Further reading

• RAG: A Technical Guide — Architecture, chunking, vector stores, and RAG vs. fine-tuning fundamentals.
• Production RAG: 5 Failure Modes We Keep Seeing — Complementary post-mortem patterns: evaluation rigor, observability, latency, and cost.
• Hybrid Search and Reranking — Deep dive on BM25+vector fusion and cross-encoder reranking implementation.
• Vector Database Comparison — Qdrant, Weaviate, Pinecone, pgvector — how to choose based on your production constraints.
• Embedding Models 2026 — Current MTEB benchmark landscape and model selection for retrieval tasks.
• Multimodal RAG: images, PDFs, tables — How to handle documents where the signal is in images, scanned pages, or complex layouts.
• Agentic RAG — When retrieval becomes dynamic and multi-step. Covers planning, tool use, and when it is worth the complexity.
• Building custom LLM judges — When generic RAGAS metrics stop correlating with user satisfaction.
• Fine-tuning vs. RAG vs. Prompting — Decision framework for when optimization is not enough and you need to specialize the model.
• Advanced prompt engineering in production — Optimize the generation layer of your RAG pipeline.
• Deploying LLMs to production — Latency, cost, observability, and operational practices for production AI systems.
• RAG systems service — Tensoria's end-to-end engagement for deploying production RAG, including eval infrastructure.

Frequently Asked Questions on RAG Optimization