Building a Production RAG Pipeline: LangChain + ChromaDB

Building a Retrieval-Augmented Generation (RAG) system is the "Hello World" of AI engineering in 2026. But 90% of tutorials stop at the "Naive RAG" stage. In production, this fails.

1. Naive vs. Production RAG

Naive RAG simply retrieves the top K chunks based on vector similarity.
Production RAG adds layers of intelligence:

• Hybrid Search: Combining Vector Search (Semantic) with BM25 (Keyword).
• Re-ranking: Using a cross-encoder to score the retrieved documents for true relevance, not just vector proximity.
• Query Expansion: Generating multiple variations of the user's question.

2. Architecture Overview

3. Advanced Ingestion Strategy

Don't just blind-chunk. Use ParentDocumentRetriever. This technique chunks documents into small pieces for embedding (better semantic match) but retrieves the entire parent section for the LLM (better context).

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_chroma import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter

# The trick: Small chunks for search, Big chunks for context
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)

store = InMemoryStore()
vectorstore = Chroma(collection_name="split_parents", embedding_function=embedding)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

Chunking Strategies Compared

Not all chunking is created equal. The strategy you choose directly impacts retrieval quality. Here's a breakdown of the most common approaches and when to use each.

4. Implementing Hybrid Search

Vector search is great for concepts ("How do I fix a leak?"), but terrible for exact matches ("Error code 503"). We need EnsembleRetriever.

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever

# 1. Keyword Retriever (BM25)
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 5

# 2. Semantic Retriever (Chroma)
chroma_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# 3. Combine them
ensemble_retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, chroma_retriever],
    weights=[0.4, 0.6]  # 40% Keyword, 60% Semantic
)

5. Re-ranking: The Secret Sauce

Retrieving 10 documents is easy. Knowing which one actually contains the answer is hard. Embedding models compress meaning into dense vectors, losing nuance. A Cross-Encoder Re-ranker takes the query and the document pair and outputs a relevance score.

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CohereRerank

compressor = CohereRerank(top_n=3)  # Only keep top 3 AFTER re-ranking
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=ensemble_retriever
)

# This pipeline is now: Hybrid Search (Top 10) -> Re-rank -> Top 3 -> LLM

6. Evaluation with RAGAS

You can't improve what you don't measure. I use the RAGAS framework to effectively "unit test" the pipeline.

• Faithfulness: Does the answer hallucinate?
• Answer Relevance: Is the answer useful?
• Context Recall: Did we retrieve the ground truth?
• Context Precision: Are the top-ranked docs actually relevant?

In my benchmarks, adding the Re-ranker increased Context Precision by 18%, drastically reducing hallucinations.

CI/CD Integration for RAG Testing

The biggest mistake teams make is treating RAG evaluation as a one-time exercise. In production, your knowledge base changes daily — new documents are indexed, old ones are deprecated. You need continuous evaluation baked into your CI/CD pipeline.

# .github/workflows/rag-eval.yml
name: RAG Evaluation
on:
  push:
    paths: ['knowledge_base/**']  # Trigger on KB changes

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run RAGAS Eval Suite
        run: |
          pip install ragas langchain-openai
          python eval/run_ragas.py \
            --dataset eval/golden_qa.json \
            --threshold-faithfulness 0.85 \
            --threshold-relevance 0.80
      - name: Fail if Regression
        run: python eval/check_regression.py

7. Tools of the Trade

Building production RAG in 2026 requires a modern stack. Here are the essential categories and my recommendations.

---