How We Reduced RAG Latency by 40% with Hybrid Search and Custom Chunking

Retrieval-Augmented Generation (RAG) has emerged as the industry standard for grounding large language model (LLM) responses in private enterprise data. However, as vector databases grow and document sizes scale, search latency often becomes a critical bottleneck. If your RAG pipeline takes more than 1.5 seconds to retrieve context and generate a response, the end-user experience will feel sluggish and unresponsive. In this article, we share how we engineered a RAG pipeline that reduced retrieval latency by 40% while improving context relevance for an enterprise SaaS platform.

The Latency Problem in Naive RAG Implementations

A typical naive RAG setup consists of taking a query, embedding it, querying a vector database for the top-K matches, appending these matches to a prompt, and sending them to an LLM. While this works well in static sandboxes, production databases present significant challenges. Large, unstructured documents (such as manuals, contracts, or engineering specifications) often contain diverse topics within the same file. In addition, dense vector embeddings are prone to capturing general semantic concepts while missing specific keywords like serial codes or exact error messages.

The Limitations of Fixed-Size Chunking

Most default RAG implementations split documents using simple character-count or token-count windows (e.g., 500-token chunks with a 50-token overlap). While simple to set up, this approach frequently bisects sentences, breaks lists, and separates key context from its reference topic. The result is poor search precision and increased query latency as the LLM struggles to parse fragmented context.

To solve this, we implemented a custom semantic chunking pipeline. Instead of using arbitrary boundaries, we analyze sentence transitions by measuring the cosine distance of adjacent sentence embeddings. When the semantic similarity between two sentences drops below a dynamically calculated threshold, we create a chunk boundary. This ensures that every chunk represents a single, complete logical concept.

Implementing a Semantic Chunking Pipeline

Our semantic chunking process follows these steps:

• Sentence Tokenization: We split the source document into individual sentences using a sentence tokenizer.
• Embedding Generation: We generate embeddings for each sentence using a fast local embedding model.
• Similarity Calculation: We calculate the cosine distance between the embedding of sentence i and sentence i+1.
• Thresholding: We set the threshold at the 85th percentile of all adjacent distances. If the distance exceeds this threshold, we split the document.

For a detailed breakdown of how we architect these systems for scale, visit our applied AI & LLM engineering services page.

The Hybrid Search Architecture

Dense vector search is excellent at capturing conceptual meaning, but it frequently fails to retrieve files containing exact product serial numbers, error codes, or niche technical terms. To solve this, we designed a parallel two-stage retrieval pipeline:

• First Stage (High Recall): We execute dense vector search and sparse keyword search (BM25) in parallel. The vector search captures conceptual meaning, while the BM25 search guarantees keyword matches.
• Second Stage (High Precision): We merge the top 20 candidates from both retrieval models and pass them through a lightweight Cross-Encoder reranking model to extract the top 5 most relevant chunks.

Implementing the Hybrid Query Router

Here is the concurrent retrieval and reranking controller we deployed in production:

import concurrent.futures
from sentence_transformers import CrossEncoder

class HybridRetriever:
    def __init__(self, vector_client, postgres_client):
        self.vector_client = vector_client
        self.postgres_client = postgres_client
        self.reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

    def retrieve_hybrid(self, query, limit=20):
        # Execute sparse and dense searches concurrently to minimize latency
        with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
            future_vector = executor.submit(self.vector_client.search, query, limit=limit)
            future_bm25 = executor.submit(self.postgres_client.bm25_search, query, limit=limit)
            
            vector_results = future_vector.result()
            bm25_results = future_bm25.result()
        
        # De-duplicate and merge results based on identifier
        merged_pool = self.deduplicate_and_merge(vector_results, bm25_results)
        
        # Prepare candidates for reranking
        pairs = [[query, doc.text] for doc in merged_pool]
        scores = self.reranker.predict(pairs)
        
        # Sort docs by reranker scores
        for doc, score in zip(merged_pool, scores):
            doc.score = float(score)
        
        merged_pool.sort(key=lambda x: x.score, reverse=True)
        return merged_pool[:5]

    def deduplicate_and_merge(self, list_a, list_b):
        seen = set()
        merged = []
        for doc in list_a + list_b:
            if doc.id not in seen:
                seen.add(doc.id)
                merged.append(doc)
        return merged

Database Indexing Settings and Optimizations

To support fast vector lookups, we optimized the database index parameters. Using PostgreSQL with pgvector, we configured a Hierarchical Navigable Small World (HNSW) index on our embedding table. We tuned the index creation parameters to balance build time against search latency:

CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops) 
WITH (m = 16, ef_construction = 64);

Setting m = 16 defines the maximum number of bi-directional link connections per node, while ef_construction = 64 sets the size of the dynamic candidate list evaluated during index construction. This configuration allowed us to maintain fast index builds while keeping search recall above 98%.

Performance Outcomes and Latency Metrics

By moving the retrieval tasks to concurrent threads, shifting index generation to HNSW (Hierarchical Navigable Small World) configurations, and leveraging semantic chunking to avoid redundant contexts, we achieved the following metrics:

• P95 Retrieval Latency: Dropped from 340ms to 92ms.
• Context Relevance Score: Increased by 28% based on RAGAS evaluation framework.
• Total Inference Tokens: Reduced by 15% due to cleaner, more concise chunk inputs.

These optimization patterns demonstrate that building production-ready AI tools requires combining database indexing, multi-threaded programming, and semantic analysis. Ensuring these components work together smoothly is key to maintaining search performance at scale.