AI Engineering Part 3: RAG Pipelines & Vector Databases

Foundation Models are trained on public static data up to a fixed cutoff date. They do not know about your private company documentation, your customer’s database records, or real-time breaking events.

Furthermore, fine-tuning an LLM merely to teach it new facts is expensive, slow, prone to hallucinations, and risks data leakage.

To ground an LLM with fresh, dynamic, or private enterprise knowledge without retraining model weights, we use Retrieval-Augmented Generation (RAG).

In Part 3 of our AI Engineering masterclass series, we explore how to build production-grade RAG architectures that achieve 95%+ retrieval accuracy.

1. The Production RAG Pipeline Architecture

At its core, RAG combines two distinct software subsystems: an Information Retrieval (IR) system and a Generative Language Model.

2. Ingestion Phase: Chunking Algorithms

You cannot feed a 500-page PDF document as a single vector into a database. You must break documents down into smaller, self-contained textual segments called Chunks.

Choosing the right chunking strategy is the single most influential decision for RAG retrieval accuracy.

1. Fixed-Size Overlapping Chunking

Splits text strictly every N tokens or characters, with a percentage overlap (e.g., 500 tokens with 50-token overlap).

• Pros: Simple, fast, predictable chunk sizes.
• Cons: Mid-sentence splits; tears apart paragraphs and semantic context.

2. Recursive Character Chunking

Recursively attempts to split text using natural boundaries in hierarchical order: ["\n\n", "\n", " ", ""].

• Pros: Preserves paragraphs and complete sentences together.
• Industry Default: Standard recommended baseline for general text documents.

3. Semantic Chunking

Employs an embedding model to compute semantic distance between adjacent sentences. When the distance between sentence i and sentence i+1 exceeds a statistical threshold, a chunk boundary is placed!

• Pros: High semantic purity per chunk.
• Cons: Slower ingestion pipeline due to embedding calls during chunking.

3. Vector Embeddings & Similarity Metrics

An Embedding Model (e.g. OpenAI text-embedding-3-small, Cohere Embed v3) is a neural network that maps a textual chunk into a dense vector space (a fixed-length array of floating-point numbers, e.g. 1,536 dimensions).

"The dog chased the cat."    ──> Embedding Model ──> [ -0.012, 0.084, 0.312, ..., -0.091 ]
"A hound ran after the kitten." ─> Embedding Model ──> [ -0.011, 0.081, 0.309, ..., -0.088 ]
                                                          ▲ (Vectors are adjacent in 1536D space!)

Vector Distance Metrics

To measure how relevant a chunk vector B is to a query vector A, Vector Databases compute distance metrics:

1. COSINE SIMILARITY (Angle between vectors, Normalized [-1, 1]):
   Cosine(A, B) = (A · B) / ( |A| * |B| )

2. DOT PRODUCT (Magnitude + Angle):
   Dot(A, B) = A · B

3. EUCLIDEAN DISTANCE (L2 Distance in space):
   L2(A, B) = √( ∑ (A_i - B_i)² )

Performance Rule: If embedding vectors are unit-normalized (|A| = 1), Dot Product is mathematically identical to Cosine Similarity and runs significantly faster on modern SIMD/AVX hardware!

4. Vector Database Indexing: HNSW vs IVF

Performing exact brute-force O(N) nearest neighbor search across millions of 1536-dimensional vectors is far too slow for real-time applications.

Vector Databases (Pinecone, Qdrant, Weaviate, Milvus, pgvector) use Approximate Nearest Neighbor (ANN) indexing algorithms.

1. HNSW (Hierarchical Navigable Small World)

HNSW builds a multi-layer graph where top layers have long-range links for fast coarse routing, and bottom layers have dense short-range links for fine-grained search (similar to a SkipList).

• Recall: Extremely high (98%+ accuracy).
• Latency: Sub-10ms queries.
• Trade-off: Requires massive RAM overhead to store graph edges in memory.

2. IVF (Inverted File Index)

Clusters vector space into K Voronoi cells using k-means. At query time, the search engine identifies the nearest cell centroids and searches only vectors within those cells.

• Memory: Far lower RAM overhead than HNSW.
• Trade-off: Slightly lower recall accuracy; requires periodic index re-clustering.

5. Hybrid Search: Combining Vector & Keyword Retrieval

Pure vector search excels at semantic intent, but often fails catastrophically on keyword edge cases:

• Part numbers or SKU IDs (SKU-9482-X)
• Exact acronyms (HNSW, 2PC)
• Specific proper nouns (John von Neumann)

Production RAG systems use Hybrid Search: executing Dense Vector Search and Sparse Keyword Search (BM25) in parallel, and combining results!

Reciprocal Rank Fusion (RRF) Algorithm

How do you combine scores from two completely different scoring systems (BM25 scores range 0-50; Cosine scores range 0-1)?

You use Reciprocal Rank Fusion (RRF), which evaluates the rank position of documents rather than raw score values:

RRF_Score(doc) = ∑ ( 1 / (k + rank_m(doc)) )

where m represents the retrieval system (Vector or BM25), rank_m(doc) is the rank position of document doc in system m, and k is a smoothing constant (typically k = 60).

6. Advanced RAG: Cross-Encoder Reranking & HyDE

Naive RAG pipelines retrieve top-K chunks and dump them straight into the LLM prompt. This often injects irrelevant noise that degrades output quality.

To reach 95%+ precision, modern architectures insert two advanced layers:

1. Cross-Encoder Reranking

Bi-Encoder models (standard embeddings) convert query and chunk into vectors independently.

A Cross-Encoder Model (e.g. Cohere Rerank, bge-reranker-large) processes the Query and Chunk together through transformer attention layers simultaneously, outputting a hyper-accurate relevance score between 0.0 and 1.0!

2. HyDE (Hypothetical Document Embeddings)

If a user query is brief (e.g. “Why is my server slow?”), embedding that short query directly yields a poor vector match against long resolution documents.

HyDE (Gao et al., 2022) works in three steps:

1. Pass the raw query to a fast LLM to generate a hypothetical ideal answer document (even if it contains false details).
2. Embed the hypothetical answer document into vector space.
3. Use that dense hypothetical vector to search the vector database! The hypothetical answer matches true technical resolution documents far more closely than the original short query vector.

Summary of Part 3

In Part 3 of our AI Engineering masterclass series, we established:

1. Recursive Character Chunking is the recommended default, while Semantic Chunking maintains high paragraph purity.
2. HNSW Graph Indexing delivers sub-10ms vector lookups with high recall at the cost of higher RAM usage.
3. Hybrid Search (Dense Vector + BM25) combined via Reciprocal Rank Fusion (RRF) is mandatory for production retrieval.
4. Cross-Encoder Reranking filters out noise, selecting the absolute best 3-5 context chunks for the final LLM prompt.

Up next: AI Engineering Part 4: Agents, Tool Use & Autonomous Systems (ReAct, Function Calling, & Multi-Agent Architecture).