The retrieval pipeline
Ask a company chatbot "what's our refund policy for annual plans?" and it hits a wall: the answer lives in a document the model has never read. A large language model (LLM) answers from what training froze into it, so it does one of two things — admits it doesn't know, or invents something fluent, confident and wrong. That second failure is a hallucination, and it is the expensive one, because nothing in the answer looks wrong.
Retrieval-Augmented Generation (RAG) is the standard fix, and the idea is almost insultingly simple: look the answer up first, then ask the model to write it up. The engineering is all in the "look it up" part. This lesson builds that machinery — the end-to-end pipeline first, then the three components inside it that do the actual work.
Two kinds of knowledge
An LLM can draw on exactly two sources of knowledge, and the distinction is the whole premise of RAG.
Parametric knowledge is baked into the model's weights during training and frozen there."Parametric" because the knowledge lives in the model's parameters — the numbers adjusted during training. It is broad, it is free at inference time, and it cannot be updated without retraining.
Source knowledge is whatever text you place in the prompt at question time. It costs tokens, it is bounded by the model's context window (the maximum amount of text the model can read in one request), and it is completely under your control.
Anything that wasn't in the training data — proprietary documents, a live code repository, one customer's order history, yesterday's policy change — can only ever reach the model as source knowledge. There is no third option.
RAG: making it an open-book exam
RAG is an architectural pattern, not a model change. You don't fine-tune and you don't retrain. You treat the LLM as a stateless compute engine — it remembers nothing between requests, so every request must carry everything it needs — and at question time the system:
- searches a store of your own documents and pulls out the few short passages (chunks) most likely to contain the answer,
- pastes those passages into the prompt, with an instruction to answer from them rather than from memory.
A closed-book exam becomes an open-book one. The model no longer has to recall the answer; it only has to read the page you handed it and write a reply. That shift is why RAG suppresses hallucination: the facts are in front of the model, not dredged out of its weights.
The naive pipeline: two paths, one database
The baseline version — usually called naive RAG — is a straight line: no branching, no retries, no second opinions. What makes it two things rather than one is when each half runs.
The ingestion path — offline, the write path
This runs ahead of time, once per version of a document, with nobody waiting on it. Take the employee handbook as an example:
- Load and parse. Pull the text out of whatever the file actually is — a PDF, a Markdown page in a repository, a row in a CMS. What survives is a long string, plus whatever metadata you choose to keep with it: source file, page number, section heading. That metadata travels with the chunk into the database and is what later lets an answer say where it came from.
- Split. Cut that string into chunks of a few hundred tokens eachA token is a unit of text from the model's vocabulary — roughly a short word or a word fragment. English prose averages about four characters per token. — small enough that one chunk is about one thing, large enough that it still makes sense read on its own. The handbook's annual-plan refund rules become their own chunk, separate from the rules on parental leave.
- Embed. Send every chunk through an embedding model, which returns a vector: a fixed-length list of numbers that encodes what the chunk means. Similar meanings produce nearby vectors.
- Store. Write each vector into a vector database, together with the chunk's original text and where it came from. The vector is the search key; the text is what you will eventually hand the LLM.
(Optional) What step 1 throws away — and what happens to images
Parsing is lossy, and the losses are not all equal. Styling is no loss at all. Layout often is: a table flattened into a run of words can stop meaning anything, which is why better parsers re-emit structure as Markdown rather than plain text.
Anything with no text in it — a diagram, a photo, a scanned page — yields nothing to extract, and a text-only pipeline simply drops those. A pipeline that needs them has two options: describe each image with a vision model and index the description as if it were text, or use a multimodal embedding model that places images and text in one shared vector space, so a question can match a picture directly.
Notice that not one step above depends on any particular question, which is why the whole path can be slow, batched and scheduled nightly. This is precomputation in exactly the sense a database index is: do the expensive work once, off the request path, so the request itself is cheap.
The inference path — online, the read path
This runs once per question, inside the few seconds a user will tolerate, and it can only use what ingestion has already stored:
- Embed the question. The same embedding model turns "what's our refund policy for annual plans?" into a vector, in the same space as the chunks.
- Search. The database finds the stored vectors nearest to the question's vector and returns the top k — typically three to five — along with their original text. This is the only step that actually decides whether the system can answer at all.
- Build the prompt. Paste those chunks into a template, alongside the question and an instruction like "answer using only the context above; if it isn't there, say so."
- Generate. The LLM reads the assembled prompt and writes the answer.
End to end: the handbook was chunked and embedded last night; this morning the question is embedded, lands near the chunk about annual-plan refunds, and that chunk is pasted into a prompt. The model never "knew" the policy — it read it, half a second ago.
In pseudo-code, stripped to the structure:
# Write path — runs offline, once per document version
def ingest(documents, embedder, store):
for doc in documents:
for chunk in split(doc.extract_text(), size=500, overlap=50): # ①
store.insert( # ③
vector=embedder.embed(chunk), # ② the search key
payload={"text": chunk, "doc_id": doc.id}, # what the LLM reads
)
# Read path — runs online, once per question
def answer(question, embedder, store, llm):
query_vector = embedder.embed(question) # ② the same model
matches = store.search_nearest(query_vector, top_k=3) # ③
context = "\n---\n".join(m.payload["text"] for m in matches)
return llm.generate(f"""
Context:
---
{context}
---
Answer the question using only the context above. If the context does not
contain the answer, say so.
Question: {question}
""")
Two lines in there carry more weight than they look:
embedderis the same object on both paths. Chunks and questions have to be turned into numbers by the same model, or the two sets of numbers mean different things and nearness between them is meaningless. That also makes the embedding model the hardest thing in the system to change: every stored vector came from the old one, so switching means re-embedding the entire corpus.payloadcarries the original text, not just the vector. The vector is only the search key — it can't be read back into words. The text stored beside it is what actually reaches the LLM.
That leaves three components doing the real work, and each one is a decision with consequences:
| Component | What it decides | |
|---|---|---|
| ① | The splitter | where documents get cut — and therefore what can be found at all |
| ② | The embedding model | what "similar meaning" means, numerically |
| ③ | The vector store | how fast search is, and what it costs in RAM |
① Chunking: cutting documents into pieces
Box ① in the pipeline diagram — "parse and chunk".
Company knowledge doesn't arrive as neat paragraphs. It is trapped in PDFs with two-column layouts, Markdown repositories, nested JSON payloads — none of which map onto an LLM's input natively. Text splitting (chunking) is the step that turns one continuous string into separate, individually addressable blocks small enough to embed and to pass to a model.
Chunking decides what can be found later, so it deserves more thought than it usually gets. Four strategies, each answering a weakness in the one before.
Character-based splitting
Cuts strictly on length. Set the threshold to 500 and the string is cut at exactly character 500, every time. It costs essentially nothing in CPU or memory, and that is its only virtue: it ignores punctuation, sentence ends and syntax entirely, so it routinely guillotines a word, a number or a clause down the middle.
Recursive character splitting
The sensible default. It pairs a priority-ordered list of separators — paragraph breaks (\n\n), then line breaks (\n), then spaces, then bare characters — with two hard constraints:
- Chunk limit (
chunk_size) — the most characters or tokens one chunk may hold, so the payload fits both the embedding model's input and the LLM's context window. - Chunk overlap (
chunk_overlap) — a lookback window that copies the tail of a finished chunk onto the front of the next one, so an idea that straddles a boundary survives in at least one chunk whole.
What makes it "recursive" is that it doesn't pick one separator for the whole document. It runs a local split → check → merge or recurse loop. Suppose the limit is 500 characters:
- Split the text on the highest-priority separator,
\n\n. - Walk the resulting paragraphs, merging them into the current chunk while the running total stays safely under 500.
- When the next paragraph won't fit, close the chunk — then count backwards through what was just written to copy
chunk_overlapworth of trailing text forward as the seed of the next chunk. - If a single paragraph exceeds 500 on its own, pause the merging loop for that paragraph only and drop to the next finer separator (
\n, then spaces) to break it up, with overlaps. Once it is resolved, pop back up and carry on at paragraph level for the rest of the document.
So the fine-grained separators are an escape hatch for the awkward blocks, not a global setting.
chunk_size 120 and chunk_overlap 30. Paragraphs merge while they fit; ¶3 busts the limit on its own, so the splitter drops to a finer separator for that block only, then returns to paragraph level for ¶4. Every chunk after the first opens with the last 30 characters of the one before it — the dashed arrows — so an idea sitting on a boundary survives whole in at least one of them.Specialized structural splitting
Uses the file's own grammar instead of generic separators — an abstract syntax tree for source code, strict markup rules for documentation. This matters because a recursive splitter will happily cut a loop body in half, or strand a Markdown table from the subheading that explains what it tabulates. A structural splitter targets the syntax explicitly, keeping an entire Python class declaration or a complete Markdown header block as one unbreakable unit, whatever its character count.
Semantic splitting
Throws out syntax boundaries and cuts where the meaning turns. It parses the document into individual sentences, embeds each one, and measures the distance between adjacent sentence vectors. Wherever that distance spikes past a calibrated threshold — typically some number of standard deviations above the document's normal sentence-to-sentence drift — it declares a topic transition and cuts between the two sentences. The chunks come out remarkably self-contained.
Two things are commonly misstated about it. First, no LLM is involved: an embedding model represents the meaning, and the topic-shift detection itself is plain vector arithmetic. Second, the expensive part is not that arithmetic — it is having to run an embedding pass over every sentence in the corpus, which is what makes semantic splitting slow and costly at ingestion time.
② Embedding: turning text into coordinates
Both ② boxes in the pipeline diagram — one model, used on both paths.
A database can't compare meanings; it compares numbers. So every chunk is translated into a mathematical object it can index — a vector, a fixed-length list of numbers — by a bi-encoder.
A bi-encoder maps text into a single high-dimensional vector space, emitting a dense vector of typically 768 to 1,536 floating-point values. "Dense" means every one of those numbers carries signal, as opposed to the mostly-zero vectors keyword methods produce. The model is trained so that texts sharing a meaning land near each other in that space even when they share no vocabulary: "how do I reset my password" and "steps to recover account access" end up neighbours, while "quarterly revenue by region" sits far away. Picture each chunk as a point in a space of hundreds of dimensions, where distance is disagreement in meaning.
Why "bi"-encoder
The name describes the wiring: a chunk and a query go through two entirely independent passes, and the model never sees the two together.
Wiring it that way is only possible because of something the model can genuinely do — place a piece of text at an absolute position in a space it learned once, during training, from meaning alone. "Steps to recover account access" gets its coordinates from what it means, and from nothing else: not from the chunks around it, not from the size or contents of the corpus, and not from a question nobody has asked yet. Two texts written years apart by people who never met land next to each other because the same learned map placed them both — never because anything ever held them side by side and compared them.
That ability is what lets the offline path exist. A chunk's vector is correct for every question anyone will ever ask, so you can compute all of them months in advance and store them, and a question only has to be embedded once, live, in a few milliseconds.
Now imagine a model that required both texts in the same pass, scoring the pair directly. It emits no chunk vector at all — only a relevance score for the pair in front of it — so there is nothing to store ahead of time, and every question means running the model once per chunk: five million model calls to answer one question.
What separates the two designs is where the comparison happens. In the paired model it happens inside the model itself — which is why it can never be done ahead of time, and never cheaply. In a bi-encoder the model's job stops at turning text into numbers; that is the expensive part, and it runs once per chunk, offline. The comparison then drops down into the vector database, which has nothing harder to do than arithmetic on numbers already sitting on disk. Arithmetic at that scale is exactly what a vector index is built for, and that is what turns retrieval from a scan into a lookup.
That trade also caps how accurate retrieval can be, and the next lesson brings the paired model back in a place where it's affordable.
Measuring "close": cosine vs L2
At query time, nearness has to be an actual number. Two metrics show up:
- Cosine similarity measures the cosine of the angle between two vectors and ignores their lengths entirely. It asks whether they point the same way — whether they mean the same thing — not how much text is behind them. That makes it the default for RAG, where a five-word question has to be matched against a multi-sentence chunk.
- Euclidean (L2) distance measures the straight-line distance between two points, and unlike cosine it is sensitive to vector magnitude. It belongs in systems where magnitude itself is meaningful signal.
Choosing between them sounds like a real decision. Usually it isn't. If the vectors are normalized to unit length — which most embedding models do by default — then L2 and cosine produce the same ranking, and picking between them changes nothing but the number you print. The two only come apart on unnormalized vectors, where magnitude can shift the ranking on its own. So: normalize and use cosine, unless magnitude is signal you specifically want.
③ The vector store: searching millions of vectors fast
Both ③ boxes in the pipeline diagram — one component, written by ingestion and read by inference.
Where do a few million vectors live, and how do you find the nearest one in milliseconds?
Not in a B-tree or a hash index.Those are covered in Finding the needle — and both assume an ordering or an exact key, neither of which exists in 1,536 dimensions. Those structures answer "equal to this" or "between these two values"; a vector query asks "nearest point in 1,536-dimensional space," which has no such ordering to exploit. The honest alternative is brute force: compute the distance from the query to every stored vector. That is linear in the number of records, and at a few million rows it stalls production traffic.
To hold a sub-hundred-millisecond retrieval budget, vector stores use Approximate Nearest Neighbor (ANN) indexes. The name describes a family of algorithms rather than one algorithm, and every member strikes the same bargain: surrender a sliver of recall — the fraction of the genuinely best matches that the search actually returns — in exchange for an enormous speedup. Four structures are worth knowing, and only two of them are actually ANN indexes: IVF and HNSW. Of the other two, a flat index is exact rather than approximate, and product quantization is a compression scheme that rides underneath an index rather than a way of searching.
Flat index
No index structure at all: at query time, compute the exact distance to every stored vector. It never misses the best match, because it checks everything. But its query time grows linearly with the dataset, which rules it out at enterprise scale while leaving it a perfectly reasonable choice for small, isolated collections — a per-tenant store under roughly ten thousand chunks.
Inverted file index (IVF)
Partitions the space by clustering. When the index is built, an unsupervised algorithm (k-means, typically) groups the vectors into clusters, each summarized by a central anchor point called a centroid. At query time the system compares the query against the centroids only, picks the closest cluster, and searches exclusively inside that bucket. The number of distance calculations collapses from millions to thousands.
Its weakness is the cluster border: a highly relevant vector that happens to sit just outside the chosen cluster is never examined. In practice you tune that away by probing the n nearest clusters rather than one, trading speed back for recall. Worth being precise about what IVF does and doesn't save: it cuts search time, not memory — every full vector is still stored.
Hierarchical Navigable Small World (HNSW)
The modern production standard. It builds a multi-layered graph over the dataset. The upper layers are sparse, with long-range links that let a search leap across wide stretches of the space in a few hops; as the search closes in on the query's neighbourhood, it drops down through progressively denser layers of short links and finishes in roughly logarithmic time. The result is exceptional speed at high recall. The cost is memory: the entire multi-layer graph has to stay resident in RAM.
Product Quantization (PQ)
Not a search structure but a lossy compression scheme for the vectors themselves, typically layered underneath an index like IVF or HNSW rather than used alone. The problem it solves is pure infrastructure budget. A single 1,024-dimension vector of 32-bit floats occupies about 4,096 bytes; ten million chunks therefore need roughly 41 GB of RAM for the raw vectors alone, before any graph or index overhead — that is expensive hardware sitting idle most of the day.
PQ cuts each vector into small sub-vectors — say 128 of them, 8 dimensions apiece — and replaces each one with a single-byte code: the ID of the nearest entry in a small learned table of representative sub-vectors. That 4,096-byte vector shrinks to something like 64 to 128 bytes, and the same ten million chunks now fit in about 1 GB. The penalty is precision — distances computed from rounded codes are slightly wrong, so some rankings come out wrong with them.
37 in place of that sub-vector's eight floats is where the 32× saving comes from, and rounding each sub-vector to its nearest codebook entry is where the lost precision comes from.That is the complete naive pipeline, and it works beautifully on a demo corpus of twenty tidy documents. Real company data breaks it in four specific ways — the next lesson walks through each failure and the layer that fixes it.