How to Build a Production-Grade RAG System in Python with FastAPI and LangChain
A 14-part build log for a real RAG service — hybrid search, reranking, JWT multi-tenancy, background jobs, caching, and the silent bugs each part caught along the way.
Most RAG tutorials stop at "load a PDF, ask a question, print an answer." That's a demo, not a system. It has no authentication, no source citations you can trust, no way to tell whether the retrieval is any good, and it falls over the moment two users share it.
This series builds the other thing: a RAG service you could actually put in front of users. Upload PDF, DOCX, or TXT files in Arabic or English, ask questions in natural language, and get answers with citations that point at a real document and a real page number. Along the way you'll add hybrid search, reranking, JWT authentication with multi-tenancy, background processing, caching, and tracing.
We'll build it in the order an experienced engineer actually would: get a thin slice working end-to-end first, then make each piece better once you can feel what's missing.
Everything here is verified by running it. Where a parameter silently does nothing, or a library quietly hands you the wrong answer, I show you the output that proves it — because those are the bugs that survive into production.
Code: the finished project is open source at github.com/Helmo21/RAG. Clone it to run the whole thing, or follow along below and build it yourself, part by part.
Table of Contents
- What You're Building
- Prerequisites
- Part 1: Project Scaffold and Configuration
- Part 2: Turning a PDF into Retrievable Chunks
- Part 3: Embeddings and the Vector Store
- Part 4: Answers With Citations
- Part 5: Putting It Behind an API
- Part 6: Multiple Formats and Arabic
- Part 7: Hybrid Search
- Part 8: Cross-Encoder Reranking
- Part 9: Measuring It
- Part 10: Authentication and Multi-Tenancy
- Part 11: Background Jobs
- Part 12: Caching
- Part 13: Observability
- Part 14: Docker and Shipping It
- What's Next
What You're Building
RAG stands for Retrieval-Augmented Generation. The idea is simple: language models don't know about your private documents, and fine-tuning them on your documents is slow and expensive. So instead, when a user asks a question, you retrieve the handful of passages most relevant to that question and paste them into the prompt. The model answers using the text you gave it.
That "retrieve the relevant passages" step is where all the engineering lives, and it's most of what this series is about.
Here's the finished architecture. Parts 1 and 2 cover the first two boxes:
Upload -> Load & chunk -> Embed -> Vector store
|
Question -> Hybrid search (BM25 + vectors) -> Rerank -> LLM -> Answer + citations
The stack: FastAPI for the API, LangChain for document loading and chaining, text-embedding-3-small for embeddings, Chroma for local vector storage, LangSmith for tracing, and RAGAS for evaluation.
Prerequisites
You'll need:
- Python 3.11 or newer. I'm on 3.12.
- An OpenAI API key, from platform.openai.com. Parts 1 and 2 don't call the API, but Part 3 does, and we set the key up now.
- Comfort with Python. You don't need to know anything about RAG, embeddings, or vector databases — that's what this explains. But you should be able to read a class and a function.
- Docker, eventually, for the final part. Not needed yet.
Part 1: Project Scaffold and Configuration
Before any AI, you need somewhere to put it. This part builds a runnable FastAPI skeleton with proper configuration and secrets handling. It's short, but two of the ideas here prevent bugs that are genuinely painful later.
Setting Up the Project
Create the project and a virtual environment:
mkdir rag && cd rag
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install fastapi uvicorn pydantic-settings
You're aiming for this layout:
rag/
├── app/
│ ├── __init__.py
│ ├── main.py # FastAPI app and routes
│ └── config.py # settings loaded from the environment
├── .env # your real secrets - never committed
├── .env.example # a template - safe to commit
├── .gitignore
└── requirements.txt
One detail that bites people: __init__.py has two underscores on each side. A single
underscore (_init_.py) looks nearly identical and Python 3 will appear to work anyway thanks to
implicit namespace packages — right up until imports between your modules start failing for reasons
that make no sense.
Handling Secrets Properly
Your OpenAI key costs real money, and keys leak through git history constantly. The convention that prevents it has three parts.
First, .env holds your real values and is never committed:
RAG_OPENAI_API_KEY=sk-your-real-key-here
Second, .env.example holds the same keys with no values, and is committed, so anyone
cloning your repo knows what to fill in:
RAG_OPENAI_API_KEY=""
Third, .gitignore keeps the real one out:
.env
.venv
__pycache__/
A subtle trap here: don't add .gitignore to .gitignore. It sounds tidy, but .gitignore must
be committed — it's how the ignore rules reach everyone else who clones the repo. If you ignore
it, it vanishes from git's view entirely and won't even show up as untracked.
Typed Configuration with pydantic-settings
You could read the key with os.getenv("OPENAI_API_KEY") wherever you need it. Don't. Scatter
os.getenv through a codebase and a missing variable becomes None, which flows downstream and
surfaces as a baffling error far from the actual cause.
Instead, define configuration once, with types, and let it validate at startup. Create
app/config.py:
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="RAG_",
env_file=".env",
env_file_encoding="utf-8",
)
openai_api_key: str
settings = Settings()
Small file, four ideas worth understanding:
BaseSettingsis a pydantic model that populates itself from environment variables instead of from arguments you pass in.env_prefix="RAG_"namespaces your variables. The fieldopenai_api_keycombined with the prefixRAG_maps to the environment variableRAG_OPENAI_API_KEY. The prefix keeps your config from colliding with unrelated variables that happen to share a name.env_file=".env"is the line people forget. Without it, pydantic-settings reads only real environment variables and ignores your.envfile completely. Your key sits there, correctly written, and is never loaded.settings = Settings()at module level instantiates the config exactly once, at import time. This is what gives you fail-fast: a missing key raises immediately, at startup, instead of becoming a confusing 401 from inside a library three parts from now.
Notice openai_api_key: str needs no Field(...) and no explicit environment variable name. In
pydantic-settings v2, the variable name is derived from the field name plus the prefix. The bare
type annotation does all of it.
The pydantic v1 to v2 Trap
This deserves its own section, because it will cost you an hour otherwise.
Search the web for pydantic-settings examples and you'll find a great deal of code like this:
# This is pydantic v1 syntax. It does NOT work in v2.
openai_api_key: str = Field(..., env="OPENAI_API_KEY")
In v2, the env= argument on Field is silently ignored — the name comes from the field plus
env_prefix instead. Blogs, older Stack Overflow answers, and language models trained on both are
full of v1 syntax presented as current.
The habit that saves you, and it takes ten seconds — ask the library what it actually has instead of trusting a tutorial:
python -c "import pydantic_settings; print([n for n in dir(pydantic_settings) if not n.startswith('_')])"
['AWSSecretsManagerSettingsSource', 'AzureKeyVaultSettingsSource', 'BaseSettings',
'CliApp', 'DotEnvSettingsSource', 'EnvSettingsSource', 'SettingsConfigDict', ...]
There's BaseSettings and SettingsConfigDict, and no Settings class to import. If your editor's
autocomplete or an AI assistant suggests an import that isn't in that list, it doesn't exist.
Get in the habit now — it matters much more in Part 2, where LangChain's API surface is enormous and
changes frequently.
The Health Endpoint
app/main.py is deliberately tiny:
from fastapi import FastAPI
from app.config import settings
app = FastAPI()
@app.get("/health")
async def root():
return {"status": "ok"}
The settings import looks unused, and that's the point. Importing it forces Settings() to run
at startup, so a broken configuration kills the app immediately rather than lurking. A health
endpoint that returns 200 while the app is misconfigured is worse than no health endpoint at all.
Note the import is from app.config import settings — not from config import settings. You run
the app as app.main:app, which makes app the package root, so imports resolve from there.
Verifying Part 1
Start it:
uvicorn app.main:app --reload
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO: Application startup complete.
curl http://127.0.0.1:8000/health
{"status":"ok"}
Now check the part that actually matters — that the key loads from .env:
python -c "from app.config import settings; print('KEY LOADED:', bool(settings.openai_api_key))"
KEY LOADED: True
Print a boolean, never the key itself — not even the first few characters. Key fragments in logs and terminal history are a real leak vector.
Finally, prove the fail-fast behavior by running from a directory with no .env and no variable
set:
pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
openai_api_key
Field required [type=missing, input_value={}, input_type=dict]
That's the payoff. A misconfigured deployment dies at boot with a precise message naming the exact field, instead of failing mysteriously mid-request.
Pin what you have, and commit:
pip freeze > requirements.txt # then trim it to your direct dependencies
git add . && git commit -m "Part 1: scaffold and config"
git show --stat HEAD # confirm .env is NOT in the list
Part 2: Turning a PDF into Retrievable Chunks
Now the RAG begins. The goal of this part: a PDF goes in, and a list of text chunks comes out, each one knowing which file and which page it came from. No embeddings yet, no AI calls.
Why Chunking Exists
The obvious question first: why not embed the whole document as one vector and be done?
Model limits. text-embedding-3-small accepts about 8,191 tokens. A 50-page PDF doesn't fit.
That's the boring reason.
Retrieval precision — the real reason. An embedding compresses meaning into a single point in space. Embed a whole 50-page report and you get the average meaning of the entire report: a blurry centroid that sits vaguely near everything and precisely on nothing. Ask "what was Q3 revenue?" and it can't point you at the sentence, because the sentence has been averaged in with 49 other pages about hiring and logistics. Chunks stay specific, so a chunk about Q3 revenue lands near that question in vector space.
Cost and context. You send only the relevant chunks to the model, not the whole book. That's the difference between a few hundred tokens and a few hundred thousand.
Citations. You cannot cite "page 34" if your unit of retrieval is the entire document. The chunk is the unit of citation. This is why the metadata work in this part matters so much — it's what makes Part 4's citations possible.
Chunk Size and Overlap
Two parameters, and you should be able to defend both.
Chunk size trades context against precision:
- Too small (say 100 characters) gives you sentence fragments. Context is lost, and an answer spanning two sentences gets split across chunks that each look irrelevant on their own.
- Too large (say 5,000 characters) brings back the blurry-average problem, and you burn tokens shipping irrelevant text to the model.
- A reasonable starting point is around 1,000 characters. Not sacred — a default to measure from.
Overlap fixes a specific failure. Imagine a chunk boundary landing mid-sentence:
chunk A: "...quarterly revenue grew to"
chunk B: "$4.2M in Q3."
Neither chunk can answer "what was Q3 revenue?" — the fact was guillotined. Overlap repeats the tail of one chunk at the head of the next, so ideas straddling a boundary survive in at least one chunk intact. The cost is mild duplication. It's worth it.
We'll start with 1,000 and 200, then measure whether the overlap is real. Spoiler: the naive choice silently gives you zero.
Installing the Ingestion Packages
Three packages, and it's worth knowing why each exists — LangChain is deliberately split into small pieces so you don't install the world:
pip install pypdf langchain-community langchain-text-splitters
pypdfis the actual PDF parser. It does the real work of reading bytes and pulling out text. LangChain doesn't parse PDFs itself; it delegates.langchain-communitycontainsPyPDFLoader, a thin adapter that wrapspypdfinto LangChain'sDocumentformat.langchain-text-splitterscontainsRecursiveCharacterTextSplitter. It's standalone and needs no LLM.
Import them and you'll notice something:
DeprecationWarning: `langchain-community` is being sunset and is no longer actively
maintained. See ... for migration guidance toward standalone integration packages.
langchain-community is on its way out, and at the time of writing there's no standalone PDF loader
package to replace it. So we'll use it — but we'll wrap it behind our own function so that swapping
it later touches exactly one file. That's the practical response to a deprecation you can't yet
avoid.
Making a Sample PDF
You need a PDF where you can verify page numbers unambiguously, so put a distinctive fact on each page. This script is a test fixture, not part of your app:
pip install reportlab
scripts/make_sample_pdf.py:
"""Generate a small sample PDF for testing ingestion."""
from reportlab.lib.pagesizes import A4
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.platypus import PageBreak, Paragraph, SimpleDocTemplate
OUTPUT = "sample.pdf"
PAGES = [
(
"PAGE ONE - COMPANY HISTORY",
"The Zorblatt Corporation was founded in 1987 in Helsinki, Finland. "
"It began as a small workshop producing precision instruments for the "
"shipping industry. Over the following decade the company expanded "
"steadily across the Nordic region, opening offices in Stockholm and "
"Oslo. By 1999 it employed over four hundred people and had become a "
"recognised supplier of navigation equipment. The founder, Erik "
"Lindqvist, remained closely involved in daily operations until his "
"retirement. The early culture of the company emphasised engineering "
"rigour above rapid growth, a philosophy that shaped its later "
"decisions and gave it a reputation for reliability among customers.",
),
(
"PAGE TWO - FINANCIAL RESULTS",
"Zorblatt reported revenue of 4.2 million euros in Q3 2025, an "
"increase of eleven percent compared with the same quarter of the "
"previous year. Operating margin improved to nineteen percent, driven "
"largely by the maritime sensors division. The board attributed the "
"result to disciplined cost control and stronger demand in the Baltic "
"market. Cash reserves at the end of the quarter stood at 8.9 million "
"euros. Analysts noted that the company had reduced its debt load for "
"the sixth consecutive quarter, leaving it well positioned to fund "
"further research without external financing in the coming year.",
),
(
"PAGE THREE - LEADERSHIP",
"The company's chief executive is Dr. Amina Rashid, appointed in 2023 "
"after eight years leading the research division. She holds a doctorate "
"in control systems engineering and has published widely on autonomous "
"navigation. Under her leadership the company launched its first "
"software subscription product, marking a shift away from pure hardware "
"sales. The executive team was restructured in early 2024 to add a "
"dedicated chief technology officer. Dr. Rashid has stated that her "
"primary objective is to double recurring revenue by the end of 2027 "
"while preserving the engineering culture the company was built on.",
),
]
def main() -> None:
doc = SimpleDocTemplate(OUTPUT, pagesize=A4)
styles = getSampleStyleSheet()
story = []
for index, (heading, body) in enumerate(PAGES):
story.append(Paragraph(heading, styles["Heading1"]))
story.append(Paragraph(body, styles["BodyText"]))
if index < len(PAGES) - 1:
story.append(PageBreak())
doc.build(story)
print(f"Wrote {OUTPUT} with {len(PAGES)} pages")
if __name__ == "__main__":
main()
SimpleDocTemplate builds the file, Paragraph wraps each block of text in a style, and
PageBreak forces the next page — that's what guarantees one fact per page.
python scripts/make_sample_pdf.py
Wrote sample.pdf with 3 pages
Three pages, each with an unmistakable fact: page 1 says founded in 1987, page 2 says revenue of 4.2 million, page 3 names the CEO.
Looking at What the Loader Gives You
Before splitting anything, look at the raw loader output. This is the most important thing in Part 2:
from langchain_community.document_loaders import PyPDFLoader
docs = PyPDFLoader("sample.pdf").load()
print(f"Loader returned {len(docs)} Document objects")
for d in docs:
print(f"metadata: {d.metadata}")
print(f"text starts: {d.page_content[:55]!r}")
Loader returned 3 Document objects
metadata: {'producer': 'ReportLab PDF Library - (opensource)', 'creator': '(unspecified)',
'creationdate': '2026-07-15T09:40:06+07:00', 'author': '(anonymous)', 'keywords': '',
'moddate': '2026-07-15T09:40:06+07:00', 'subject': '(unspecified)', 'title': '(anonymous)',
'trapped': '/False', 'source': 'sample.pdf', 'total_pages': 3, 'page': 0, 'page_label': '1'}
text starts: 'PAGE ONE - COMPANY HISTORY\nThe Zorblatt Corporation was'
...
Three things in that output.
The loader returns one Document per page, not one per file. A 3-page PDF gives 3 Documents.
That's deliberate: the loader's job is to preserve the document's natural structure and attach
metadata before any splitting happens. Page identity is captured here or lost forever.
page is 0-indexed. Look closely: the page whose text literally says PAGE ONE reports
'page': 0. Build citations on that field and your app will confidently tell users "see page 0", or
cite page 2 for something a human reads on page 3. It's plausible, silent, and wrong — the worst
kind of bug, because the answer looks right and nobody checks.
page_label is the label printed on the page — the string '1' here. It's a string because real
documents have pages labelled 'iv', 'A-1', or 'ix'. Page labels aren't always numbers, which
is exactly why the field exists alongside page.
The metadata is noisy. producer, trapped, creationdate — junk from the PDF spec. You don't
want that flowing through your pipeline into the vector store. We'll keep only what we need.
Splitting into Chunks
Now bring in the splitter and watch what happens to metadata:
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=80)
chunks = splitter.split_documents(docs)
print(f"{len(docs)} pages -> {len(chunks)} chunks")
for i, c in enumerate(chunks):
print(f'[{i}] page={c.metadata["page"]} len={len(c.page_content)}')
print(f" {c.page_content[:70]!r}")
3 pages -> 6 chunks
[0] page=0 len=398
'PAGE ONE - COMPANY HISTORY\nThe Zorblatt Corporation was founded in 198'
[1] page=0 len=293
'equipment. The founder, Erik Lindqvist, remained closely involved in d'
[2] page=1 len=326
'PAGE TWO - FINANCIAL RESULTS\nZorblatt reported revenue of 4.2 million '
[3] page=1 len=301
'and stronger demand in the Baltic market. Cash reserves at the end of '
...
Metadata propagates automatically. split_documents() copies each parent page's metadata onto
every child chunk. Chunk [3] is a fragment from the middle of page 2 and still knows it's page
2. This propagation is the entire citation mechanism — Part 4 won't build citations, it will just
read what you preserved here.
Chunk sizes vary — 398, 293, 326, not a uniform 400. That's the splitter doing its job:
chunk_size is a ceiling, not a target. This is also what "recursive" means in
RecursiveCharacterTextSplitter: it tries a hierarchy of separators — paragraph, then line, then
sentence, then word — and only descends to a finer one when a piece is still too big. The effect is
that it breaks at natural semantic seams rather than blindly at character 400. Plain
CharacterTextSplitter doesn't do this, which is why the recursive one is the sensible default.
Chunks never span two pages, because we split per-page Documents. Every chunk has exactly one unambiguous page — a free consequence of loading before splitting.
The Overlap Parameter That Silently Does Nothing
Look at the boundary between chunk 0 and chunk 1 above. Chunk 0 ends with "...supplier of navigation" and chunk 1 begins with "equipment. The founder...". That's contiguous, not
overlapping — and we asked for chunk_overlap=80.
Measure it rather than squint at it. This finds the longest suffix of A that is also a prefix of B:
def overlap_len(a, b):
for n in range(min(len(a), len(b)), 0, -1):
if a[-n:] == b[:n]:
return n
return 0
for sz, ov in [(400, 80), (400, 0), (200, 100)]:
ch = RecursiveCharacterTextSplitter(chunk_size=sz, chunk_overlap=ov).split_documents(docs)
pairs = [(ch[i], ch[i+1]) for i in range(len(ch)-1)
if ch[i].metadata["page"] == ch[i+1].metadata["page"]]
print(f"size={sz} overlap={ov} -> {[overlap_len(a.page_content, b.page_content) for a, b in pairs]}")
size=400 overlap=80 -> [0, 0, 0]
size=400 overlap=0 -> [0, 0, 0]
size=200 overlap=100 -> [95, 96, 93, 84, 95, 96, 97, 97, 0, 0, 98, 96, 86, 91, 98, 92]
Requesting 80 gives zero. Requesting 100 gives about 95. Why?
Look at the line lengths the PDF extraction produces:
print([len(l) for l in docs[0].page_content.split("\n")])
[26, 95, 96, 93, 84, 95, 96, 100]
There it is. PDF text extraction inserts a newline at every visual line wrap, so the text arrives as ~95-character lines. And here's the crucial mechanic:
chunk_overlap is applied in whole units, not characters. The splitter breaks text into units
by separator — here "\n" — then merges units into chunks. For overlap, it keeps trailing units
from the previous chunk while their total stays under chunk_overlap.
- With
chunk_overlap=80: one line is ~95 characters. 95 > 80, so not even a single line fits in the budget. Everything gets dropped, and you get zero overlap. You asked for 80 and silently received nothing. - With
chunk_overlap=100: one line fits, so you get ~95 characters of overlap. And notice the measured values[95, 96, 93, 84, ...]are exactly the line lengths — overlap arrives in whole lines.
The lesson: chunk_overlap must comfortably exceed your text's natural unit granularity, or it
degrades to nothing without complaining. Your app still runs. Retrieval is just quietly worse, and
you'd have no idea why.
This is why we measure instead of copying numbers from a blog post. It also settles our parameters:
chunk_size=1000, chunk_overlap=200. With ~95-character lines, 200 buys roughly two full
lines of genuine overlap.
Writing the Ingestion Module
Now the real thing. app/ingestion.py:
"""Load documents from disk and split them into retrievable chunks."""
from pathlib import Path
from langchain_community.document_loaders import PyPDFLoader
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200
def _page_number(metadata: dict) -> int:
"""Return the human-facing 1-based page number."""
label = metadata.get("page_label")
if label is not None and str(label).isdigit():
return int(label)
return int(metadata.get("page", 0)) + 1
def load_and_chunk(path: str | Path) -> list[Document]:
"""Load a PDF and split it into chunks carrying citation metadata."""
path = Path(path)
if not path.exists():
raise FileNotFoundError(f"No such document: {path}")
if path.suffix.lower() != ".pdf":
raise ValueError(f"Expected a .pdf file, got: {path.suffix or 'no extension'}")
pages = PyPDFLoader(str(path)).load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=CHUNK_SIZE,
chunk_overlap=CHUNK_OVERLAP,
)
chunks = splitter.split_documents(pages)
cleaned: list[Document] = []
for chunk in chunks:
if not chunk.page_content.strip():
continue
cleaned.append(
Document(
page_content=chunk.page_content,
metadata={
"source": path.name,
"page": _page_number(chunk.metadata),
},
)
)
return cleaned
What each piece is doing:
_page_number()is the fix for the 0-indexing trap. It preferspage_label, the label a reader would actually look for, and falls back topage + 1when the label isn't numeric — because'iv'and'A-1'are real page labels andint('iv')would crash.- The validation up front — existence and extension — fails fast with a clear message rather
than letting
pypdfthrow something cryptic from three frames down. - The rebuild loop is where metadata gets cleaned. We construct a fresh
Documentwith exactly two keys,sourceandpage, so none of the PDF-spec noise reaches the vector store. It also drops chunks that are only whitespace, which real PDFs produce more often than you'd like. path.namestoressample.pdf, not/home/you/docs/sample.pdf. A citation should show a filename, not your directory structure.
The module boundary is the other half of the design. PyPDFLoader is referenced in exactly one
place. When langchain-community finally disappears, or you decide pypdf handles tables badly
and want PyMuPDF instead, you change this file and nothing else. In Part 6, DOCX and TXT will
enter through this same door.
Verifying Part 2
Run it:
from app.ingestion import load_and_chunk
chunks = load_and_chunk("sample.pdf")
for i, c in enumerate(chunks):
print(f"[{i}] {c.metadata} len={len(c.page_content)}")
print(f" {c.page_content[:60]!r}")
[0] {'source': 'sample.pdf', 'page': 1} len=692
'PAGE ONE - COMPANY HISTORY\nThe Zorblatt Corporation was foun'
[1] {'source': 'sample.pdf', 'page': 2} len=628
'PAGE TWO - FINANCIAL RESULTS\nZorblatt reported revenue of 4.'
[2] {'source': 'sample.pdf', 'page': 3} len=636
"PAGE THREE - LEADERSHIP\nThe company's chief executive is Dr."
The chunk whose text says PAGE ONE now reports 'page': 1. The off-by-one is gone, and it's
verifiable by eye.
But don't verify by eye. This is exactly the bug class that survives a casual glance, so trace the distinctive facts:
facts = {"founded in 1987": 1, "4.2 million": 2, "Amina Rashid": 3}
for fact, expected in facts.items():
hits = [c.metadata["page"] for c in chunks if fact in c.page_content]
print(f"{'PASS' if hits == [expected] else 'FAIL'} {fact!r} -> {hits}")
PASS 'founded in 1987' -> [1]
PASS '4.2 million' -> [2]
PASS 'Amina Rashid' -> [3]
Every fact lands on the page a human would turn to. That's the property Part 4's citations depend on, and now it's pinned down by a test rather than a hope.
One honest caveat: each page here is around 650 characters, comfortably under the 1,000 ceiling, so each page produces exactly one chunk and nothing actually overlaps in this sample. The setting is right for real documents — try a longer PDF if you want to watch it work.
Pinning Dependencies From Reality
Pin the versions you actually ran, not the ones you remember. Hand-typed pins drift, and a
requirements.txt that installs a different version than the one you developed against defeats the
entire purpose of pinning:
pip list --format=freeze | grep -iE "^(fastapi|uvicorn|pydantic-settings|pypdf|langchain)"
requirements.txt:
# Web layer
fastapi==0.139.0
uvicorn==0.51.0
pydantic-settings==2.14.2
# Ingestion: PDF parsing + chunking
pypdf==6.14.2
langchain-core==1.4.9
langchain-text-splitters==1.1.2
langchain-community==0.4.2
reportlab is deliberately missing — it builds test fixtures and has no business in production,
where it would only add image-parsing code to your attack surface. Split it out:
requirements-dev.txt:
-r requirements.txt
reportlab==5.0.0
Add the generated fixture to .gitignore too, since the script rebuilds it on demand:
sample.pdf
Part 2 Takeaways
You now have a PDF turning into clean, citable chunks — the raw material for everything that follows. Two ideas carry forward more than any code here:
Metadata attached at load time is what makes citations possible. Every later part just passes it along.
Parameters can fail silently. chunk_overlap=80 did nothing at all, and the only reason you know
is that you measured it. RAG is full of these — the system runs, the answers are just quietly worse.
Before moving on, try feeding it a real PDF and checking whether the page numbers still line up.
Documents with cover pages and roman-numeral front matter are where page_label earns its keep.
Part 3: Embeddings and the Vector Store
Chunks of text aren't searchable by meaning. This part turns them into vectors — the representation that lets you ask "what did they earn?" and retrieve a passage that never uses the word "earn."
What an Embedding Actually Is
Strip away the mystique and an embedding is a list of numbers:
from langchain_huggingface import HuggingFaceEmbeddings
emb = HuggingFaceEmbeddings(model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
v = emb.embed_query("What was the revenue?")
print("dimensions:", len(v))
print("first 8 values:", [round(x, 4) for x in v[:8]])
print("magnitude:", round(sum(x*x for x in v) ** 0.5, 4))
dimensions: 384
first 8 values: [0.0081, 0.2163, -0.3072, 0.0217, -0.2065, 0.0587, 0.5927, 0.4252]
magnitude: 5.6184
384 floats. That's the whole thing. An embedding model is trained so that text with similar meaning produces vectors that point in similar directions. "Similar direction" is something you can compute, which is what makes meaning searchable.
Note the magnitude is 5.62, not 1.0. This model returns unnormalized vectors, while OpenAI's are unit length. That detail comes back to bite us shortly, when we pick a distance metric.
Seeing Semantic Search Work — and Fail
Here's the test that matters. Compare a query against candidate sentences using cosine similarity — the cosine of the angle between two vectors, which measures direction while ignoring magnitude:
def cosine(a, b):
dot = sum(x*y for x, y in zip(a, b))
return dot / ((sum(x*x for x in a)**0.5) * (sum(x*x for x in b)**0.5))
Run five queries against three candidate sentences and look at which one wins:
'What was the revenue?'
+0.3738 Zorblatt reported revenue of 4.2 million euros. <-- top
+0.2333 The corporation was established in Helsinki in 1987.
-0.0112 My cat is sleeping on the sofa.
'How much money did they make?'
+0.2432 Zorblatt reported revenue of 4.2 million euros. <-- top
+0.1291 The corporation was established in Helsinki in 1987.
-0.0208 My cat is sleeping on the sofa.
'When was it founded?'
-0.0360 Zorblatt reported revenue of 4.2 million euros.
+0.2781 The corporation was established in Helsinki in 1987. <-- top
-0.0942 My cat is sleeping on the sofa.
'What did the company earn?'
+0.2473 Zorblatt reported revenue of 4.2 million euros.
+0.3534 The corporation was established in Helsinki in 1987. <-- top
-0.0797 My cat is sleeping on the sofa.
"How much money did they make?" matched "revenue" with zero words in common. "When was it founded?" matched "established." No keyword search does that — this is retrieval on meaning, and it's the reason RAG works at all.
Now look at the fourth query. "What did the company earn?" ranked the founding sentence above the revenue sentence. The word "company" pulled hard toward "corporation" and beat the actual answer.
Don't skip past that. Semantic search is powerful and unreliable. Embeddings compress meaning into 384 numbers, and compression loses things. This exact failure is why Parts 7 and 8 exist: hybrid search adds keyword matching back in, and reranking re-scores candidates with a model that reads query and passage together instead of comparing two independently-made summaries.
Any tutorial that shows you only the queries that work is selling you something.
Choosing a Provider: Local or Hosted
project.txt calls for text-embedding-3-small. But when I first ran this, I got:
openai.RateLimitError: Error code: 429 - insufficient_quota
'You exceeded your current quota, please check your plan and billing details'
A billing error, not a code error — my key was valid, the account just had no credit. If you're on a free-tier key you'll hit exactly this.
That's annoying, but it points at a real design principle: which embedding model you use should be configuration, not code. Production wants the hosted model; development and CI want something free and offline. Hardcoding either one is a mistake.
So we support both. Extend app/config.py:
from typing import Literal
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="RAG_",
env_file=".env",
env_file_encoding="utf-8",
)
embedding_provider: Literal["local", "openai"] = "local"
local_embedding_model: str = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
openai_embedding_model: str = "text-embedding-3-small"
openai_api_key: str | None = None
chroma_dir: str = ".chroma"
chroma_collection: str = "documents"
settings = Settings()
Four decisions in there:
Literal["local", "openai"]means pydantic rejects a typo like"openia"at startup with a clear message, instead of failing deep inside a factory later.openai_api_keyis now optional. In Part 1 it was required, and that was right when OpenAI was the only path. The local provider needs no key, so demanding one would block the free route. We don't lose the fail-fast property — we move it into the factory, which raises if you selectopenaiwithout a key. Validate what's actually required, when it's actually required.openai_embedding_modelis explicit becauseOpenAIEmbeddingsdefaults totext-embedding-ada-002— the legacy model. Omitmodel=and you silently get an older, worse, more expensive embedding. Another quiet default.- The local model is multilingual. Arabic support is a requirement in Part 6, and picking an English-only model now would mean re-indexing everything later.
Install both paths:
pip install langchain-chroma langchain-openai
pip install langchain-huggingface sentence-transformers # pulls in torch: large
These are standalone integration packages, not langchain-community — exactly the migration
target that Part 2's deprecation warning pointed to.
The Embedding Factory
app/embeddings.py turns config into a model, and is the only place that names a provider:
"""Build the embedding model named by configuration."""
from functools import lru_cache
from langchain_core.embeddings import Embeddings
from app.config import settings
@lru_cache(maxsize=1)
def get_embeddings() -> Embeddings:
provider = settings.embedding_provider
if provider == "local":
# Imported lazily so the openai path never pays for importing torch.
from langchain_huggingface import HuggingFaceEmbeddings
return HuggingFaceEmbeddings(model_name=settings.local_embedding_model)
if provider == "openai":
from langchain_openai import OpenAIEmbeddings
if not settings.openai_api_key:
raise ValueError(
"embedding_provider='openai' requires RAG_OPENAI_API_KEY to be set"
)
return OpenAIEmbeddings(
model=settings.openai_embedding_model,
api_key=settings.openai_api_key,
)
raise ValueError(f"Unknown embedding_provider: {provider!r}")
Three things doing real work here:
- The return type is
Embeddings, LangChain's interface — notHuggingFaceEmbeddingsorOpenAIEmbeddings. Everything downstream depends on the interface, so nothing else in the codebase knows or cares which provider is live. @lru_cache(maxsize=1)matters more than it looks. A local model loads several hundred megabytes of weights; rebuilding it per request would be ruinous. One instance per process.- The imports are inside the branches.
import torchcosts seconds and hundreds of MB of RAM. Someone running the OpenAI provider should never pay for it.
Storing Vectors in Chroma
A vector store does one job: hold vectors and find the nearest ones to a query vector, fast. A brute-force scan compares your query against every chunk — fine at 3 chunks, hopeless at a million. Chroma uses an approximate index (HNSW) to search a large collection in sub-linear time, trading a little exactness for a lot of speed.
app/vectorstore.py:
"""Persistent Chroma vector store: index chunks, search them back out."""
import hashlib
from functools import lru_cache
from langchain_chroma import Chroma
from langchain_core.documents import Document
from app.config import settings
from app.embeddings import get_embeddings
@lru_cache(maxsize=1)
def get_vectorstore() -> Chroma:
return Chroma(
collection_name=settings.chroma_collection,
embedding_function=get_embeddings(),
persist_directory=settings.chroma_dir,
collection_metadata={"hnsw:space": "cosine"},
)
def _chunk_id(chunk: Document) -> str:
"""Derive a stable id from the chunk's content and origin."""
source = chunk.metadata.get("source", "")
page = chunk.metadata.get("page", "")
payload = f"{source}|{page}|{chunk.page_content}".encode("utf-8")
return hashlib.sha256(payload).hexdigest()
def index_chunks(chunks: list[Document]) -> list[str]:
"""Embed chunks and upsert them into the collection. Returns their ids."""
if not chunks:
return []
ids = [_chunk_id(c) for c in chunks]
get_vectorstore().add_documents(documents=chunks, ids=ids)
return ids
def search(query: str, k: int = 4) -> list[tuple[Document, float]]:
"""Return the k chunks nearest to the query, each with its distance."""
return get_vectorstore().similarity_search_with_score(query, k=k)
def count() -> int:
return get_vectorstore()._collection.count()
persist_directory is what makes this survive a restart. Leave it out and Chroma runs purely in
memory — you re-embed every document on every boot, which is slow locally and expensive on a hosted
provider.
Note you never call an "embed" function yourself. You hand Chroma an embedding_function at
construction, and add_documents embeds the text on the way in while similarity_search embeds the
query on the way out. The vector store owns that.
The Duplicate Trap
_chunk_id deserves an explanation, because the default behaviour here will hurt you.
Chroma assigns a random UUID to every document you add without an explicit id. Random ids are never equal, so re-indexing a document you already indexed doesn't replace anything — it appends a complete second copy.
Watch it happen:
chunks = load_and_chunk("sample.pdf")
print("count before:", count())
index_chunks(chunks); print("after re-index #1:", count())
index_chunks(chunks); print("after re-index #2:", count())
store = get_vectorstore()
store.add_documents(documents=chunks) # no ids= -> random uuids
print("after add without ids:", count())
store.add_documents(documents=chunks)
print("after add without ids again:", count())
count before: 3
after re-index #1: 3
after re-index #2: 3
--- now WITHOUT stable ids (the default behaviour) ---
after add without ids: 6
after add without ids again: 9
3 → 6 → 9. Every re-upload duplicates the entire document, silently.
Hashing source | page | content gives each chunk an id derived from what it is, so the same chunk
always lands on the same id and a repeat ingestion becomes an upsert instead of a duplicate. A
user re-uploading a file — which they will — costs you nothing.
Why it's not merely untidy: retrieval returns your top k chunks. If a passage exists three times, it can occupy all three of your top-3 slots, crowding out other relevant material. Your answers get worse and nothing anywhere raises an error.
That's the same shape as Part 2's overlap bug. The dangerous failures in RAG are the silent ones.
Choosing a Distance Metric
Run a search and look closely at the numbers:
dist=19.7961 page=2 PAGE TWO - FINANCIAL RESULTS
dist=22.2743 page=1 PAGE ONE - COMPANY HISTORY
Distances of 19 and 22? Cosine distance lives in 0–2. So Chroma isn't using cosine. Ask it:
print(get_vectorstore()._collection.configuration_json["hnsw"])
{'space': 'l2', 'ef_construction': 100, 'ef_search': 100, ...}
Chroma defaults to l2 — squared Euclidean distance. That's fine for normalized vectors, where
L2 and cosine rank identically. But remember our local model's magnitude of 5.62: these vectors
aren't normalized. L2 responds to magnitude and direction, and in text embeddings magnitude often
tracks incidental things like text length rather than meaning.
Set it explicitly:
collection_metadata={"hnsw:space": "cosine"}
Being honest about the evidence: on this 3-chunk corpus, cosine and L2 both got 5/5 queries
right. The metric made no measurable difference at this scale. Cosine is still the better default —
it's magnitude-robust as documents vary in length, it's the convention for text embeddings, and its
distances are interpretable (0.37 reads as "close"; 17.34 reads as nothing).
One caveat: hnsw:space is fixed when the collection is created. Changing it means deleting
.chroma/ and re-indexing.
Verifying Part 3
from app.ingestion import load_and_chunk
from app.vectorstore import index_chunks, count, search
chunks = load_and_chunk("sample.pdf")
index_chunks(chunks)
print("count:", count(), "chunks:", len(chunks))
for q in ["How much money did the company make?", "Who is the CEO?", "When was it founded?"]:
doc, dist = search(q, k=1)[0]
print(f"{q!r} -> page {doc.metadata['page']} dist={dist:.4f}")
count: 3 chunks: 3
'How much money did the company make?' -> page 2 dist=0.5308
'Who is the CEO?' -> page 3 dist=0.3741
'When was it founded?' -> page 1 dist=0.4712
Every question retrieves the right page, and the metadata survived the round trip through the vector store — which is what Part 4's citations will read.
Now the property that persist_directory buys. In a fresh process, with no indexing:
from app.vectorstore import count, search
print("count from disk:", count())
doc, dist = search("Who is the chief executive?", k=1)[0]
print(f"page={doc.metadata['page']} dist={dist:.4f}")
count from disk: 3
page=3 dist=0.3500
The vectors came back off disk. Chroma stores them in chroma.sqlite3 plus a directory holding the
HNSW index. Add it to .gitignore — it's binary, regenerable, and grows:
.chroma/
Update .env.example so the new knobs are discoverable:
RAG_EMBEDDING_PROVIDER="local"
RAG_OPENAI_API_KEY=""
RAG_CHROMA_DIR=".chroma"
RAG_CHROMA_COLLECTION="documents"
Switching to the hosted model, once your account has credit, is now one line in .env:
RAG_EMBEDDING_PROVIDER="openai"
Delete .chroma/ and re-index when you do. Embeddings from different models are not comparable —
they're different dimensions in a different space, and mixing them produces nonsense rather than an
error.
Part 4: Answers With Citations
Retrieval finds passages. Part 4 turns them into an answer that a user can check.
Choosing an LLM
project.txt asks for OpenAI. My account had no credit, so this series uses Ollama —
llama3.1:8b, running locally and free:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
pip install langchain-ollama
app/llm.py mirrors app/embeddings.py exactly — the same factory pattern, so the LLM is a config
choice too:
@lru_cache(maxsize=1)
def get_llm() -> BaseChatModel:
if settings.llm_provider == "ollama":
from langchain_ollama import ChatOllama
return ChatOllama(
model=settings.ollama_model,
base_url=settings.ollama_base_url,
temperature=0,
)
if settings.llm_provider == "openai":
from langchain_openai import ChatOpenAI
if not settings.openai_api_key:
raise ValueError("llm_provider='openai' requires RAG_OPENAI_API_KEY to be set")
return ChatOpenAI(model=settings.openai_chat_model, api_key=settings.openai_api_key, temperature=0)
raise ValueError(f"Unknown llm_provider: {settings.llm_provider!r}")
temperature=0 everywhere. RAG is an extraction task, not a creative one. You want the same
context to produce the same answer, and you want the model repeating the retrieved text rather than
improving on it. Temperature is the dial that decides whether it embellishes.
How citations actually work
Here is the trick, and it is simpler than people expect: number the passages and ask the model to cite the numbers.
def format_context(docs: list[Document]) -> str:
blocks = []
for i, d in enumerate(docs, start=1):
source = d.metadata.get("source", "unknown")
page = d.metadata.get("page", "?")
blocks.append(f"[{i}] (source: {source}, page: {page})\n{d.page_content}")
return "\n\n".join(blocks)
The model sees [1], [2], [3] and writes [2] after a claim. You map [2] back to a
filename and page — the model never has to reproduce "sample.pdf" correctly, because it only ever
handles a small integer. That is the whole design: give the model the easiest possible job and do
the bookkeeping yourself.
The prompt
SYSTEM_PROMPT = """You answer questions using only the numbered passages provided.
Rules:
1. Use only the passages. Never use outside knowledge, even if you are confident.
2. After every claim, cite the passage it came from in square brackets: [1], [2].
A claim drawn from two passages cites both: [1][3].
3. If the passages do not contain the answer, reply with exactly this and nothing
else: {no_answer}
4. Do not invent a citation. Only cite passage numbers that appear below.
5. Be concise. Do not repeat the question."""
Rule 3 is the one that matters. Without an explicit escape hatch, a model asked a question it cannot answer will answer anyway — that is what it was trained to do. Giving it an exact sentence to produce turns "I don't know" from a failure into a valid output.
Composing with LCEL
LCEL is LangChain Expression Language: the | operator composes runnables into a pipeline.
prompt | llm | StrOutputParser() means render the prompt, call the model, take the text.
But a naive retriever | prompt | llm throws away something we need:
return (
RunnableParallel(
question=RunnableLambda(lambda x: x["question"]),
docs=RunnableLambda(lambda x: _retrieve(x["question"], x["tenant_id"])),
)
.assign(context=lambda x: format_context(x["docs"]))
.assign(answer=RunnableLambda(timed_generate))
)
docs is carried all the way to the end. The answer text says [2], but only the retrieved
document list knows what [2] was. Pipe the docs straight into the prompt and discard them, and
you have an answer with markers pointing at nothing.
Trusting only real citations
def _used_citations(answer: str, docs: list[Document]) -> list[Citation]:
markers = {int(m) for m in re.findall(r"\[(\d+)\]", answer)}
citations = []
for marker in sorted(markers):
if 1 <= marker <= len(docs): # <- the important line
meta = docs[marker - 1].metadata
citations.append(Citation(marker=marker, source=meta["source"], page=meta["page"]))
return citations
The bounds check is not defensive padding. A model handed 4 passages will occasionally write [7].
Without the check you either crash on an index error or, worse, surface a citation to a document
that was never retrieved. Markers that don't exist are silently dropped.
Does it work?
Q: What was the revenue in Q3 2025?
A: 4.2 million euros [1]
citations: [('sample.pdf', 2)]
Q: Who is the CEO?
A: Dr. Amina Rashid [1].
citations: [('sample.pdf', 3)]
Q: What is the refund policy?
A: I don't know based on the provided documents.
citations: []
Right answers, right pages, and a clean refusal. Push harder — try to make it use outside knowledge:
Q: Who founded Microsoft?
A: I don't know based on the provided documents.
Q: What is the capital of France?
A: I don't know based on the provided documents.
llama3.1:8b knows both of these. It refused anyway. That's rule 1 working, and it is the
single most important behaviour in a RAG system: a system that answers from its training data while
appearing to answer from your documents is worse than no system.
Multi-source citation works too:
Q: Where is the company based and what was its Q3 margin?
A: The Zorblatt Corporation was founded in Helsinki, Finland [1].
Zorblatt reported an operating margin of nineteen percent in Q3 2025 [2].
citations: [('sample.pdf', 1), ('sample.pdf', 2)]
One caution on reading these: retrieval order defines the marker numbers. For a different query
the retrieval order was [1] -> page 3, [2] -> page 1, and the answer cited [2] for the founding
date — which looks wrong until you check the mapping and find it is exactly right. The numbers are
positions in the candidate list, not page numbers.
Part 5: Putting It Behind an API
Two endpoints wrap the pipeline. The interesting parts are not the routes.
@app.post("/documents", response_model=IngestResponse)
async def upload_document(file: UploadFile = File(...)):
filename = file.filename or ""
suffix = Path(filename).suffix.lower()
if suffix not in ALLOWED_SUFFIXES:
raise HTTPException(status_code=400, detail=f"Unsupported file type {suffix or '(none)'}")
# The loaders read from a path, not a stream, so the upload must land on disk.
with tempfile.NamedTemporaryFile(suffix=suffix, delete=True) as tmp:
shutil.copyfileobj(file.file, tmp)
tmp.flush()
chunks = load_and_chunk(tmp.name)
# load_and_chunk recorded the temp file's name. Overwrite it with the real
# one, or every citation points at /tmp/tmpab12cd.pdf.
for chunk in chunks:
chunk.metadata["source"] = filename
index_chunks(chunks)
return IngestResponse(source=filename, chunks_indexed=len(chunks))
Two things worth stealing:
The temp file. PyPDFLoader takes a path, and an upload is a stream. It has to touch disk. The
with block guarantees cleanup even if ingestion raises.
Overwriting source. load_and_chunk faithfully records /tmp/tmpab12cd.pdf — the temp name,
different on every upload. Left alone, every citation names a file that no longer exists. This is a
one-line fix and an unfixable bug if you don't notice it.
The bug that only appears under a server
The API worked in tests and then crashed the moment I ran it for real:
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
$ nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv
6141 MiB, 5171 MiB, 603 MiB
Ollama had the GPU. llama3.1:8b occupies ~5GB of a 6GB card, and sentence-transformers tried
to allocate CUDA on top of the remaining 603MB. It never showed up in CLI testing because Ollama
had unloaded the model between runs; under a live server the LLM stays resident.
The fix is what you'd want anyway:
embedding_device: str = "cpu"
The embedding model is a 384-dimension MiniLM — CPU is fast enough. The 8B LLM genuinely needs the GPU. If you run a local LLM and local embeddings on one machine, they compete for VRAM, and the small model should yield.
$ curl -X POST -F "file=@sample.pdf" localhost:8000/documents
{"source":"sample.pdf","chunks_indexed":3}
$ curl -X POST -H "Content-Type: application/json" \
-d '{"question":"What was the revenue in Q3 2025 and who is the CEO?"}' localhost:8000/query
{
"answer": "The revenue in Q3 2025 was 4.2 million euros [1]. The company's chief executive is Dr. Amina Rashid [2].",
"citations": [
{"marker": 1, "source": "sample.pdf", "page": 2},
{"marker": 2, "source": "sample.pdf", "page": 3}
]
}
Part 6: Multiple Formats and Arabic
One door for every format
Part 2 put PyPDFLoader behind load_and_chunk. That boundary now earns its keep — adding two
formats touches one function:
def _load_pages(path: Path) -> list[Document]:
suffix = path.suffix.lower()
if suffix == ".pdf":
return PyPDFLoader(str(path)).load()
if suffix == ".docx":
return Docx2txtLoader(str(path)).load()
if suffix in {".txt", ".md"}:
# autodetect_encoding matters for Arabic: a file written as UTF-16 or
# windows-1256 would otherwise raise, or decode to mojibake.
return TextLoader(str(path), autodetect_encoding=True).load()
raise ValueError(f"Unsupported file type {suffix or '(none)'}")
Only PDFs have pages. DOCX and TXT load as a single Document and report page 1. That is honest: there is no page to cite in a text file, and inventing one makes the citation a lie.
Unicode normalisation
def _normalise(text: str) -> str:
return unicodedata.normalize("NFC", text)
Arabic can encode the same visible character two ways — a base letter plus a combining mark, or one precomposed codepoint. They look identical and compare unequal. Without normalisation the same word hashes differently, splits across embeddings, and silently breaks the keyword matching Part 7 depends on. NFC picks one form consistently.
Choosing a multilingual model, by measurement
Part 3 flagged that paraphrase-multilingual-MiniLM was weak on Arabic. Rather than guess a
replacement, I benchmarked three on Arabic queries against Arabic documents, English against
English, and Arabic against English:
model dims AR->AR EN->EN AR->EN
paraphrase-multilingual-MiniLM-L12-v2 384 3/4 3/3 1/2
multilingual-e5-small 384 4/4 3/3 2/2
paraphrase-multilingual-mpnet-base-v2 768 4/4 3/3 2/2
multilingual-e5-small wins at the same size. Perfect scores at 384 dimensions — no cost over
the model it replaces, and it fixes cross-lingual retrieval too.
The e5 prefix trap
e5 models need asymmetric prefixes: questions are marked "query: ", documents "passage: ". That
asymmetry is why they are good at retrieval — the model is trained to tell a question from
content.
HuggingFaceEmbeddings has separate hooks for exactly this:
return HuggingFaceEmbeddings(
model_name=settings.local_embedding_model,
model_kwargs={"device": settings.embedding_device},
encode_kwargs={"prompt": settings.local_embedding_passage_prefix}, # documents
query_encode_kwargs={"prompt": settings.local_embedding_query_prefix}, # questions
)
Verify it actually applies rather than trusting the parameter name:
embed_query matches manual 'query: ' prefix? 1.0
embed_query matches NO prefix? 0.992942
embed_documents matches manual 'passage: '? 1.0
1.0 confirms it works. But look at the middle number: 0.9929 between prefixed and unprefixed.
Forget the prefix and nothing breaks — retrieval just gets quietly worse. Another silent failure,
the third in this series.
Changing the embedding model invalidates your index. Delete .chroma/ and re-ingest. Vectors
from two models are not comparable, and mixing them produces nonsense, not an error.
It works, in both languages
[EN] Q: What is the warranty period?
A: Thirty-six months from date of installation [1].
cites: [('sample.docx', 1)]
[EN] Q: What is the refund window?
A: Thirty days [1].
cites: [('sample.txt', 1)]
[AR] Q: ما هي إيرادات الشركة في الربع الثالث؟
A: بلغت إيرادات شركة زوربلات 4.2 مليون يورو في الربع الثالث من عام 2025 [1][2].
cites: [('sample_ar.docx', 1), ('sample.pdf', 2)]
The Arabic question got an Arabic answer citing both the Arabic document and the English PDF — e5's cross-lingual ability found the same fact in both.
One honest gap: Arabic PDFs
My Arabic PDF fixture extracted backwards:
original : الصفحة الأولى - تاريخ الشركة
extracted: ةكرشلا خيرات - ىلوألا ةحفصلا
reversed(extracted) == original? True
An exact character reversal. The cause is reportlab, my fixture generator, which has no bidirectional-text support and lays Arabic out left-to-right. It is a generator bug, not an app bug — Arabic PDFs from Word or InDesign carry a correct logical-order text layer. So the Arabic path is tested with DOCX and TXT, which store plain logical-order Unicode, and Arabic PDFs remain untested. Better to say that than to pretend.
Part 7: Hybrid Search
Part 3 showed semantic search matching "money did they make" to "revenue". Here is what it cannot do.
An embedding of "6672WN5504" carries no meaning — the tokeniser shreds it into fragments that mean
nothing. Semantic search is close to useless on part numbers, SKUs, error codes, and surnames.
BM25 — classic keyword ranking — matches those exactly, and is blind to synonyms. Each covers
the other's blind spot.
Fusing without comparable scores
BM25 returns unbounded term-frequency sums; Chroma returns a cosine distance in 0–2. You cannot average them. Normalising requires knowing each one's distribution, which changes per query.
Reciprocal Rank Fusion sidesteps it by throwing the scores away and using only rank:
def reciprocal_rank_fusion(rankings, k=60):
scores, docs = {}, {}
for ranking in rankings:
for rank, doc in enumerate(ranking, start=1):
key = _doc_key(doc)
docs.setdefault(key, doc)
scores[key] = scores.get(key, 0.0) + 1.0 / (k + rank)
ordered = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
return [(docs[key], score) for key, score in ordered]
Fifteen lines, no dependency. (LangChain's EnsembleRetriever lives in langchain_classic — the
legacy compatibility package. Building on a sunset package to escape another sunset package is a bad
trade.)
The k=60 damps the top ranks. Without it, rank 1 scores 1.0 and rank 2 scores 0.5, so one
retriever's confidence dominates. At k=60 the gap between ranks 1 and 2 is slight, and agreement
between retrievers becomes the signal.
The tokenizer bug
First measurement, on exact-SKU lookups:
semantic: 4/8 top-1 bm25: 1/5 hybrid: 4/8
BM25 scoring 1/5 on exact code lookup is not a weak result, it's a broken one. That's the one thing BM25 cannot fail at. So:
>>> from langchain_community.retrievers.bm25 import default_preprocessing_func
>>> inspect.getsource(default_preprocessing_func)
def default_preprocessing_func(text: str) -> List[str]:
return text.split()
>>> default_preprocessing_func("What is the warranty for the SR-33?")
['What', 'is', 'the', 'warranty', 'for', 'the', 'SR-33?'] # <- note the '?'
>>> default_preprocessing_func("Model SR-33 The SR-33 is a sonar receiver")
['Model', 'SR-33', 'The', 'SR-33', 'is', 'a', 'sonar', 'receiver']
'SR-33?' != 'SR-33'. LangChain's default BM25 tokenizer is literally text.split() — no
lowercasing, no punctuation stripping. A trailing question mark destroys the exact match. It does
not error; it just returns nothing useful.
_TOKEN_RE = re.compile(r"[^\W_]+(?:-[^\W_]+)*", re.UNICODE)
def tokenize(text: str) -> list[str]:
return _TOKEN_RE.findall(text.lower())
The internal-hyphen rule keeps SR-33 as one token instead of sr and 33. \w is Unicode-aware
in Python, so Arabic tokenises too.
BM25Retriever.from_documents(corpus, preprocess_func=tokenize)
That fix took BM25 from 1/5 to 8/8.
Where hybrid earns its keep
On meaningful codes like NX-40, semantic scored 8/8 — e5 is good enough that hybrid adds nothing.
The difference appears on high-entropy identifiers with no semantic content at all:
sku | semantic | bm25 | hybrid (rank of correct chunk)
8829XQ4471 | 2 | 2 | 1
6672WN5504 | None | 2 | 5 <- semantic missed it ENTIRELY
1096FT9938 | 4 | 2 | 2
-------------------------------------------
TOP-1 | 4/8 | 0/8 | 6/8
Semantic drops from 8/8 to 4/8 and misses one completely. Hybrid gets 6/8.
Be suspicious of any tutorial claiming hybrid always wins. On a 6-chunk corpus I measured no difference; only 20 near-identical products with arbitrary SKUs made the gap visible. The benefit is real and narrow: it appears exactly where exact tokens matter.
One architectural note: BM25 is rebuilt per query from the tenant's chunks, because unlike the vector store it has no on-disk index. Fine at thousands of chunks, wrong at millions — at that point BM25 belongs in Elasticsearch.
Part 8: Cross-Encoder Reranking
Wiring hybrid search in produced this:
Q: Which product has SKU 6672WN5504?
A: I don't know based on the provided documents.
Hybrid ranks that chunk 5th. retrieval_k=4 cuts it off. The chunk is found, then discarded
before the LLM sees it. A correct chunk at rank 5 is worth exactly as much as one never retrieved:
nothing.
Bi-encoder vs cross-encoder
Everything so far uses a bi-encoder: query and document are embedded separately, never meeting, compared by vector distance. That's what makes it fast — document vectors are precomputed — but the model never reads the pair together.
A cross-encoder takes (query, document) as one input and runs a transformer over both, so
attention relates question to passage directly. Far more accurate. Far too slow to run over a
corpus: nothing can be precomputed, so every candidate costs a forward pass.
They compose: retrieve widely with the cheap model, re-score the short list with the expensive one.
def rerank(query: str, docs: list[Document], top_n: int) -> list[tuple[Document, float]]:
if not docs:
return []
pairs = [(query, doc.page_content) for doc in docs]
scores = get_cross_encoder().predict(pairs)
ranked = sorted(zip(docs, scores), key=lambda ds: ds[1], reverse=True)
return [(doc, float(score)) for doc, score in ranked[:top_n]]
Model choice is not free here. The common default is cross-encoder/ms-marco-MiniLM-L-6-v2 —
English only. Using it would silently undo the Arabic support from Part 6. mmarco-mMiniLMv2 is MS
MARCO translated into 14 languages, Arabic included.
The measurement
sku | hybrid | reranked
6672WN5504 | 5 | 1 <- rescued
1096FT9938 | 2 | 1
-------------------------------------
TOP-1 | 6/8 | 8/8
rerank latency: mean 1545ms over 20 candidates
6/8 → 8/8, and the rank-5 chunk that was being discarded now sits at rank 1:
Q: Which product has SKU 6672WN5504?
A: The PB-01 has stock keeping unit (SKU) 6672WN5504 [1].
cites: [('catalog.txt', 1)]
The cost is real: ~1.5 seconds per query for 20 candidates on CPU. That's the trade — the GPU is holding Ollama, so the cross-encoder runs on CPU. Fewer candidates, a smaller model, or a spare GPU are your dials.
Part 9: Measuring It
Everything so far has been anecdote. Time for numbers.
Why not RAGAS
project.txt specifies RAGAS. It doesn't install:
ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'
ragas==0.4.3 (the latest) imports a module that langchain-community 0.4.x deleted as part of the
sunsetting we've been dodging since Part 2. Both ragas.evaluate and ragas.metrics fail at
import. Downgrading langchain-community would put the working loaders, BM25 and Chroma at risk —
trading a working runtime for an eval tool is a bad trade.
So eval/ implements the metrics directly. That turns out to teach more, because the point is
understanding what they measure.
Two kinds of metric
Deterministic metrics compare against facts recorded in the eval set. Free, instant, reproducible, and they cannot lie. Every one of them scores retrieval, which is where most RAG failures live.
def hit_at_k(rank, k):
"""A cliff: chunks below k never reach the LLM."""
return 1.0 if rank is not None and rank <= k else 0.0
def reciprocal_rank(rank):
"""A slope: rank 1 -> 1.0, rank 2 -> 0.5, rank 4 -> 0.25."""
return 1.0 / rank if rank else 0.0
def citation_accuracy(citations, expected_source, expected_page):
"""The metric this whole project rests on."""
if expected_source is None:
return None
if not citations:
return 0.0
return 1.0 if any(c["source"] == expected_source and c["page"] == expected_page
for c in citations) else 0.0
Citation accuracy is the one that matters most. An answer can be correct and cite the wrong page — which is worse than an obvious error, because the user checks, finds nothing, and stops trusting the system.
LLM-judged metrics ask a model to grade an answer. They measure what no string comparison can — but the judge is a fallible model.
The eval set
Fourteen hand-written cases in eval/dataset.py, including three unanswerable ones:
{
"question": "Who founded Microsoft?",
"ground_truth": "",
"unanswerable": True,
},
A RAG system that scores well on answerable questions and hallucinates on these is worse than useless: it is confidently wrong. Scoring refusal separately also prevents gaming — a system that refuses everything would otherwise look perfect on faithfulness.
Small and hand-written on purpose. An eval set is not a test fixture: every entry encodes a judgement about what the system should do, and that judgement has to come from someone who knows the documents. Twelve careful questions beat a thousand generated ones.
The results
config hit@k MRR cite_acc refusal secs
semantic only 0.91 0.86 0.91 1.00 38.4
hybrid 1.00 0.89 1.00 1.00 29.9
hybrid + rerank 1.00 0.91 1.00 1.00 75.9
Hybrid takes hit@k and citation accuracy from 0.91 to 1.00. Reranking adds MRR 0.89 → 0.91 — it
ranks the right chunk higher, and costs 2.5× the time to do it. Refusal is 1.00 throughout:
llama3.1:8b never hallucinated on the unanswerable questions.
When the judge is the bug
The faithfulness numbers were incoherent:
semantic only hit@k 0.91 cite_acc 0.91 faithful 0.80
hybrid + rerank hit@k 1.00 cite_acc 1.00 faithful 0.73 <- perfect retrieval, worse score?
My first assumption was judge noise. Wrong — grading identical answers three times gave 0/11
flipped verdicts. The judge is perfectly stable at temperature 0.
So I looked at what it actually said:
'UNSUPPORTED. The answer claims "Thirty days", but this refers to a refund period
for unused licenses, which is stated as thirty days of purchase in the SUPPORT
POLICY passage. However, it does not appear in the Model NX-40 or MARITIME
SENSORS DIVISION passages.'
The judge found the claim supported, then marked it UNSUPPORTED because it wasn't in all four passages. My prompt said "does every claim appear in the passages above?" and the model read that as "in every passage". Since retrieval always returns k=4 chunks and a fact lives in one, every answer failed by construction. The metric was measuring my prompt's ambiguity.
Question: is every factual claim in the answer supported by AT LEAST ONE of the
passages above?
Important:
- A claim only needs to appear in ONE passage. It does NOT need to appear in all
of them. The passages cover different topics and most will be irrelevant.
That fixed most of it (0.73 → 0.91), but one case still fails: "Thirty days [1]." — correct, and
literally in the context. An 8B model is unreliable at judging two-word answers; it wants prose to
reason over.
The honest conclusion: the deterministic metrics are trustworthy; the judged one has a known false-negative on terse answers. That is exactly why RAGAS defaults to strong judge models. Judge with the best model you can afford, and never the model being graded.
Part 10: Authentication and Multi-Tenancy
A bug here means one customer reads another's documents.
The tenant lives in the token
payload = {
"sub": user["username"],
"tenant_id": user["tenant_id"],
"exp": expire,
}
return jwt.encode(payload, settings.jwt_secret, algorithm=settings.jwt_algorithm)
The tenant id comes from the signed token, never from the request body. If a client could send its own tenant id, every isolation control below would be decoration — anyone could read anyone's documents by editing a JSON field.
payload = jwt.decode(token, settings.jwt_secret, algorithms=[settings.jwt_algorithm])
algorithms= is not optional. Omit it and a token can nominate its own algorithm — including
"none", which means "no signature", and PyJWT will honour it.
Making it a FastAPI dependency means a route opts in by asking for a Principal:
CurrentPrincipal = Annotated[Principal, Depends(get_current_principal)]
@app.post("/query", response_model=QueryResponse)
async def query(principal: CurrentPrincipal, request: QueryRequest):
result = ask(request.question, tenant_id=principal["tenant_id"])
Forgetting auth now requires actively deleting a parameter, rather than merely forgetting to add a line. Make the safe thing structural.
Isolation at the only door
def search(query: str, tenant_id: str, k: int = 4):
return get_vectorstore().similarity_search_with_score(
query, k=k, filter={"tenant_id": tenant_id}
)
tenant_id is a required argument, not an option — a route cannot forget it. And the filter runs
during search, not after: filtering post-retrieval would let a neighbouring tenant's chunks consume
the top-k slots and quietly starve the caller's own results.
BM25 needs the boundary applied differently. It has no filter — it's an in-memory index — so the tenant scope has to be applied when selecting what goes in:
raw = get_vectorstore()._collection.get(
where={"tenant_id": tenant_id}, include=["documents", "metadatas"]
)
Build BM25 over every tenant's chunks and filter afterwards, and you've leaked their text.
Two smaller details worth copying:
payload = f"{tenant_id}|{source}|{page}|{chunk.page_content}".encode("utf-8")
tenant_id is in the chunk id hash. Two tenants uploading an identical file must get separate rows, or one overwrites the other and deleting one tenant's document deletes the other's.
if user is None:
hash_password(password) # hash anyway before failing
return None
Timing. Returning immediately for an unknown user makes the "user exists" path measurably slower, and that difference is enough to enumerate valid usernames.
Proving it
POST /query without token -> HTTP 401
POST /documents without token -> HTTP 401
wrong password -> HTTP 401
alice: sub=alice tenant_id=acme
bob: sub=bob tenant_id=globex
alice uploads sample.pdf -> {"chunks_indexed": 3}
bob uploads sample.txt -> {"chunks_indexed": 1}
{"tenant_id":"acme","chunks":3}
{"tenant_id":"globex","chunks":1}
The test that matters:
ALICE asks about HER OWN doc -> "4.2 million euros [1]" cites: [('sample.pdf', 2)]
ALICE asks about BOB'S doc -> "I don't know based on the provided documents."
BOB asks about HIS OWN doc -> "Thirty days [1]." cites: [('sample.txt', 1)]
BOB asks about ALICE'S doc -> "I don't know based on the provided documents."
And the attacks:
Bob edits his token to claim tenant 'acme' -> HTTP 401 (signature check)
alg:none token -> HTTP 401 (algorithms= pin)
One more, because the likeliest failure is not an attacker:
DEV_JWT_SECRET = "dev-only-insecure-change-me"
def _warn_insecure_defaults() -> None:
secret = settings.jwt_secret
if not secret or not secret.strip():
warnings.warn("RAG_JWT_SECRET is empty. Tokens can be forged by anyone.")
elif secret == DEV_JWT_SECRET:
warnings.warn("RAG_JWT_SECRET is the built-in development default.")
elif len(secret) < 32:
warnings.warn(f"RAG_JWT_SECRET is {len(secret)} characters; HS256 wants at least 32.")
Three cases, not one. .env.example ships RAG_JWT_SECRET="", so copying it produces an empty
secret that signs tokens perfectly happily — and would have sailed past a default-only check in
silence. The likeliest mistake is the one the naive check misses.
Part 11: Background Jobs
Embedding a large PDF takes minutes. Holding an HTTP connection open for it invites client timeouts and blocks a worker for the duration.
@app.post("/documents", status_code=status.HTTP_202_ACCEPTED)
async def upload_document(principal: CurrentPrincipal, background_tasks: BackgroundTasks, file: UploadFile = File(...)):
...
with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
shutil.copyfileobj(file.file, tmp)
tmp_path = tmp.name
job = create_job(tenant_id=principal["tenant_id"], source=filename)
background_tasks.add_task(_ingest_job, job["job_id"], tmp_path, filename, principal["tenant_id"])
return IngestAcceptedResponse(job_id=job["job_id"], status=job["status"], source=filename)
202 Accepted, not 200 OK. The work has been queued, not done. The status code is part of the
contract.
delete=False now, and the background task owns cleanup. Part 5 used delete=True, which was
right when ingestion was synchronous. Keep it and the file is deleted the moment the handler
returns — before the job reads it.
Cheap validation still runs synchronously, so an obviously bad request gets a 400 immediately
rather than becoming a job you must poll to discover was rejected.
The rule for background work
def _ingest_job(job_id, tmp_path, filename, tenant_id):
try:
update_job(job_id, status=JobStatus.PROCESSING)
chunks = load_and_chunk(tmp_path)
if not chunks:
raise ValueError("No extractable text found. Scanned PDFs need OCR first.")
...
update_job(job_id, status=JobStatus.DONE, chunks_indexed=len(chunks))
except Exception as exc:
update_job(job_id, status=JobStatus.FAILED, error=f"{type(exc).__name__}: {exc}")
finally:
Path(tmp_path).unlink(missing_ok=True)
Nothing here may raise. This runs after the response was sent, so an exception vanishes into the
server log and leaves the job stuck in PROCESSING forever — the client polls a job that can
never finish. Catching Exception is usually a smell; here it is the requirement. Every failure must
land on the job.
_LOCK guards the job dict: BackgroundTasks runs in a threadpool, and read-modify-write of a job
is not atomic even under the GIL.
Verified
=== upload returns immediately (202 + job_id)? ===
{"job_id":"8311f225-...","status":"queued","source":"catalog.txt"}
HTTP:202
request returned in .010026632s
poll 1: processing None
...
poll 6: done 20
10 milliseconds for work that took ~11 seconds. And the failure paths:
corrupt PDF -> {"status":"failed","error":"PdfStreamError: Stream has ended unexpectedly"}
empty file -> {"status":"failed","error":"ValueError: No extractable text found."}
bob GET alice's job -> HTTP 404
That last one matters: job ids are UUIDs, but "unguessable" is not access control. Without the tenant check, one tenant could read another's job — which leaks their filenames.
Scope, stated plainly: _JOBS is an in-process dict. Correct for one worker, wrong for several —
with --workers 4, a job created in worker 1 is invisible to the worker serving the status poll, and
the client gets a 404 for a job running perfectly well. That's a deliberate trade to keep the job
model visible without a Redis dependency. The interface is what production needs; only the storage
changes.
Part 12: Caching
The key is the whole design
def cache_key(tenant_id: str, question: str) -> str:
payload = f"{tenant_id}|{normalise_question(question)}".encode("utf-8")
return hashlib.sha256(payload).hexdigest()
tenant_id in the key is the whole ballgame. Two tenants asking the identical question must get
different answers, because they have different documents. A question-only key serves one tenant's
private document contents to another — a data breach caused by a cache, and one your isolation tests
would never catch, because the leak only happens on the second request.
def normalise_question(question: str) -> str:
text = unicodedata.normalize("NFC", question)
text = re.sub(r"\s+", " ", text)
return text.strip().lower()
Deliberately conservative: exact-match caching after cleanup, not semantic caching. Treating "What is the revenue?" and "How much did they earn?" as one key needs an embedding comparison and a similarity threshold — and a threshold slightly too loose serves a confidently wrong answer to a question nobody asked.
The payoff
cache MISS: 12140 ms
cache HIT : 0.0 ms
same answer: True
--- normalisation: these must share one entry ---
0.0 ms 'what was the revenue in q3 2025?'
0.0 ms ' What Was The Revenue In Q3 2025? '
0.0 ms 'What was the revenue in Q3 2025?'
stats: {'entries': 1, 'hits': 4, 'misses': 1, 'hit_rate': 0.8}
All three variants share one entry — entries stays at 1.
The bug that reaches users
1. ask -> "I don't know based on the provided documents." (cached)
2. upload the answering document
3. ask again -> "I don't know based on the provided documents." <- THE BUG
4. invalidate, ask again -> "Thirty days [1]." cites: [('sample.txt', 1)]
Without invalidation, uploading a document appears to do nothing. A cached answer is a claim about a set of documents; once that set changes the claim may be false. The most visible form is a cached refusal for the very question the upload answers.
index_chunks(chunks, tenant_id=tenant_id)
dropped = cache.invalidate_tenant(tenant_id)
Whole-tenant invalidation is blunt — one upload discards every cached answer for that tenant, including valid ones. Working out precisely which answers a new document affects means knowing which would now retrieve it, which costs about as much as answering again. Blunt and correct beats clever and wrong.
Two more details: refusals are cached (they cost as much to produce as answers, and invalidation
handles the staleness), and the eval harness passes use_cache=False — measuring a configuration
change is meaningless if the second config replays the first one's cached answers.
Part 13: Observability
A bad RAG answer has several possible causes — retrieval missed the chunk, the reranker buried it, the prompt was malformed, the model ignored its context — and they are indistinguishable from the outside. All you see is a wrong answer.
Two layers, because they answer different questions. LangSmith answers "what did the model actually see?" — rendered prompt, retrieved passages, token counts. It needs an account, so it is opt-in:
os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = settings.langsmith_api_key
os.environ["LANGSMITH_PROJECT"] = settings.langsmith_project
That's the entire integration. Every chain, retriever and model call is traced without touching the chain code — the payoff for building on LCEL instead of hand-rolling the pipeline.
The local stage timer answers "where did the 12 seconds go?" and works offline, in CI, with no account:
_current_trace: ContextVar[dict | None] = ContextVar("current_trace", default=None)
A ContextVar, not a global: FastAPI serves requests concurrently, and a global would interleave two
requests' timings into one nonsense trace.
The instrumentation was wrong, and the trace showed it
{"stages": {"retrieve": 96.0, "rerank": 6663.3, "generate": 9718.1}, "total_ms": 9718.3}
generate equals the total. It wrapped build_chain().invoke(), which calls _retrieve
inside it — so "generate" was silently double-counting the other two stages. A stage that contains
the other stages is worse than no stage: it makes the LLM look like the bottleneck when it may not
be. The timer has to sit around the LLM call only:
def timed_generate(inputs: dict) -> str:
with observability.stage("generate"):
return generate.invoke(inputs)
{
"stages": {"retrieve": 8868.6, "rerank": 8017.8, "generate": 2901.3},
"cache": "miss", "candidates": 20, "retrieval_mode": "hybrid",
"total_ms": 19879.7
}
sum of stages: 19787.7 ms
total_ms : 19879.7 ms
unaccounted : 92.0 ms
Now the stages sum to the total — and the numbers overturn the obvious assumption. The LLM is
2901ms of 19879ms: the fastest stage. The time is in model loading (this is a cold start;
warm, retrieve is 96ms) and reranking. Everyone assumes the model is slow. Measure it.
A cache hit, for contrast:
{"stages": {}, "cache": "hit", "total_ms": 0.1}
Part 14: Docker and Shipping It
The image
COPY requirements.txt .
RUN pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements.txt
COPY --chown=appuser:appuser app/ ./app/
Requirements before source. Docker caches layers by content, so a source edit reuses the dependency layer instead of reinstalling torch — a 10-second rebuild instead of 5 minutes.
The CPU torch wheel. The default bundles ~2.5GB of CUDA runtime this image will never use: the container is CPU-only, and the GPU belongs to Ollama on the host.
RUN useradd --create-home --uid 1000 appuser
USER appuser
A container process running as root that escapes its namespace is root on the host.
The container found a bug my venv was hiding
ModuleNotFoundError: No module named 'jwt'
requirements.txt was stale. I'd installed pyjwt, bcrypt, python-multipart, rank_bm25,
langchain-ollama and docx2txt across Parts 4–12 and added none of them. My machine worked
perfectly. A clean checkout would not.
This is the single best argument for building the image. Docker starts from nothing, so it is the only environment that tells the truth about your dependencies. Your venv accumulates packages and will lie to you indefinitely.
The Ollama networking trap
$ ss -tlnp | grep 11434
LISTEN 127.0.0.1:11434
$ docker exec ragtest getent hosts host.docker.internal
172.17.0.1 host.docker.internal
Two problems stacked:
- Inside a container,
localhostis the container. Pointing atlocalhost:11434gives connection-refused that looks exactly like Ollama being down when it's running fine.host.docker.internalresolves to the host (Linux needs anextra_hostsentry for it to exist). - Ollama listens on
127.0.0.1by default — host-local only. So addressing the host correctly still fails, because nothing is listening on the docker0 address.
Fix the second with:
sudo systemctl edit ollama
# [Service]
# Environment="OLLAMA_HOST=0.0.0.0"
Understand it first: that exposes Ollama to your network, not just Docker.
Or on Linux, share the host's network stack and sidestep both:
docker run --network=host -e RAG_OLLAMA_BASE_URL=http://localhost:11434 \
-e RAG_JWT_SECRET=$(openssl rand -hex 32) rag-api
=== status: Up 30 seconds (healthy)
=== health === {"status":"ok"}
=== runs as user: appuser
=== query (container -> host Ollama) ===
{
"answer": "The warranty for the NX-40 is 36 months from the date of installation [1].",
"citations": [{"marker": 1, "source": "catalog.txt", "page": 1}]
}
Volumes are not optional
volumes:
- chroma-data:/app/.chroma # or re-embed every document on restart
- model-cache:/home/appuser/.cache # or re-download ~500MB every restart
What's Next
You have the system the introduction promised: upload a document in Arabic or English, ask a question, get an answer with a citation you can check — behind authentication, isolated per tenant, ingested in the background, cached, traced, and in a container.
config hit@k MRR cite_acc refusal
semantic only 0.91 0.86 0.91 1.00
hybrid 1.00 0.89 1.00 1.00
hybrid + rerank 1.00 0.91 1.00 1.00
The one lesson
Nearly every bug in these fourteen parts was silent. Not one raised an exception:
| What broke | What it looked like |
|---|---|
chunk_overlap=80 | Requested 80, got 0. Chunks just didn't overlap. |
| Missing Chroma ids | Corpus tripled on re-upload. 3 → 6 → 9. |
page 0-indexed | Citations off by one. Plausibly, checkably wrong. |
| Missing e5 prefix | 0.99 cosine to the correct vector. Just slightly worse. |
BM25 text.split() | 'SR-33?' != 'SR-33'. Exact match never fired. |
OpenAIEmbeddings() | Legacy ada-002 instead of the model you asked for. |
| Chroma default metric | L2 on unnormalised vectors. |
| Cached refusal | Uploading a document appeared to do nothing. |
| Judge prompt ambiguity | Every answer graded UNSUPPORTED, by construction. |
| Mislabeled timer | The LLM looked like the bottleneck. It was the fastest stage. |
Stale requirements.txt | Worked on my machine. Would fail on yours. |
Every one produced a system that ran, returned plausible output, and was quietly worse. This is what makes RAG different from ordinary backend work. A broken SQL query throws. A broken retrieval pipeline returns four chunks and a confident paragraph, and you cannot tell by looking.
There is only one defence, and it's the habit this whole series was really teaching: measure the thing you assumed. Count the overlap. Check the page against the physical page. Ask the library what it exports. Build the container. Every number in this tutorial exists because assuming would have been wrong.
Where to take it
Close the gaps this build left open. They're listed honestly in the README: in-process jobs and cache (Redis is the drop-in), BM25 rebuilt per query (Elasticsearch at scale), untested Arabic PDFs, a judge model too weak to grade its own work, hardcoded demo users, no rate limiting.
Switch to hosted models and re-measure. Set RAG_EMBEDDING_PROVIDER=openai and
RAG_LLM_PROVIDER=openai, delete .chroma/, re-index, and run python -m eval.run. The whole point
of the provider factories is that this is two lines. Does text-embedding-3-small beat e5 on your
documents? You now have a harness that can answer that instead of a vendor's benchmark.
Grow the eval set. Fourteen cases is enough to catch regressions and too few to trust a small delta. Add the questions your users actually ask — especially ones your system gets wrong. An eval set of real failures is worth more than a thousand generated questions.
Then break it on purpose. Feed it a scanned PDF with no text layer. A 500-page document. A table-heavy financial report where the numbers live in cells that flatten into nonsense. A question whose answer spans two documents. Each will fail in an interesting way, and each failure is the next thing worth fixing.