Building a RAG Pipeline from Scratch: Lessons Learned

When I set out to build DocuChat, I assumed the hard part would be the model. It wasn't. The hard part was everything around the model — the retrieval layer that decides what the model even gets to see.

Chunking is a design decision, not a detail

My first pass split documents into fixed 500-token chunks. It worked, technically — retrieval returned something for every query. But answers were subtly wrong, missing context that lived one paragraph above or below the retrieved chunk.

Switching to semantic chunking (splitting on section boundaries, with a small sliding-window overlap) fixed most of it. The lesson: chunk size is a tradeoff between precision and context, and the "right" answer depends entirely on your document structure, not a universal default.

Embeddings alone aren't enough

Pure vector similarity search retrieved plausible-looking but wrong chunks often enough that I added a re-ranking step. A cheap cross-encoder re-ranker on the top 20 candidates, cutting down to the top 4, improved answer accuracy more than any embedding model swap did.

Grounding and citations build trust

Users don't trust an answer they can't verify. Returning the source passages alongside every generated answer — and refusing to answer when retrieval confidence was low — did more for perceived quality than any prompt tweak.

What I'd do differently

• Start with evaluation, not generation. I built a golden question/answer set far too late.
• Log retrieval quality separately from generation quality — they fail independently, and conflating them makes debugging much harder.
• Treat the retriever as the product. The LLM is replaceable; the retrieval pipeline is where the domain expertise lives.

RAG systems look simple in a diagram and are genuinely difficult in practice. Most of the difficulty isn't the "generation" half — it's making sure the model never has to guess.