If you've seen a product demo where someone "chats with their documents" — asks a question in plain English and gets an answer grounded in their company's own files — you've seen Retrieval-Augmented Generation (RAG) in action. It's currently the most practical way to make a large language model useful for your data, and it's one of the most requested projects we build.
This post explains how it works without the hype, and — more importantly — where these projects actually succeed or fail.
The problem RAG solves
Large language models know a lot about the public internet, but they know nothing about your contracts, your support tickets, your internal wiki, or last quarter's incident reports. Two naive fixes both fall short:
- Paste everything into the prompt. Context windows are big now, but they're not "your entire company" big — and stuffing hundreds of pages into every request is slow, expensive, and degrades answer quality.
- Fine-tune the model on your data. Fine-tuning changes how a model behaves, not what it reliably knows. It's poor at memorizing facts, goes stale the moment your documents change, and is the wrong tool for knowledge lookup.
RAG takes a third path: look up the relevant material first, then let the model answer using it — like an open-book exam instead of asking the model to memorize the textbook.
How it works, step by step
A production RAG pipeline has two phases.
Indexing (done ahead of time):
- Ingest your documents — PDFs, wikis, tickets, transcripts.
- Chunk them into passages of a few hundred tokens each.
- Embed each chunk — convert it into a vector that captures its meaning.
- Store the vectors in a vector database.
Answering (at question time):
- Embed the user's question the same way.
- Retrieve the chunks whose vectors are closest to the question's.
- Optionally re-rank those candidates with a more precise model.
- Hand the question plus the retrieved chunks to the LLM with instructions like "answer using only this material, and cite it."
- Return the answer — ideally with links back to the source passages.
The model never needs to have memorized your data. It just needs to read well — which is exactly what LLMs are good at.
Where RAG projects actually fail
The demo takes a week. The production system takes longer, and almost always for the same reasons:
Retrieval quality, not the model. When a RAG system gives a bad answer, the cause is usually that the right passage was never retrieved. The fix lives in unglamorous places: better chunking (respecting document structure instead of cutting every 500 tokens), hybrid search that combines vectors with plain keyword matching, and re-ranking. Swapping in a fancier LLM rarely fixes retrieval.
Messy source data. Scanned PDFs, tables, screenshots of spreadsheets, six near-identical copies of the same policy with different dates — real corpora are ugly. Deciding which version is authoritative is a business problem, and no vector database solves it for you.
No evaluation. Teams ship a chatbot, collect vibes, and can't say whether last week's change made answers better or worse. A small test set of real questions with known-good answers — even fifty of them — turns tuning from guesswork into engineering.
Permissions. If your documents have access controls, your RAG system must respect them at retrieval time. Getting this wrong means an intern can ask the chatbot about executive compensation — and get an answer.
RAG or fine-tuning? A simple rule
- The model needs to know things (your docs, your data, current facts) → RAG.
- The model needs to behave differently (your tone, a strict output format, a specialized skill) → fine-tuning.
- Many strong systems use both: RAG for knowledge, a light fine-tune for style and format.
What a realistic first project looks like
The pattern we recommend: pick one high-value document set and one user group — support agents searching past tickets, engineers querying runbooks, sales searching contract clauses. Build the pipeline end to end: ingestion, retrieval, cited answers, and an evaluation set from day one. Ship it to a small group, measure, then expand.
Scoped like that, a first production RAG system is a matter of weeks, not months — and you'll know it works, because you're measuring it, not just demoing it.
Building something like this? Get in touch — we design, build, and operate RAG systems end to end.