How to Build a RAG System That Actually Knows Your Business

RAG stands for Retrieval-Augmented Generation. The pitch is simple: instead of asking an LLM to answer from its training data (which doesn't include your business), you give it relevant context from your own documents before it answers.

The reality is more nuanced. Most RAG implementations I've seen either hallucinate confidently, retrieve irrelevant chunks, or answer so generically they might as well be a basic ChatGPT wrapper. Getting it right takes more than throwing documents at a vector database.

This is the architecture I've settled on after building RAG systems for clients across SaaS, real estate, professional services, and e-commerce.

What RAG actually is (and isn't)

RAG is not fine-tuning. Fine-tuning bakes knowledge into the model's weights — it's expensive, slow to update, and not what you want for most business use cases. RAG retrieves relevant chunks from external storage at query time and injects them into the prompt. Your documents stay up-to-date because you control the index.

A good RAG system has three layers: ingestion (processing your documents into retrievable chunks), retrieval (finding the right chunks when a question is asked), and generation (using an LLM to synthesize those chunks into an answer).

Most failures happen in the first two layers — not the third.

The architecture I use

The details that actually matter

Chunking strategy

The most common mistake: chunking by token count. This splits sentences mid-thought and destroys semantic coherence. Use semantic chunking — break at paragraph and section boundaries, keeping related context together. For structured documents (FAQs, SOPs), keep question-answer pairs intact. Aim for chunks of 200-500 tokens with 20% overlap.

Metadata is underrated

Every chunk should carry metadata: source document, section heading, date, document type (policy, FAQ, SOP, etc.), author. This lets you filter retrieval — "only return chunks from documents updated in the last 6 months" — and give users proper citations. Most RAG systems skip this. It's the difference between "I found something relevant" and "here's the answer from page 3 of your refund policy, last updated March 2026."

Hybrid search, not just vectors

Pure vector search misses exact matches. If someone asks "what's the SLA for enterprise?", vector search might return semantically similar chunks that don't contain the actual SLA number. BM25 (keyword) search catches exact terms. The combination — hybrid search — covers both cases. Reciprocal rank fusion is the standard way to merge the two result sets.

Reranking

First-pass retrieval gets you 10-20 candidate chunks. A cross-encoder reranker (Cohere Rerank, or a local model) then scores each chunk against the query for true relevance — not just semantic similarity. This improves answer quality significantly for the cost of one extra API call.

Common pitfalls

Where to deploy it

For most clients, I start with Slack. It has zero onboarding friction — the team already lives there. A slash command or @mention triggers the RAG system, returns an answer with sources, and threads naturally into existing conversations.

Second most common: an internal web portal. Useful when you want a dedicated interface, search history, or multi-document comparison.

Third: customer-facing (embedded on your site or in your product). Higher complexity — you need tighter guardrails to prevent hallucination in public contexts.

What to index

Start with high-value, high-query-frequency documents. For most businesses, that's: internal SOPs and processes, product/service documentation, pricing and policy docs, past proposals and case studies, FAQ content.

Don't try to index everything on day one. Start with the 20 documents that would answer 80% of the questions your team actually asks. You can always expand the index later.