Build a RAG system: architecture, retrieval, evaluation & operations

A RAG system can look convincing in a demo and still pull the wrong policy, an outdated contract version or an irrelevant passage once real users depend on it. In that situation, switching to a larger model or tuning the prompt is often the wrong first move.

The fastest way to improve RAG quality is to make retrieval, context preparation and generation measurable as separate stages. Only then can you see whether a bad answer comes from search, chunking, permission filters, index freshness or the LLM itself.

This is therefore not another introductory RAG article. It focuses on how to investigate an existing system, reproduce common failures and test improvements against a stable evaluation set.

The RAG pipeline in 60 seconds

Retrieval-augmented generation extends a language model with information from your own data sources at runtime. In simplified form, the chain contains five stages:

1. Ingestion — documents, wikis, databases or other sources are read into the system.
2. Chunking and preparation — content is split into retrievable units and enriched with metadata.
3. Indexing — embeddings, keyword indexes or hybrid search structures are created.
4. Retrieval — relevant documents or chunks are found for the user query and may be reranked.
5. Generation — the LLM produces an answer from the supplied context.

If the final answer is wrong, any one of these stages may be responsible. That is why “the model hallucinated” is often too broad to be a useful diagnosis.

Building a RAG system: the components that matter

If you are starting from scratch, treat RAG as an information and systems architecture problem before treating it as a model problem. A production-capable implementation usually needs these layers:

1. Source systems and ownership — which systems are authoritative?
2. Ingestion and synchronization — how do new, changed and deleted objects propagate?
3. Parsing and normalization — can the system preserve document structure, tables and metadata?
4. Chunking — how is content split without destroying meaning?
5. Indexing — vector, keyword or hybrid search structures.
6. Identity and permissions — which user may retrieve which content?
7. Retrieval and reranking — how does the right evidence reach the top?
8. Context assembly — how much context reaches the model, and in what form?
9. Generation and citations — can users verify the answer against the source?
10. Evaluation and operations — how are quality, regressions, freshness, latency and cost monitored?

The first three layers are frequently underestimated. If you need to connect SharePoint, a DMS, ERP, CRM, databases, file shares or APIs, the dedicated guide Connect RAG data sources covers synchronization, metadata, permissions and deletion handling in depth.

Start with real questions, not technology

Before choosing a vector database or embedding model, collect representative questions from the people who will use the system. For each question, identify the expected source and what a useful answer should contain.

That gives you a small evaluation set before implementation starts. It also exposes whether the intended source actually contains the information users expect.

Decide where RAG ends and tools begin

Not every enterprise question should be answered from indexed text. Exact current values such as inventory, account balances, open orders or system status often belong behind controlled APIs or tools rather than a vector index.

A good architecture can combine both:

• retrieval for policies, manuals, project knowledge and documents,
• structured tools or APIs for current transactional data.

This boundary improves reliability and makes it clearer which system is responsible for each answer.

Why RAG systems produce poor answers despite strong models

A conventional software defect is usually attributable to a specific function. RAG combines several deterministic and probabilistic components: parsers, indexes, embeddings, filters, ranking, prompts and models.

One wrong answer can be caused by very different failures:

• The correct document exists in the index but is not retrieved.
• The document is found, but the relevant paragraph is in a different chunk.
• The right chunk ranks 12th and never reaches the LLM.
• A permission filter removes the exact source the user needs.
• An outdated document version is still present in the index.
• The context is correct, but the model draws the wrong conclusion.

Without separating these cases, optimisation becomes guesswork.

Failure class 1: retrieval finds the wrong documents

The first question after a poor RAG answer should not be “which model are we using?” but: which documents did the system actually retrieve?

For a concrete test question, inspect:

• Which documents or chunks are in top-k?
• Is the technically correct source present at all?
• At what rank does it appear?
• Which irrelevant results displace it?
• Are metadata, tenant and permission filters working as expected?

If the correct source never appears in retrieval, generation cannot reliably repair the problem. The useful levers are more likely query handling, embeddings, keyword search, hybrid search, filters or domain terminology than the LLM.

Product names, part numbers, abbreviations, legal terminology and internal vocabulary are common examples where purely semantic search can underperform a combination of semantic and lexical search. See Hybrid Search vs Vector Search for RAG for the technical comparison and decision matrix.

Failure class 2: the relevant chunk is not found

A document can be retrieved correctly while the answer is still wrong because the decisive information was split into an unsuitable chunk.

Typical problems include:

• A heading is separated from the paragraph it qualifies.
• Tables lose row or column context.
• An exception sits in the next chunk and is omitted.
• Lists are indexed without their introductory context.
• Very large chunks dilute the relevant passage.
• Very small chunks lose semantic relationships.

Chunking is therefore not a one-off infrastructure setting. It is part of the information architecture. Contracts, technical manuals, wiki pages and tables do not necessarily benefit from the same strategy. See RAG Chunking Strategies for Enterprise Data for concrete approaches and a document-type matrix.

Failure class 3: ranking or reranking is poor

Sometimes the relevant content is retrieved but ranked too low. That matters when only a small number of results are sent to the model.

Do not measure only whether a source was found. Measure how highly relevant passages rank. For difficult corpora, a second ranking stage can help evaluate the initial candidates more precisely against the query.

But reranking is not automatically better because it is more sophisticated. The real question is whether it puts more useful context into the context window on your evaluation set.

Failure class 4: parsing or chunking destroys context

Before embeddings and retrieval can work, source documents must be converted into usable text. PDFs are particularly prone to problems:

• multi-column layouts are read in the wrong order,
• headers and footers appear in the middle of content,
• tables become hard-to-interpret text sequences,
• scanned pages require OCR,
• footnotes and captions are attached to the wrong passage.

When investigating suspicious answers, inspect the actual indexed text, not only the original PDF. If ingestion already damages the content, a new embedding model merely improves search over broken data.

Failure class 5: permission filters change retrieval

In enterprise systems, retrieval must be relevant and authorised. This makes evaluation more demanding.

An administrator test may work perfectly while a normal user receives worse answers because department, role or tenant filters remove relevant sources. The opposite is more serious: a user might retrieve content they are not authorised to access.

A useful RAG evaluation therefore includes tests with realistic identities and roles. Check both:

• False denials: a source needed for the task is removed by access control.
• Unauthorised retrieval: restricted content reaches retrieval or model context.

This distinction is especially important for multi-tenant systems or confidential knowledge domains.

Failure class 6: the LLM ignores or misinterprets sources

Once retrieval provides the correct context, generation becomes the focus.

Typical failures include:

• A source is cited correctly, but it does not support the claim.
• Information from several documents is combined incorrectly.
• A limitation or exception is ignored.
• The model adds plausible knowledge that is not in the context.
• Conflicting sources are presented without uncertainty.

Clear answer rules and structured outputs can help, but the essential tool remains a test set for which the expected domain outcome is known. Depending on the use case, the system should also be able to say: “The available sources are insufficient for a reliable answer.”

Failure class 7: the index and source data drift apart

A RAG system can behave perfectly from a technical perspective and still be wrong if the index no longer reflects the current source data.

Check:

• How are changed documents detected?
• Are deleted documents reliably removed from the index?
• Can old and new versions exist at the same time?
• Can you trace when a source was last indexed?
• What happens when ingestion jobs fail?

For production knowledge systems, index freshness is part of reliability. Otherwise, the evaluation set only describes a static laboratory state.

Why “hallucination” is often the wrong diagnosis

“Hallucination” describes the visible symptom: an answer is not supported by the facts. It is not specific enough for technical remediation.

A better failure analysis asks:

1. Did the correct information exist in the source?
2. Was the source parsed and indexed correctly?
3. Was the relevant chunk retrieved?
4. Did it rank high enough to be passed to the model?
5. Was the user authorised to see it?
6. Did the model use the supplied context correctly?
7. Does the cited source actually support the answer?

Only this decomposition shows where the fix belongs.

Making RAG quality measurable

A production-oriented system needs a representative evaluation set. A handful of favourite demo questions from the development team is not enough. RAG Evaluation: Metrics, Golden Datasets & Regression Tests shows how to turn that set into a versioned baseline with retrieval metrics and regression tests.

Golden questions instead of demo prompts

Collect real or realistic questions from the eventual business process. The set should include more than happy paths:

• frequent standard questions,
• ambiguous wording,
• cases with similar documents,
• current and outdated versions,
• questions that should not be answerable,
• role and permission cases,
• domain-specific terminology and abbreviations.

For each case, you should at least know which source is relevant and what constitutes an acceptable answer. An exact reference answer can be useful in some processes, but criteria or expected key facts are often more robust.

Retrieval quality

Evaluate retrieval separately from the final answer. Useful questions include:

• Is at least one relevant source found?
• How highly does it rank?
• Does the supplied context contain the facts required for the answer?
• How much irrelevant context is sent along with it?

The right metric depends on the system and evaluation set. A single universal score matters less than a stable baseline that makes changes comparable.

Groundedness

Groundedness asks: is the answer supported by the provided context?

That is not the same as domain correctness. An answer can be fully grounded in a source that is itself outdated. Groundedness and source freshness therefore need separate checks.

Correctness

Correctness asks whether the answer is factually right and sufficiently complete for the specific question. This requires either a trustworthy reference answer or clearly defined assessment criteria.

Citation accuracy

A citation is useful only if it supports the concrete claim. Do not check merely whether a citation exists. Verify whether document, passage and statement actually match.

Task success

For B2B systems, the most important question is often: can the user complete the task correctly with this answer?

A support assistant can produce elegant prose and still fail if it recommends the wrong next action. Task success therefore connects technical evaluation with the business workflow.

Regression tests

Once a usable baseline exists, the evaluation set becomes a regression suite. Changes to

• prompts,
• chunking,
• embeddings,
• search methods,
• reranking,
• models,
• filters or
• data sources

should be tested against the same core cases. This reveals whether an improvement in one area introduces new failures elsewhere.

Latency and cost

Quality alone is not enough for production. A system can be accurate yet unusable if retrieval, reranking and generation are too slow or disproportionately expensive.

End-to-end latency and cost per relevant task therefore belong in the same baseline. Not as a generic benchmark, but for your workflow and expected load.

Compact troubleshooting matrix

| Symptom | Possible cause | What to check first | | --- | --- | --- | | Correct information exists, but answer is wrong | retrieval or generation failure | inspect top-k results and actual model context | | Wrong document is preferred | embeddings, keyword match, missing filters | compare ranking on representative questions | | Correct document, wrong passage | chunking or parsing | inspect indexed text and chunk boundaries | | Relevant result ranks too low | ranking/reranking | measure rank of relevant chunks across the eval set | | Answer cites a source that does not support the claim | generation/citation mapping | compare statement with the cited passage | | Admin gets good answers, users get poor ones | permission or tenant filters | compare retrieval using real roles | | Answers rely on outdated rules | stale index | inspect version, deletion and reindex logic | | New errors appear after model or prompt changes | regression | run the identical eval set before and after changes | | Quality looks good but adoption is poor | latency, cost or workflow fit | measure end-to-end task success, latency and cost |

Technical proof: RAG beyond a demo

The references overview documents, among other work, a self-hosted AI platform built for experdoo: RAG for internal knowledge, AI agents, and a multi-tenant architecture with clear data and permission separation. For systems like this, the important question is not whether a chat interface can produce plausible text, but whether data access, retrieval and system boundaries remain controllable under real operating conditions.

If sensitive data or hosting requirements determine the architecture, the next question is whether public cloud, EU hosting, private cloud, on-premise or a hybrid design is appropriate. I cover that separately under Private AI and on-premise AI for companies. For the broader privacy framework, see GDPR-compliant AI for companies.

When RAG troubleshooting becomes production readiness

A single retrieval fix is not a production release. Once the RAG system becomes business-critical, evaluation should cover the complete operating context:

• a representative evaluation set and explicit release criteria,
• retrieval, groundedness and domain correctness,
• permission boundaries,
• regressions after model, prompt or data changes,
• logging and failure analysis,
• latency and cost,
• and for agents, tool calls, approvals and relevant attack paths.

That is the role of AI system evaluation and production readiness. It does not start with “what should we build?” but with an existing PoC, RAG system, copilot or AI agent whose quality needs to be assessed before go-live.

Your RAG system already produces answers, but you cannot reliably explain or measure its quality? Start with retrieval and failure analysis: have the RAG system reviewed.