Enterprise RAG: Build, Buy or Hybrid?

The question “should we build or buy RAG?” is too coarse for enterprise RAG.

The platform is not the decision unit; the layer is. The real decision is what the organisation must own, deliberately build, configure, buy or keep replaceable.

A RAG system connects sources, parsing, permissions, retrieval, models, evaluation and an application. Those parts do not carry the same risk or create the same business value. A platform can accelerate delivery, but it should not quietly take ownership of data logic, access rules or quality criteria.

For the technical foundation, see Build a RAG system the right way.

The architecture view

Business logic, the permission model and evaluation should belong to the organisation. They determine which answers are useful and permissible in a real business process.

Commodity infrastructure does not need to be built in-house. Search, observability, standard connectors and model access can be configured, bought or managed. The architecture must make them genuinely replaceable through portable data, explicit contracts and independently owned quality checks.

Ownership in an enterprise RAG system

• Application & integration boundary

  Keep UI, use case, approval rules, APIs and differentiating business logic independent of the platform

  Organisation-owned

• Data lifecycle & derived processing

  Own source IDs, updates, deletions and archives; keep parsing, chunking and indexing reproducible and replaceable

  Own the meaning, keep the pipeline replaceable

• Permission model

  ACLs, roles, tenants and security trimming from authoritative systems

  Organisation-owned

• Evaluation

  Golden dataset, permission cases, no-answer cases and release gates

  Build deliberately

• Connectors

  Configure a standard connector; build critical specialist integration deliberately

  Configure or build

• Retrieval & search

  Operate index, filters, hybrid search and reranking for the use case

  Managed or OSS

• Operations & observability

  Deliberately buy, share or operate monitoring, updates, backup, recovery, scaling and runbooks

  Assign shared operational responsibility

• Foundation model

  Keep model access behind a clear boundary and comparable

  Replaceable

Why the binary question misleads

“Buy” may mean a packaged product, a managed service or just a connector. “Build” may mean owning an orchestration layer without writing a vector database or model. The labels become unhelpful when the architecture is not decomposed into layers.

For each layer, ask three questions:

1. Is it specific to our data, workflow or permissions?
2. What does losing control of it cost?
3. Can we realistically replace or operate it in two years?

If the first two answers are high, its responsibility should not disappear into opaque platform configuration.

A concrete enterprise scenario

A support team needs to search approved technical documentation from SharePoint and a DMS in an existing service application, with citations. Each answer may use only documents the individual employee is allowed to see.

The hard part is not selecting a chat interface. It is stable document identity, delta synchronisation, deletion handling, groups and ACLs, source citations and test questions drawn from real support cases. Those rules and the golden dataset should remain with the organisation.

A standard connector may be right for SharePoint or the DMS. Before buying, verify that it handles permissions, versions and deletions correctly. If it does not, a focused integration adapter is often smaller and safer than a later platform migration. See Connect RAG data sources and RAG permissions & security trimming.

Search can be managed or self-operated open source. The model may come from a provider or run privately. Both are valid while the application does not lock ACL logic or evaluation data into a proprietary format.

Where the workload runs and who operates infrastructure or model runtime is the upstream decision covered by Private AI vs public cloud. This article allocates ownership within the RAG system itself.

The decision logic

Do not begin with a vendor shortlist. First describe a real task, its sources, permissions, freshness requirements and the cost of a wrong answer. Then assign ownership by layer and test two or three options against the same cases.

A platform does not win because its feature list is longer. It wins when it reliably serves the defined process with the right sources, permissions, costs and operating boundaries.

A component is only replaceable when its data, configuration and quality evidence survive the vendor. At minimum, preserve source data, document IDs and metadata, ACL mapping, the golden dataset, evaluation results, API contracts and integration configuration.

Document layer decisions explicitly

A short decision memo for each relevant capability is enough. It does not replace judgement with a scorecard, but it makes ownership and exit testable. Hybrid is only strategically sound when ownership and exit are not left implicit.

Once the decision is made, the concrete architecture still needs a pre-production review across data, identity, security, evaluation, failure modes, cost and operations. Use the Enterprise AI Architecture Checklist for that review.

When build is justified

Build is justified when the RAG layer itself differentiates the product or process: unusual data models, complex permissions, critical specialist integrations or proprietary workflow logic are typical examples.

It does not mean reinventing infrastructure. A lean custom architecture may own an API and orchestration boundary, reliable permission mapping and evaluation while using purchased search and model infrastructure.

When buy is justified

Buy is justified when time-to-value matters, standard sources dominate and the team should not carry operations, updates or commodity capabilities itself.

The checks still apply: does the solution enforce permissions correctly? Are sources, exports and interfaces transparent? Can model access change? A fast demo is not evidence of production readiness.

Why hybrid is often the better default

Hybrid separates business control from commodity infrastructure. The organisation owns the process, permission rules and definition of quality, then combines them with components that are sensibly bought, managed or self-operated.

This does not remove every dependency. It makes dependencies explicit and confines them to layers where replacement is commercially reasonable.

Concrete recommendation

For most enterprise RAG initiatives, create a small but firm ownership boundary: the existing application and its business logic stay authoritative. Permissions are derived from source systems and enforced before retrieval. Evaluation runs independently of the model and search provider. Connectors, search, observability and model access are then selected by operational and cost requirements.

That replaces an artificial “build or buy” choice with an architecture that responds to real data, risk and switching costs. For independent quality checks, see RAG Evaluation: Metrics, Golden Datasets & Regression Tests.

If you are making a RAG platform decision, start with AI consulting & solution architecture. If the architecture is decided and data sources, retrieval and integrations need implementation, continue with RAG implementation & AI integration.