Connect RAG data sources: SharePoint, DMS, ERP, CRM & databases

A RAG system is only as reliable as the path between the source system and the answer. In demos, a few PDFs are uploaded, embeddings are generated and plausible answers appear. Enterprise environments are different: knowledge lives in SharePoint, document management systems, ERP and CRM platforms, databases, file shares, wikis, ticketing tools and line-of-business applications – each with different owners, permissions and update cycles.

That is where a RAG prototype becomes either a dependable system or a long-running data-quality problem.

This guide explains how I would connect RAG data sources when the system is expected to become production-ready, traceable and operable. The focus is not a particular framework. It is the architecture between source, ingestion, retrieval and answer.

The short answer: what makes a data source RAG-ready?

For almost every source, seven questions need answers:

1. What is the canonical source? Which system owns the original?
2. How do changes reach RAG? Batch, polling, events, webhooks or APIs?
3. How is a document identified consistently? So updates do not create duplicates.
4. Which permissions apply? Users, groups, tenants and confidential areas.
5. Which metadata must survive? Source, type, date, owner, status and business context.
6. How are deletions handled? Removed information must disappear from retrieval.
7. How is answer quality evaluated? Using real questions, not only successful indexing.

If one of these remains unresolved, the connector is not finished – even if vectors are already stored.

RAG is a data pipeline, not a PDF upload

A production retrieval-augmented generation system can be simplified into two paths.

Ingestion path:
Source → extraction → parsing → normalization → chunking → metadata → embeddings/index → permissions

Query path:
User → identity/permissions → query → retrieval → filtering → reranking → context → LLM → answer + citations

The vector database is only one component. Many failures happen earlier: change detection, parsing, document identity or lost access rights.

For the full system view, see Setting up RAG the right way: architecture, retrieval and production readiness.

Connecting SharePoint to RAG

SharePoint is a common enterprise source because documents, team knowledge and metadata come together there. It is also more complex than a folder of PDFs.

A robust integration should consider at least:

• sites, libraries and folder structures
• stable document IDs instead of filenames as identity
• versions and modification timestamps
• user and group permissions
• deleted or moved documents
• Office files, PDFs and potentially page content
• business metadata such as department, document type or status

The central architectural issue is permission-aware retrieval. An employee in department A must not receive information from department B just because both documents happen to live in the same vector index.

Permission information can be stored as filterable metadata during ingestion. Retrieval then restricts candidates based on the current user context.

Permissions also change. A one-time ACL import is therefore not enough. Permission synchronization is part of operations.

For the Microsoft 365 architecture, see Connect SharePoint to RAG. For ACLs, groups, revocation and query-time filtering, see RAG permissions & security trimming.

DMS: preserve document lifecycle and version context

In a document management system, the business document is often more important than the file itself.

A DMS may track:

• draft, approved or archived status
• document class and case assignment
• validity periods
• versions
• retention rules
• roles and groups
• relationships between documents

Do not discard this information during export. It is valuable retrieval metadata.

Example: when a user asks for a policy, a three-year-old version should not win merely because one chunk has slightly higher semantic similarity. Validity and version need to influence retrieval.

ERP: not every value belongs in a vector index

ERP platforms contain highly structured and often fast-changing information: orders, inventory, prices, bookings, master data or production data.

The reflex to "index the ERP into RAG" is therefore often wrong.

Retrieval can be useful for documentation, manuals, process descriptions or relatively stable knowledge. For a question such as "What is the current inventory of item X?", a controlled live API or tool call is usually more appropriate.

A robust architecture separates:

• knowledge retrieval for unstructured or semi-structured content
• structured queries/tools for exact current system data

The LLM can orchestrate both. The important part is that exact numbers are not inferred from potentially stale text chunks.

CRM: useful context, with deliberate data minimization

CRM systems combine structured fields with free text: companies, contacts, activities, notes, opportunities and communication history.

Potential RAG sources include:

• approved product and customer information
• documented meeting summaries
• internal account notes with clear role permissions
• proposal and project documentation

At the same time, privacy, purpose limitation and role models become more important. Not everything that can technically be retrieved should automatically be copied into a retrieval index.

The architectural question is therefore not "What can we index?" but "What information does this use case actually need?"

SQL databases: retrieval or query?

Relational databases present the same fundamental choice as ERP systems.

When users need exact current records, a controlled query is often better than semantic retrieval. When databases contain large text fields, knowledge articles or historical document information, RAG can be appropriate.

Many production systems therefore expose two tools:

• retrieval through a document/vector index
• structured data access through defined tools, views or APIs

This lowers hallucination risk and keeps system responsibilities explicit.

File shares and network drives

File shares look simple but often contain:

• unclear ownership
• duplicates
• deeply nested folders
• obsolete versions
• inconsistent naming
• historically grown ACLs
• temporary files
• many file formats

Before generating embeddings, perform an inventory. Which folders are authoritative? Which are archives? Where are duplicates of the same information?

A RAG system should not merely make existing information chaos searchable faster.

Confluence, wikis and knowledge bases

Wikis are attractive for RAG because content is already textual and linked. Still, page title, space, hierarchy, labels, author, modification date and permissions should remain available.

Freshness matters. If old and new process descriptions exist side by side, retrieval needs enough signals to prefer the valid context.

Internal links can also provide structural information. They do not replace semantic search, but they help identify related content.

APIs and line-of-business systems

Not every system needs a packaged "RAG connector". If a business application exposes a stable API, a small custom ingestion service can be easier to control than a heavyweight universal connector.

Define:

• which objects are read
• which fields enter the index
• how pagination works
• how rate limits are handled
• how changes are detected
• how errors and partial imports become visible
• how deletions propagate

A connector without monitoring will eventually become a silent data-quality problem.

Parsing: the file is not the content

PDF, Word, PowerPoint, HTML and scanned documents need to be translated into useful text and structure first.

Important information can easily be lost:

• tables
• heading hierarchy
• page references
• lists
• footnotes
• captions
• multi-column layouts
• OCR content

Retrieval quality therefore depends not only on whether text was extracted but on whether business structure survives.

A simple test: can a human still tell which number belongs to which table row in the parsed output? If not, the RAG system will struggle too.

Chunking: avoid fixed character counts as a default

Chunks should represent meaningful units. A blanket "split every 800 characters" rule can break paragraphs, tables or process steps.

Depending on document type, better options include:

• section and heading boundaries
• tables as dedicated units
• document-type-specific rules
• moderate overlap where boundaries otherwise lose context
• parent-child approaches with small retrieval chunks and larger answer context

The right chunking strategy cannot be chosen in isolation. Evaluate it against real questions.

Metadata is a retrieval feature

Metadata is often treated as plumbing. In enterprise RAG, it is part of search quality.

Typical fields include:

| Metadata | Purpose | |---|---| | Source system | provenance and debugging | | Document ID | updates and deduplication | | Title | result presentation | | URL/path | citation link | | Owner/department | business context | | Document type | filtering | | Created/modified | freshness | | Status/validity | correct version | | ACL/group | permission filtering | | Tenant | hard data separation |

Without a stable document ID, synchronization becomes difficult. Without a source URL, answers are harder to verify. Without ACL metadata, retrieval can become a security risk.

Synchronization: full re-indexing is not an operating model

For a pilot with 200 documents, a full re-import may be acceptable. Production data sources should usually be processed incrementally.

The pipeline needs to distinguish:

• new object
• changed object
• unchanged object
• moved object
• deleted object
• changed permission

Ideally, every ingestion run is observable: what was processed, what failed and which source has not synchronized for hours or days?

Deletions matter as much as new documents

A commonly missed requirement: if a document is deleted or access is revoked in the source, it must disappear from retrieval as well.

Otherwise you create a zombie index: search still works technically, but it contains information that should no longer exist or be visible.

Delete propagation and permission revocation belong in acceptance criteria.

Permissions: security trimming belongs in retrieval

With internal company data, checking access only after generation is dangerous.

Retrieval should only return candidates the user is allowed to see. Depending on the architecture and source, this can use user, group, role or tenant filters.

Also clarify:

• Where does user identity come from?
• How are groups resolved?
• What happens when permissions change?
• Are service accounts overprivileged?
• Are permissions cached?
• How is tenant separation enforced?

This is architecture, not prompt engineering.

Hybrid search and reranking

Semantic vector search is useful but not sufficient for every query.

Production retrieval pipelines often combine:

• semantic similarity
• keyword/full-text search
• metadata filters
• freshness
• business priority
• reranking

Evaluation determines which combination helps. More retrieval stages do not automatically mean better quality.

Citations: answers should be verifiable

For internal knowledge, an answer without a traceable source is significantly less useful.

Store enough provenance during ingestion to link back to the original:

• document title
• source system
• URL or object ID
• section or page where technically reliable
• version/date

This increases trust and makes debugging much faster.

Evaluation starts before the first connector

Before building a connector, collect real questions.

Examples:

• "Which current travel expense policy applies in Austria?"
• "Which approvals are required for an offer above amount X?"
• "What are the technical prerequisites for product Y?"
• "Who may approve process Z?"

Use these to create a small golden set with expected sources and answers.

Then evaluate:

• Was the correct source retrieved?
• Was it in the top-K?
• Was the version current?
• Were permissions respected?
• Was the answer supported by the sources?
• Did the system abstain when evidence was missing?

That turns "it feels good" into an engineering quality process.

A practical target architecture

A common enterprise pattern looks like this:

Source systems
SharePoint · DMS · ERP · CRM · SQL · wiki · file shares · APIs

↓

Ingestion layer
Connectors · change detection · parsing · normalization · ACLs · metadata

↓

Knowledge layer
Document store · search index · vector store · metadata · versions

↓

Retrieval layer
Identity · permission filters · hybrid search · reranking · context assembly

↓

AI / application layer
LLM · tools/APIs · citations · business workflow · evaluation · monitoring

The specific technologies can remain replaceable. The important part is the boundaries and data contracts between these layers.

Build vs buy for RAG connectors

Connector platforms can accelerate delivery. Custom integrations can make sense when permissions, data models, specialist formats or operational constraints are unusual.

I would compare at least:

• supported sources
• permission model
• change detection
• delete propagation
• data residency
• error handling
• observability
• export/exit options
• cost at growing data volume

The connector should remain a replaceable component – not become the architecture.

What to clarify before implementation

Do not start with a tool list. Start with a worksheet:

1. Use case and users
2. real questions
3. source systems
4. data classes
5. owners
6. permissions
7. freshness requirement
8. document volume and change rate
9. citation requirements
10. acceptance criteria

Only then does it make sense to choose connectors, parsers, a vector database or framework.

Connecting RAG data sources in production

Once the source systems are known, the task is not "install a connector". It is building a controlled data and retrieval path.

I help companies with RAG implementation and AI integration: from SharePoint, DMS, ERP, CRM and databases through permissions and retrieval to evaluation and integration into existing applications.

If the target architecture is still open, AI architecture consulting is the better starting point. For sensitive data and hosting constraints, private and sovereign AI compares the operating models.

You already know your data sources and want to turn them into a dependable RAG pilot?
→ Discuss RAG implementation

Technical guidance on privacy and compliance does not constitute legal advice.