Connect SharePoint to RAG: architecture, Microsoft Graph & permissions

For many companies, SharePoint is already the de facto knowledge system. Policies, project documentation, templates, presentations, technical documents and team knowledge live there – but not necessarily in a form that makes the right answer easy to find.

RAG can turn that repository into a conversational knowledge interface. The real challenge is not the LLM. It is connecting SharePoint to retrieval without losing permissions, freshness and source traceability.

Architecture choice: does SharePoint remain the retrieval system, or do you take it over?

For SharePoint RAG, the API integration is not the most important decision. The real choice is whether retrieval stays on the source system or selected content is replicated into your own index. Both can be production-ready, but they move responsibility for ACLs, freshness, deletes, latency and search quality to different places.

Path A: remote retrieval

Here, SharePoint and an appropriate Microsoft interface remain in the query path. The source stays authoritative and content is not persistently copied into a separate index.

Remote retrieval: the source remains the retrieval system

1. 01

   User & Authorization

   The application sends the query and current user context; it does not need to decide from a replicated ACL model.

2. 02

   SharePoint / Graph Retrieval

   Source search and source ACLs constrain results close to SharePoint; the source remains authoritative.

3. 03

   Authorized Results

   Changes and deletes can become visible without maintaining an index; query and ranking behaviour depend on the API and search service.

4. 04

   Context Assembly & LLM

   The application assembles permitted evidence, while API latency, rate limits and availability remain in the query path.

Remote retrieval is often plausible when SharePoint search covers the relevant query types, ACL semantics should stay as close to the source as possible, data replication should be minimised, and freshness matters more than maximum ranking control. Query volume, latency and API limits still need to fit the use case.

For current Azure AI Search scenarios, Microsoft documents a remote SharePoint knowledge source that uses the Copilot Retrieval API and preserves SharePoint authorization during retrieval. For Microsoft-centric architectures this can be a useful starting point. See the current feature notes in Microsoft Learn – SharePoint in Microsoft 365 Indexer.

Path B: a replicated retrieval index

Here, a connector reads selected content through Microsoft Graph or another suitable interface and puts it into your own search, vector or hybrid index. You take ownership of the retrieval path.

Own index: retrieval becomes your responsibility

1. 01

   SharePoint & Change Detection

   The connector recognises relevant changes; the source remains authoritative and the index is a derived copy.

2. 02

   Identity & ACL Mapping

   Stable IDs and permissions are replicated; permission changes and revocations must reach the index promptly.

3. 03

   Parsing & Own Index

   Chunking, metadata, hybrid search and reranking are under your control – along with data copies and index quality.

4. 04

   ACL-aware Retrieval

   The application enforces security trimming, controls query latency and observes sync, stale state, deletions and retrieval.

An own index is often plausible when several sources need to be combined, hybrid search or reranking is required, proprietary ranking logic is needed, or search quality should be improved measurably through your own evaluation. Additional normalisation, chunking and metadata enrichment can also justify the effort. For the retrieval choice itself, see Hybrid Search vs. Vector Search.

Architecture view: do not copy everything automatically

I would not automatically copy every SharePoint repository into a vector store. If source search, permissions and latency are sufficient, remote retrieval can be the more robust architecture.

An own index is worthwhile when the additional retrieval control measurably justifies the extra lifecycle and security work. A hybrid approach is possible, but not an automatic default: it needs a clear split between queries and content that remain remote and those deliberately replicated.

Microsoft Graph and least privilege in the replicated path

A common architecture mistake is an app registration with broad tenant-wide access. A connector should access only the sites needed for the specific use case; Microsoft supports scoped patterns including Sites.Selected.

This is not a Graph tutorial, but an architectural boundary: content and authorization information are read only in the required scope and treated separately as security data. Microsoft also published a 2026 reference pattern that materializes SharePoint document permissions into a downstream search/RAG index and applies those permissions as query-time filters. See Microsoft ISE – Propagating SharePoint Document Permissions to AI Search and RAG Pipelines.

The critical requirement: SharePoint permissions must not disappear in RAG

An internal RAG system must not turn a permissioned SharePoint environment into a flat knowledge pool where everyone can retrieve everything. With remote retrieval, source ACLs remain closer to retrieval. Whoever replicates content into an own index also replicates responsibility for permissions.

Example:

• HR can access confidential HR material.
• Sales can access offers and customer documentation.
• Management has additional restricted documents.
• Project teams only see their approved spaces.

If all files are embedded without ACL context, that model is effectively lost.

An own index therefore needs security metadata on every document or chunk, such as:

• allowed users,
• allowed groups,
• explicit deny principals where applicable,
• site/library context,
• tenant or organisational boundary.

During retrieval, the current user's permissions must be applied before content reaches the model context. The dedicated guide RAG permissions & security trimming goes deeper into this architecture.

Document identity: filenames are not enough

A SharePoint document can be renamed or moved without becoming a new business document.

Use stable source identifiers and keep at least:

Without stable identity, re-indexing quickly creates duplicates and zombie versions.

Freshness and deletes: only an own index turns them into a lifecycle

Remote retrieval usually follows the source's current state. With an own index, create, update, move, rename, permission change, delete and restore must propagate in a controlled way to every affected index entry. A permission revocation matters just as much as a content change.

Define a business-level freshness requirement: an internal policy may need to update within minutes, while archived project material may tolerate an hour of delay. Sync, reconciliation, stale state and delete propagation are not side issues; they are operating responsibility. The broader ingestion and lifecycle pattern is covered in Connect RAG data sources, so that each connector does not need to reinvent it.

Parsing Word, PowerPoint, Excel and PDF

SharePoint is not a PDF folder. Typical repositories include:

• Word documents,
• PowerPoint,
• Excel,
• PDFs,
• text/Markdown,
• SharePoint pages,
• sometimes scans and images.

A generic parser can be acceptable for a pilot, but production quality requires testing by document class.

PowerPoint

Slides need order, titles and context. Independent text boxes without slide context make poor chunks.

Excel

Tables are rarely useful as flat prose. Depending on the use case, a structured query can be better than embeddings.

PDF

Multi-column layouts, scans, tables and headers/footers require careful extraction. Inspect the actual indexed text.

Word

Heading hierarchy, tables and lists should survive into semantic chunking.

Chunking: use SharePoint structure

Do not cut every document into identical text windows.

Useful signals include:

• headings,
• pages,
• sections,
• tables,
• document type,
• folder/library context,
• SharePoint metadata.

That improves both retrieval and source presentation.

For the broader system view across chunking, hybrid search and evaluation, see Build a RAG system the right way.

Citations: answers should lead back to SharePoint

A useful internal RAG answer should be verifiable.

Ideally the application exposes:

• document title,
• SharePoint link,
• supporting passage,
• modification date or version,
• page/section when reliable.

This allows users to confirm whether the AI interpreted the source correctly.

Replace SharePoint Search? Usually not

RAG and classic search solve different jobs.

SharePoint Search is useful when users know they need a specific file.

RAG is stronger when a question requires synthesis across documents or a concise answer with evidence.

A robust system can combine:

• keyword search for exact names, IDs and terms,
• semantic retrieval,
• metadata filters,
• reranking,
• LLM synthesis.

That is usually stronger than "vector search only".

When SharePoint-RAG is a good fit

Common use cases:

• internal policies,
• technical support,
• project knowledge,
• product and process documentation,
• quality management,
• knowledge access for business teams.

Poorer fits:

• highly transactional real-time values,
• data with unclear ownership or permissions,
• uncurated legacy archives,
• use cases without concrete user questions.

Practical architecture checklist

Before building, answer:

1. Which sites and libraries are in scope?
2. Which users and groups use the system?
3. Will retrieval stay remote or replicate content?
4. How will ACLs be preserved?
5. Which file types need support?
6. How are changes detected?
7. How do deletions and permission revocations propagate?
8. Which metadata survives?
9. How are citations presented?
10. Which real questions form the evaluation set?
11. What freshness is required?
12. Who operates connector, index and monitoring?

If several answers are missing, the vector database is not the next decision.

Implement SharePoint-RAG for production

SharePoint is often only the first source. DMS, Confluence, ERP, CRM, SQL and APIs follow later. The architecture should therefore be a reusable ingestion and retrieval layer rather than a one-off connector.

The broader guide Connect RAG data sources covers that multi-source pattern.

If you want to connect SharePoint or Microsoft 365 to your own RAG system, I support RAG implementation & AI integration across data flow, permissions, retrieval, evaluation and application integration.

You already have SharePoint content and a concrete knowledge use case?
→ Review the RAG architecture and integration path

Technical guidance on privacy and compliance does not constitute legal advice.