Connect SharePoint to RAG: architecture, Microsoft Graph & permissions

For many companies, SharePoint is already the de facto knowledge system. Policies, project documentation, templates, presentations, technical documents and team knowledge live there – but not necessarily in a form that makes the right answer easy to find.

RAG can turn that repository into a conversational knowledge interface. The real challenge is not the LLM. It is connecting SharePoint to retrieval without losing permissions, freshness and source traceability.

Architecture choice: retrieve remotely or build your own index?

There are two basic SharePoint-RAG patterns.

Option A: retrieve from SharePoint at query time

The retrieval layer queries SharePoint or a Microsoft-provided retrieval interface directly. Content is not permanently replicated into your own search index.

Advantages:

• stays closer to the existing permission model,
• fewer data copies,
• changes can become visible faster,
• less synchronization logic to operate yourself.

Trade-offs:

• stronger dependency on Microsoft interfaces and capabilities,
• less control over parsing, chunking and ranking,
• limitations for specialist formats or custom retrieval logic.

For current Azure AI Search scenarios, Microsoft documents a remote SharePoint knowledge source that uses the Copilot Retrieval API and preserves SharePoint authorization during retrieval. For Microsoft-centric architectures this can be a useful starting point. See the current feature notes in Microsoft Learn – SharePoint in Microsoft 365 Indexer.

Option B: synchronize SharePoint into your own retrieval index

The second pattern extracts selected content through Microsoft Graph or another suitable interface and processes it in your own pipeline:

SharePoint → Graph/API → parsing → chunking → metadata → ACLs → search/vector index

Advantages:

• full control over parsers and document types,
• custom chunking and metadata,
• free choice of search engine, vector store, embeddings and LLM,
• hybrid search and custom reranking,
• better fit for multi-source RAG across SharePoint + DMS + ERP + CRM.

Trade-offs:

• data is replicated,
• synchronization and delete propagation become your responsibility,
• permissions need to be materialized and kept current,
• monitoring and operations become part of your platform.

For technology-neutral or sovereign architectures, this pattern is often more flexible – when the additional operating cost is justified.

Microsoft Graph: start with least privilege

A common architecture mistake is an app registration with broad tenant-wide access.

A controlled connector should access only the SharePoint sites required for the use case. Microsoft supports scoped patterns including Sites.Selected.

The principle:

1. Register an application in Microsoft Entra ID.
2. Grant only the API permissions that are needed.
3. Explicitly scope access to the required sites.
4. Read content and metadata through Microsoft Graph.
5. Treat authorization information as first-class security data.

Microsoft also published a 2026 reference pattern that materializes SharePoint document permissions into a downstream search/RAG index and applies those permissions as query-time filters. See Microsoft ISE – Propagating SharePoint Document Permissions to AI Search and RAG Pipelines.

The critical requirement: SharePoint permissions must not disappear in RAG

An internal RAG system must not turn a permissioned SharePoint environment into a flat knowledge pool where everyone can retrieve everything.

Example:

• HR can access confidential HR material.
• Sales can access offers and customer documentation.
• Management has additional restricted documents.
• Project teams only see their approved spaces.

If all files are embedded without ACL context, that model is effectively lost.

Each indexed document or chunk therefore needs security metadata such as:

• allowed users,
• allowed groups,
• explicit deny principals where applicable,
• site/library context,
• tenant or organisational boundary.

During retrieval, the current user's permissions must be applied before content reaches the model context. The dedicated guide RAG permissions & security trimming goes deeper into this architecture.

Document identity: filenames are not enough

A SharePoint document can be renamed or moved without becoming a new business document.

Use stable source identifiers and keep at least:

| Field | Purpose | |---|---| | Site ID | source context | | Drive/Library ID | document library | | Item ID | stable document identity | | Web URL | source navigation | | Modified date | change detection | | Version | freshness and traceability | | Content hash | detect real content changes | | ACL/principals | security trimming | | Title/file type | presentation and filtering |

Without stable identity, re-indexing quickly creates duplicates and zombie versions.

Synchronization: polling, delta or events?

For a small pilot, a scheduled batch may be enough. Production systems should detect changes more deliberately.

Depending on the source and interface, options include:

• delta queries,
• modification timestamps,
• webhooks,
• queue/event processing,
• periodic reconciliation.

The goal is not maximum "real time". It is an explicit freshness requirement.

A policy change may need to appear within minutes. Archived project material may tolerate an hour of delay.

Delete propagation: removed documents must disappear

RAG discussions often focus on new and updated content.

Equally important:

What happens when a document is deleted or a user's permission is revoked?

The retrieval index must reflect both.

Otherwise you keep:

• outdated content,
• withdrawn documents,
• invalid policies,
• or information a user should no longer access.

Delete propagation and permission revocation belong in the connector's definition of done.

Parsing Word, PowerPoint, Excel and PDF

SharePoint is not a PDF folder. Typical repositories include:

• Word documents,
• PowerPoint,
• Excel,
• PDFs,
• text/Markdown,
• SharePoint pages,
• sometimes scans and images.

A generic parser can be acceptable for a pilot, but production quality requires testing by document class.

PowerPoint

Slides need order, titles and context. Independent text boxes without slide context make poor chunks.

Excel

Tables are rarely useful as flat prose. Depending on the use case, a structured query can be better than embeddings.

PDF

Multi-column layouts, scans, tables and headers/footers require careful extraction. Inspect the actual indexed text.

Word

Heading hierarchy, tables and lists should survive into semantic chunking.

Chunking: use SharePoint structure

Do not cut every document into identical text windows.

Useful signals include:

• headings,
• pages,
• sections,
• tables,
• document type,
• folder/library context,
• SharePoint metadata.

That improves both retrieval and source presentation.

For the broader system view across chunking, hybrid search and evaluation, see Build a RAG system the right way.

Citations: answers should lead back to SharePoint

A useful internal RAG answer should be verifiable.

Ideally the application exposes:

• document title,
• SharePoint link,
• supporting passage,
• modification date or version,
• page/section when reliable.

This allows users to confirm whether the AI interpreted the source correctly.

Replace SharePoint Search? Usually not

RAG and classic search solve different jobs.

SharePoint Search is useful when users know they need a specific file.

RAG is stronger when a question requires synthesis across documents or a concise answer with evidence.

A robust system can combine:

• keyword search for exact names, IDs and terms,
• semantic retrieval,
• metadata filters,
• reranking,
• LLM synthesis.

That is usually stronger than "vector search only".

When SharePoint-RAG is a good fit

Common use cases:

• internal policies,
• technical support,
• project knowledge,
• product and process documentation,
• quality management,
• knowledge access for business teams.

Poorer fits:

• highly transactional real-time values,
• data with unclear ownership or permissions,
• uncurated legacy archives,
• use cases without concrete user questions.

Practical architecture checklist

Before building, answer:

1. Which sites and libraries are in scope?
2. Which users and groups use the system?
3. Will retrieval stay remote or replicate content?
4. How will ACLs be preserved?
5. Which file types need support?
6. How are changes detected?
7. How do deletions and permission revocations propagate?
8. Which metadata survives?
9. How are citations presented?
10. Which real questions form the evaluation set?
11. What freshness is required?
12. Who operates connector, index and monitoring?

If several answers are missing, the vector database is not the next decision.

Implement SharePoint-RAG for production

SharePoint is often only the first source. DMS, Confluence, ERP, CRM, SQL and APIs follow later. The architecture should therefore be a reusable ingestion and retrieval layer rather than a one-off connector.

The broader guide Connect RAG data sources covers that multi-source pattern.

If you want to connect SharePoint or Microsoft 365 to your own RAG system, I support RAG implementation & AI integration across data flow, permissions, retrieval, evaluation and application integration.

You already have SharePoint content and a concrete knowledge use case?
→ Review the RAG architecture and integration path

Technical guidance on privacy and compliance does not constitute legal advice.