Deleting Knowledge from RAG: How to Verify Source, Index, Cache and Answers
A document is functionally deleted from RAG only when neither the document nor its derivatives can influence any permitted response path. Removing a source file or receiving a successful delete response does not establish that. Verification must cover search indexes, caches, conversations, pending ingestion and restored system states.
There are two milestones: functional exclusion from the running application and physical removal from storage. They require different evidence.
Why deleting the source document is insufficient
Ingestion turns a source into temporary copies, parsed and OCR output, chunks, embeddings and search records. Retrieval may cache candidates and reranked results, while the application may retain full answers, conversation summaries or agent memory.
A chunk is a segment derived from a document; an embedding is a numerical representation used for similarity search. Both can survive deletion of the original file.
Your inventory should therefore include source objects, temporary and normalised files, OCR/parser results, chunk manifests, dense vector records, sparse or keyword indexes, hybrid search, reranker and retrieval caches, response caches, conversation histories, persistent memory, ingestion queues, snapshots, backups and external providers.
The OWASP RAG Security Cheat Sheet addresses deletion propagation to derived indexes and caches. For source ingestion and lifecycle architecture, see Connecting RAG Data Sources. The focus here is narrower: proving deletion.
Functional exclusion is not physical erasure
Functional exclusion means that no permitted new retrieval or response can rely on the deleted content. This includes cached answers and existing conversations; removing vector records alone is insufficient.
Physical removal means accounting for remaining copies across index storage, objects, temporary processing, backups and external systems, with evidence of removal or documented cleanup deadlines.
Storage implementations differ. Azure AI Search describes background physical cleanup after deletion. Weaviate documents asynchronous HNSW cleanup. These product-specific behaviours do not establish complete erasure across an application.
Recommendation: Sign off functional exclusion only after all active response paths pass verification. Track residual physical storage as a separate milestone.
Use stable identity and artefact lineage
An auditable deletion needs traceability from the original source to each generated artefact. Record at least:
| Field | Purpose |
|---|---|
tenant_id or workspace_id | Tenant isolation |
source_id | Upstream source |
document_id | Stable document identity |
version_id | Specific document revision |
chunk_ids | Derived retrieval units |
index_generation | Affected search releases |
cache_dependencies | Retrieval and response cache references |
delete_job_id | Correlated operation and evidence |
The identifiers must remain unambiguous across tenants and reindexing. New parsing or embedding pipelines may generate new chunk IDs. A deletion limited to an old list of IDs can miss regenerated material.
Block re-ingestion before cleanup
A pending worker, retry job, batch rebuild, migration or snapshot restore may recreate content after deletion.
A tombstone or authoritative delete registry records the blocked document and version. Ingestion and retrieval must consult it.
A safe sequence is to register the deletion and establish the block first, clean derived stores and caches second, reject stale workers third, and reconcile restored or rebuilt indexes with the latest deletion journal before exposing them to users.
The authoritative deletion state must not disappear when an older snapshot is restored. Tombstone expiry needs its own rules: if stale imports remain possible, removing the tombstone may permit re-entry.
Deletion verification matrix
| Layer | Deletion action | Evidence | Expected result |
|---|---|---|---|
| Source | Remove or revoke source | Source API lookup | No active source |
| Temporary, OCR and parser outputs | Remove derived copies | File/object inventory | No active reusable copy |
| Chunk manifest | Remove or block chunks | ID and document lookup | Zero active chunks |
| Dense vector index | Remove or exclude points | Point lookup and semantic search | No eligible hits |
| Sparse/keyword index | Remove lexical records | Exact search | No eligible hits |
| Hybrid search and reranker | Check candidate paths | Separate and combined traces | No deleted evidence |
| Retrieval/response caches | Invalidate dependencies | Old-key replay | No stale answer |
| Conversations and memory | Revoke old source context | Existing-session test | No new reliance on deleted source |
| Ingestion queue | Block pending job | Controlled replay | No re-ingestion |
| Snapshots and backups | Reconcile current deletion state | Isolated restore test | No reactivation |
| External providers | Check storage/retention controls | Provider configuration/evidence | Residual exposure documented |
Mark a layer not applicable only when absent from the actual system. Missing evidence is not a pass.
Eight concrete acceptance tests
1. Look up known IDs
Query the canonical document ID, version and every recorded chunk ID. Pass: nothing remains accessible through an authorised production response path; blocked-but-retained records are disclosed separately. Azure AI Search deletion is idempotent and may report success for a key that was already absent. Check the resulting state, not just the API response. Microsoft documentation.
2. Search for a unique marker
Plant a synthetic marker before ingestion, then search for it using exact-match, lexical and metadata paths after deletion. Pass: zero active matches.
3. Ask a semantic question
Ask about a fact unique to the document without using its title or marker. Inspect retrieved candidates, reranker inputs, final model context and citations. Pass: the excluded source contributes no evidence. Independently authorised knowledge may still support a valid answer.
4. Test dense, sparse and hybrid retrieval separately
Run semantic vector search, lexical/sparse search and the full hybrid pipeline as distinct tests. Pass: each returns zero eligible results from the deleted document. A clean hybrid result cannot substitute for checking its component indexes.
5. Replay an old application response-cache key
Cache an answer before deletion and preserve its key. After deletion, request the old key and repeat the normal question. Pass: no stale response is served as valid evidence.
Distinguish application response caching from provider prompt caching. OpenAI's documentation describes internal prompt-cache representations and retention options. Clearing a local answer cache does not prove removal of a provider's internal cache.
6. Continue an existing conversation
Have the assistant quote the document, delete it, then ask a follow-up in the same session. Pass: transcripts, summaries and memory do not act as an authorised replacement source for a new answer. Removing historic displayed messages is a separate retention issue.
For related access boundaries, see RAG Permissions and Security Trimming.
7. Replay a pending ingestion job
Hold a worker with the document, complete deletion and resume it. Pass: the deletion registry explicitly blocks republishing. A worker failing for an unrelated reason does not pass.
8. Restore an older snapshot
Restore an image containing the document into an isolated environment. Reconcile it with the current deletion journal before enabling retrieval. Repeat ID, marker and semantic checks. Pass: the document cannot reappear in permitted results.
Fictional example: policy-2026-017
Entirely fictional: These are invented fixture values and expected outcomes, not measured test results or a customer case.
| Property | Synthetic value |
|---|---|
| Document ID | policy-2026-017 |
| Version | v3 |
| Tenant | tenant-demo-at |
| Marker | RAG-DEL-7Q4M |
| Chunks | 8, policy-2026-017-v3-c01 through c08 |
| Search | Vector index and keyword index |
| Synthetic statement | “The inspection interval for RAG-DEL-7Q4M is 17 days.” |
| Deletion job | delete-policy-2026-017-001 |
Before deletion, establish the fixture: the document exists; eight chunks are retrievable; exact and semantic search find the statement; an answer is cached.
Execute deletion: register a block on version v3, remove source-derived artefacts and eight chunks, clean both indexes, invalidate cache dependencies and revoke old document context in conversations.
Expected post-deletion outcome:
| Check | Required result |
|---|---|
| Document lookup | Zero active records |
| Chunk lookup | Zero active chunks |
| Exact marker search | Zero matches |
| Dense / keyword / hybrid search | Zero results from this document in each path |
| Old response cache | No stale answer |
| Existing conversation | No new reliance on the deleted policy |
| Retry and restore | No reactivation |
A question about the inspection interval must not reuse “17 days” from the deleted policy. If no other authorised source supports a value, the assistant should disclose the absence of evidence.
These are expected results only. Actual acceptance requires execution logs and observed outcomes.
Failure modes that invalidate a deletion claim
HTTP 200 for a missing ID: Operation-level success is not evidence of the resulting state.
Asynchronous deletion: Qdrant distinguishes acknowledgement from completed execution and supports a wait option. Even completion does not verify application caches or conversations; see its points documentation.
Vector index cleaned, keyword index left intact: Lexical retrieval still exposes the content.
Old response cache: A precomputed answer bypasses the cleaned index.
Conversation as replacement evidence: Previously retrieved text remains in a summary or long-term memory.
Retry re-ingestion: A stale worker republishes the source.
Old backup restores deleted knowledge: Snapshot reconciliation is missing.
Functional exclusion misreported as physical erasure: Inaccessible records may still reside in storage or backups.
Simply telling an LLM to forget the source does not replace any of these controls.
Operational deletion metrics
| Metric | What it reveals |
|---|---|
| Pending deletion jobs | Outstanding work |
| Failed deletion jobs | Components needing intervention |
| Functional-exclusion latency | Time from authorised deletion to verified exclusion |
| Residual retrieval hits | Remaining matches by search path |
| Cache invalidation failures | Stale response dependencies |
| Blocked re-ingestion attempts | Effectiveness of tombstone checks |
| Deadline breaches | Missed technical or retention deadlines |
Record tenant, deletion job, system version, index generation and test timestamp. Avoid copying full sensitive source documents into observability logs.
When can deletion be signed off?
Functional exclusion acceptance requires:
- Stable document/version identity and complete known lineage.
- Effective re-ingestion block.
- All eight tests passed or individually justified as inapplicable.
- No deleted evidence enters permitted retrieval, reranking, model context, answers or citations.
- Response caches, existing conversations and persistent memory addressed.
- Retry, rebuild, migration and restore cannot reactivate the source.
- Reproducible records of test outputs, timestamps, configuration and index generations.
Physical removal additionally requires an inventory of residual stores, backup expiry or cleanup timelines, provider controls and available deletion evidence.
Article 17 of the GDPR defines erasure rights, conditions and exceptions. The German Data Protection Conference's October 2025 RAG guidance discusses data-subject rights in RAG systems. Technical acceptance is not a legal compliance determination.
Do not claim “the document has been completely deleted from all systems” without evidence covering that scope. A defensible statement is: “The document has been excluded from all verified production response paths since the recorded verification time. Remaining physical storage and backup cleanup are documented separately.”
Conclusion: deletion is a system acceptance process
Reliable RAG deletion requires traceable document identity, propagation through all retrieval paths, cache and memory controls, a re-entry block and repeatable negative tests.
See RAG Evaluation and Golden Datasets for broader evaluation methodology. Deletion requires its own acceptance boundary.
If a RAG pilot ingests documents but has not reproducibly tested delete propagation, cache invalidation and re-ingestion prevention, that path can be assessed as a bounded part of AI Production Readiness. The deliverable is an evidence-based review of failure modes, outstanding storage risks and release criteria.