Updating RAG Documents: Versioning Over Overwriting

1. Overview

In RAG (Retrieval-Augmented Generation) systems, source documents are never static: policies change, prices change, regulations change. If you simply “overwrite” old chunks with new chunks in the vector store, the system easily falls into a state where it answers both correctly and incorrectly the same question. This article explains why simple overwriting is a dangerous design, and presents a safer update process: prepare new chunks, verify them, then switch the active version — with concrete Java code examples.

2. The Problem: Why Direct Overwriting Causes Inconsistency

Suppose your company increases the hotel reimbursement limit from 5,000,000 VND to 7,000,000 VND. Policy documents are divided into multiple chunks, each chunk is embedded as a vector and stored in a vector database. When the policy changes, if you:

• Insert the new chunk first, delete the old chunk after → there is a period of time when both chunks coexist and are both considered “valid for the current question”. Semantic search queries might return the old chunk (5,000,000 VND) because its vector similarity with the question is still high.
• Delete the old chunk first, insert the new chunk after → there is a period of time when no chunk can answer the question about the limit, forcing the LLM to “make up” or respond that there is no information.

Both approaches have problems because they treat “update” as a single action, when in fact it’s two operations (adding new + removing old) that are not atomic.

// Wrong approach: implicit overwriting, no control over transition state
public void updatePolicyNaive(VectorStore store, String oldChunkId, PolicyChunk newChunk) {
    store.delete(oldChunkId);          // (1) creates a temporary gap
    store.upsert(newChunk.id(), newChunk.embedding(), newChunk.metadata());
}

Line (1) creates a gap: while delete has run but upsert is not finished (or fails midway), queries will not find any chunk about the reimbursement limit.

Note: With production RAG systems, this gap is not theoretical — it happens every time you deploy an update job without transaction or versioning mechanisms.

3. When to Re-embed, When Not To

Not every change requires recalculating the vector. Vectors are generated from the text content of the chunk. If you only change metadata (e.g., effective date, applicable department) without changing the wording in the chunk, you don’t need to re-embed — just update the metadata.

public record PolicyChunk(
        String id,
        String text,              // content used to generate vector
        float[] embedding,
        Map<String, String> metadata // NOT included in text used for embedding
) {}

If text changes (e.g., “5,000,000 VND” → “7,000,000 VND”), the old vector no longer reflects the correct content → must re-embed that chunk. But other chunks in the document (e.g., the section about applicable conditions, where the amount doesn’t change) keep their vectors unchanged — no need to touch them.

public interface VectorStore {
    void upsertVector(String chunkId, float[] embedding);
    void upsertMetadata(String chunkId, Map<String, String> metadata);
    Optional<PolicyChunk> findById(String chunkId);
    void delete(String chunkId);
}

Separating upsertVector and upsertMetadata into two separate methods lets you choose deliberately: change metadata without re-embedding, change text then re-embed. Some vector databases (Qdrant, Weaviate, Pinecone) support updating payload/metadata independently from vectors — check your product’s documentation before writing update logic.

4. Designing Version and Active-Version Flag

Instead of editing chunks in place, a safer approach is to treat each content change as a new version with its own versionId. Old chunks are not deleted immediately — they are only marked as no longer “active” for current policy queries.

public record PolicyVersion(
        String versionId,      // e.g., "hotel-reimbursement-v2"
        String policyKey,      // e.g., "hotel-reimbursement"
        List<PolicyChunk> chunks,
        Instant effectiveFrom
) {}

public class ActiveVersionRegistry {

    private final Map<String, String> activeVersionByPolicyKey = new ConcurrentHashMap<>();

    public void setActive(String policyKey, String versionId) {
        activeVersionByPolicyKey.put(policyKey, versionId);
    }

    public Optional<String> getActiveVersion(String policyKey) {
        return Optional.ofNullable(activeVersionByPolicyKey.get(policyKey));
    }
}

ActiveVersionRegistry only stores a mapping: for each policyKey, which version is considered “current”. When searching, you filter by metadata.versionId == activeVersion, not by whether the chunk still exists or not. This way, version switching becomes a pointer-swap operation, fast and nearly atomic — not a series of scattered insert/delete across many chunks.

5. Safe Update Process: Prepare → Verify → Switch Active

The process consists of three clear steps, avoiding both pitfalls from section 2.

@Service
public class PolicyUpdateService {

    private final VectorStore vectorStore;
    private final EmbeddingClient embeddingClient;
    private final ActiveVersionRegistry registry;

    public PolicyUpdateService(VectorStore vectorStore,
                                EmbeddingClient embeddingClient,
                                ActiveVersionRegistry registry) {
        this.vectorStore = vectorStore;
        this.embeddingClient = embeddingClient;
        this.registry = registry;
    }

    public void applyNewVersion(String policyKey, PolicyVersion newVersion) {
        // Step 1: prepare — embed and write new chunks, NOT YET changing active version
        for (PolicyChunk chunk : newVersion.chunks()) {
            float[] vector = embeddingClient.embed(chunk.text());
            Map<String, String> metadata = new HashMap<>(chunk.metadata());
            metadata.put("versionId", newVersion.versionId());
            metadata.put("policyKey", policyKey);
            vectorStore.upsertVector(chunk.id(), vector);
            vectorStore.upsertMetadata(chunk.id(), metadata);
        }

        // Step 2: verify — new chunks are truly readable, not empty
        boolean ready = newVersion.chunks().stream()
                .allMatch(c -> vectorStore.findById(c.id()).isPresent());
        if (!ready) {
            throw new IllegalStateException("New chunks not ready, not switching active version");
        }

        // Step 3: switch active version — fast operation, nearly atomic
        registry.setActive(policyKey, newVersion.versionId());
    }
}

Flow explanation:

• Step 1 runs before any queries are affected — new chunks exist alongside old chunks, but the old chunk is still active so users still get consistent answers (5,000,000 VND) throughout the preparation process.
• Step 2 ensures you don’t switch active version when new data hasn’t finished writing or fails partially — avoiding the situation “switch flag but chunks are missing”.
• Step 3 only changes one value in the registry. This is the single switching point, making it easy to control and easy to rollback (just set the old versionId again).

Old chunks (v1) are not deleted immediately. You can clean them up later, in a separate job, not on the critical path of the policy update.

6. Maintaining the Ability to Answer Historical Questions

A side benefit of the versioning design: you can still answer questions like “What was the hotel reimbursement limit last year?”. Since chunk v1 hasn’t been deleted, you just need a different filter at the retrieval layer.

public List<PolicyChunk> searchCurrentPolicy(String policyKey, float[] queryVector) {
    String activeVersion = registry.getActiveVersion(policyKey)
            .orElseThrow(() -> new IllegalStateException("No active version for " + policyKey));
    return vectorStore.searchSimilar(queryVector, Map.of(
            "policyKey", policyKey,
            "versionId", activeVersion
    ));
}

public List<PolicyChunk> searchHistoricalPolicy(String policyKey, String versionId, float[] queryVector) {
    return vectorStore.searchSimilar(queryVector, Map.of(
            "policyKey", policyKey,
            "versionId", versionId
    ));
}

searchCurrentPolicy always filters by activeVersion from the registry — this is the single place that decides “which version is current”. searchHistoricalPolicy allows explicit querying of a specific version, serving historical or audit questions.

7. Testing Active Version Switching Behavior

Write tests to ensure the process never exposes a “half-done” state to the outside.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.assertj</groupId>
    <artifactId>assertj-core</artifactId>
    <scope>test</scope>
</dependency>

class PolicyUpdateServiceTest {

    @Test
    void shouldKeepOldVersionActiveUntilNewVersionVerified() {
        InMemoryVectorStore store = new InMemoryVectorStore();
        ActiveVersionRegistry registry = new ActiveVersionRegistry();
        registry.setActive("hotel-reimbursement", "v1");

        PolicyUpdateService service = new PolicyUpdateService(
                store, text -> new float[]{0.1f, 0.2f}, registry);

        PolicyVersion v2 = new PolicyVersion(
                "v2", "hotel-reimbursement",
                List.of(new PolicyChunk("chunk-v2-1", "Limit 7,000,000 VND", null, Map.of())),
                Instant.now());

        service.applyNewVersion("hotel-reimbursement", v2);

        assertThat(registry.getActiveVersion("hotel-reimbursement"))
                .contains("v2");
        // chunk v1 is still in the store, serving historical queries
        assertThat(store.findById("chunk-v1-1")).isPresent();
    }
}

This test asserts two things: after applyNewVersion completes, the active version must be v2, and chunks from v1 are not lost — you can still query history.

8. Changing Embedding Model: A Bigger Problem

Everything above assumes the embedding model doesn’t change. If you switch from model A to model B (e.g., change provider, change vector dimensions), all existing chunks need to be re-embedded with the new model — not just chunks with modified content. Vectors from model A and model B do not lie in the same space; comparing cosine similarity between them is meaningless.

public void migrateEmbeddingModel(EmbeddingClient newModel, List<PolicyChunk> allChunks) {
    for (PolicyChunk chunk : allChunks) {
        float[] newVector = newModel.embed(chunk.text());
        vectorStore.upsertVector(chunk.id() + ":model-v2", newVector);
    }
    // After all chunks have new vectors, change the entry point of the query pipeline
    // so queries also use newModel when embedding questions.
}

The key note: user queries must also be embedded with the correct new model at search time. If you only change document vectors but forget to change the question embedding model, search results will be completely wrong because both sides are no longer in the same vector space.

9. Common Mistakes and When You Don’t Need Complexity

• Common mistake: treating update as a single upsert(), not separating “write new data” and “switch active” — leading to gaps or duplicates mentioned in section 2.
• Common mistake: letting metadata be part of the text used for embedding (e.g., embedding the effective date into the sentence), forcing every metadata change to re-embed, wasting unnecessary resources.
• Common mistake: forgetting that old chunks should be kept for historical queries — deleting right after switching active, losing the ability to answer “what was the policy before”.

When you DON’T need this entire versioning process: if your documents are nearly static (rarely change, no historical query requirements, system only runs internally with a few dozen users), a batch re-index job on a schedule (e.g., nightly) is simpler and safe enough — no registry, no complex versionId needed.

10. Conclusion

Updating documents in a RAG system is not about “overwriting old vectors with new ones” — it’s about managing the transition state between two data versions without letting the system fall into gaps or inconsistencies. Separate vectors and metadata to avoid unnecessary re-embedding, use versionId with active-version registry for fast and rollback-able switching, and always keep old data long enough to serve historical queries. When changing the embedding model itself, treat it as a complete migration, not a regular update.