Are Vector Databases Still Necessary With Longer Context Windows?

1. Overview

Each generation of large language models (LLMs) comes with a longer context window. So many teams are asking: if we can fit an entire document collection into a prompt, do we still need to build RAG with a vector database?

This article doesn’t answer that question with a simple yes or no. You’ll see where the two approaches differ. The article includes Java code to estimate your token budget, a minimal vector store to understand how retrieval works, and a simple strategy selector. By the end, you’ll be able to make your own decision for your system.

2. Two ways to bring knowledge into an LLM

LLMs don’t know your internal data—for example, refund policies, API documentation, or a customer’s order history. There are two main ways to get that data into the answer.

2.1. Retrieval with a vector database

A typical RAG (Retrieval-Augmented Generation) process has four steps:

1. Break documents into smaller chunks.
2. Use an embedding model to turn each chunk into a numerical vector.
3. Store the vectors in a vector database with an index for finding nearest neighbors.
4. When a user asks, embed their question, fetch the top-K nearest chunks, and put them in the prompt.

The final prompt contains only a few relevant chunks, not the entire document collection.

2.2. Long context: “stuff it all in the prompt”

This approach is much simpler: concatenate all documents into the prompt and let the model find the information itself. You don’t need chunking, embedding, or additional infrastructure.

For a small document collection, this usually works well. The model sees the full context, so it can answer questions that require synthesizing multiple parts—something top-K retrieval might miss.

3. Why this question is getting attention now

There are several reasons this debate is heating up:

• Context windows are growing fast. Document collections that once had to be split can now fit in a single prompt.
• Prompt caching. Many providers let you cache a fixed prompt prefix. This significantly reduces the cost and latency of sending the same document block repeatedly.
• RAG is hard to get right. Incorrect chunking, language-mismatched embeddings, and top-K missing important chunks are common errors. Any team that’s debugged a RAG pipeline knows the desire to drop at least one layer of complexity.

Long context has its own limits, though. Information retrieval quality in very long prompts can be uneven: information in the middle of a large text block is sometimes overlooked. Cost and latency still grow with the number of input tokens. There are also problems that a long context window doesn’t solve, like access control and constantly changing data. The sections below dive into each point.

4. Starting with numbers: estimating your token budget

Before choosing an architecture, you need to know how many tokens your document collection weighs. The example below uses only standard Java.

import java.util.List;

public class TokenBudget {

    // Rough estimate; a real tokenizer gives different results, especially for Vietnamese
    private static final double CHARS_PER_TOKEN = 4.0;

    public static long estimateTokens(String text) {
        return Math.round(text.length() / CHARS_PER_TOKEN);
    }

    public static long estimateCorpus(List<String> documents) {
        return documents.stream().mapToLong(TokenBudget::estimateTokens).sum();
    }

    public static void main(String[] args) {
        List<String> policies = List.of(
            "Refund policy: customers receive a refund if an order has defects.",
            "Shipping fee is calculated based on weight and delivery distance.",
            "Customer accounts are locked after multiple failed login attempts."
        );
        long total = estimateCorpus(policies);
        long budget = 8_000; // the part of context you allocate for documents
        System.out.printf("Corpus ~%d tokens, budget %d, fits=%b%n",
            total, budget, total <= budget);
    }
}

Here’s what each part does:

• CHARS_PER_TOKEN is a very rough heuristic. Vietnamese with diacritics typically costs more tokens than English. For precise numbers, you should use the tokenizer of your actual model.
• budget is not your entire context window. You must reserve space for the system prompt, conversation history, and responses.
• For the three short passages, the result is:

Corpus ~48 tokens, budget 8000, fits=true

If fits=true and your document collection changes rarely, long context is a serious candidate. If the collection is many times larger than your budget, you still need retrieval, no matter how long the context window is.

5. A minimal vector store to understand the mechanism

To see what retrieval really does, let’s write an in-memory vector store. The embedding here is a “hashing bag-of-words” type, purely for illustration. A real system would call an embedding model.

import java.util.*;

public record Document(String id, String text) {}

public record ScoredDocument(Document document, double score) {}

public class HashingEmbedder {
    private final int dimensions;

    public HashingEmbedder(int dimensions) { this.dimensions = dimensions; }

    public double[] embed(String text) {
        double[] vector = new double[dimensions];
        for (String token : text.toLowerCase().split("\\P{L}+")) {
            if (token.isBlank()) continue;
            vector[Math.floorMod(token.hashCode(), dimensions)] += 1.0;
        }
        double norm = Math.sqrt(Arrays.stream(vector).map(v -> v * v).sum());
        if (norm > 0) for (int i = 0; i < dimensions; i++) vector[i] /= norm;
        return vector;
    }
}

public class InMemoryVectorStore {
    private record Entry(Document document, double[] vector) {}

    private final HashingEmbedder embedder;
    private final List<Entry> entries = new ArrayList<>();

    public InMemoryVectorStore(HashingEmbedder embedder) { this.embedder = embedder; }

    public void add(Document document) {
        entries.add(new Entry(document, embedder.embed(document.text())));
    }

    public List<ScoredDocument> search(String query, int topK) {
        double[] q = embedder.embed(query);
        return entries.stream()
            .map(e -> new ScoredDocument(e.document(), dot(q, e.vector())))
            .sorted(Comparator.comparingDouble(ScoredDocument::score).reversed())
            .limit(topK)
            .toList();
    }

    private static double dot(double[] a, double[] b) {
        double sum = 0;
        for (int i = 0; i < a.length; i++) sum += a[i] * b[i];
        return sum;
    }
}

Here are the key points:

• embed splits tokens by non-letter characters (\\P{L}+), so it preserves Vietnamese diacritics. Each word is hashed into one dimension of the vector.
• Vectors are normalized to unit length. Then the dot product (dot) is exactly the cosine similarity.
• search scans all entries. This is a brute-force search with O(n) complexity. A real vector database uses approximate indices like HNSW to stay fast with many vectors.

Running the example:

var store = new InMemoryVectorStore(new HashingEmbedder(1024));
store.add(new Document("refund", "Refund policy: customers receive a refund if an order has defects."));
store.add(new Document("shipping", "Shipping fee is calculated based on weight and delivery distance."));
store.add(new Document("account", "Customer accounts are locked after multiple failed login attempts."));

store.search("customer wants a refund", 2)
     .forEach(r -> System.out.printf("%s -> %.3f%n", r.document().id(), r.score()));

refund -> 0.6xx
shipping -> 0.1xx

The exact numbers depend on the hashing function. The order is stable: refund ranks first because it shares many words with the question. Only that chunk goes into the prompt, not all three.

Note: Real embeddings capture semantics (refunding money is similar to getting money back), but hashing bag-of-words doesn’t. Don’t use the example above in production.

In production, if you’re already using PostgreSQL, the pgvector extension is usually a sensible first step before considering a dedicated vector database:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE policy_chunk (
    id        BIGSERIAL PRIMARY KEY,
    tenant_id BIGINT NOT NULL,
    content   TEXT NOT NULL,
    embedding vector(768)
);

CREATE INDEX ON policy_chunk USING hnsw (embedding vector_cosine_ops);

-- :query_embedding provided by the application
SELECT id, content
FROM policy_chunk
WHERE tenant_id = :tenant_id
ORDER BY embedding <=> :query_embedding
LIMIT 5;

The dimension 768 must match your embedding model. The <=> operator computes cosine distance. The WHERE tenant_id = ... clause points to an important reason retrieval is still necessary, discussed in section 7.

6. Combining both approaches: choosing strategy by data

In practice, the answer is often “it depends on your data”. We can codify this decision-making so your team aligns on criteria.

public enum ContextStrategy { FULL_CONTEXT, RETRIEVAL }

public record CorpusProfile(long totalTokens,
                            boolean changesFrequently,
                            boolean needsPerUserAccess) {}

public class ContextStrategySelector {
    private final long documentBudgetTokens;

    public ContextStrategySelector(long documentBudgetTokens) {
        this.documentBudgetTokens = documentBudgetTokens;
    }

    public ContextStrategy choose(CorpusProfile profile) {
        if (profile.needsPerUserAccess()) return ContextStrategy.RETRIEVAL;
        if (profile.totalTokens() > documentBudgetTokens) return ContextStrategy.RETRIEVAL;
        if (profile.changesFrequently() && profile.totalTokens() > documentBudgetTokens / 2) {
            return ContextStrategy.RETRIEVAL; // prompt cache works poorly when data changes constantly
        }
        return ContextStrategy.FULL_CONTEXT;
    }
}

The logic follows priority order:

• If each user can only see part of the data, you must filter before putting it in the prompt. Retrieval’s filtering happens naturally.
• If the collection exceeds your token budget, there’s no choice but retrieval.
• If data changes constantly and already uses a significant budget, prompt caching is less effective, so retrieval is usually cheaper.
• For other cases: use long context for simplicity.

The / 2 threshold is just an example. You should measure cost and quality on your own data and adjust accordingly.

7. Testing main behaviors

For testing dependencies, if your project uses a Spring Boot parent, versions are already managed:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;

class RetrievalAndStrategyTest {

    @Test
    void givenRefundQuestion_whenSearch_thenRefundPolicyRanksFirst() {
        var store = new InMemoryVectorStore(new HashingEmbedder(1024));
        store.add(new Document("refund", "Refund policy: customers receive a refund if an order has defects."));
        store.add(new Document("shipping", "Shipping fee is calculated based on weight and delivery distance."));

        var results = store.search("customer wants a refund", 1);

        assertThat(results).hasSize(1);
        assertThat(results.getFirst().document().id()).isEqualTo("refund");
    }

    @Test
    void givenSmallStaticCorpus_whenChoose_thenFullContext() {
        var selector = new ContextStrategySelector(50_000);
        var profile = new CorpusProfile(10_000, false, false);

        assertThat(selector.choose(profile)).isEqualTo(ContextStrategy.FULL_CONTEXT);
    }

    @Test
    void givenPerUserAccess_whenChoose_thenRetrievalEvenIfSmall() {
        var selector = new ContextStrategySelector(50_000);
        var profile = new CorpusProfile(1_000, false, true);

        assertThat(selector.choose(profile)).isEqualTo(ContextStrategy.RETRIEVAL);
    }
}

The third test is the most important from a business perspective. Even with a small document collection, access control requirements force you to use retrieval. That’s a constraint that a long context window can’t solve.

8. When you still need vector database, when you don’t

You still need retrieval (a vector database, or a vector index in an existing DB) when:

• Your document collection is many times larger than the context window—for example, all support tickets or an entire internal wiki.
• You have per-user or per-tenant access control. You can’t let the model “see” data the user isn’t authorized to view, even just in the prompt.
• Data updates constantly and needs to be reflected almost immediately.
• You have high query volume. Sending a huge prompt for every request makes cost and latency scale with token count.
• You need clear source attribution—knowing which chunk the answer is based on.

You can skip a vector database when:

• The collection is small, stable, and fits in your token budget—like a product’s policy set or a single module’s documentation.
• Questions typically require synthesizing the entire collection, like “summarize all differences between these two contracts”. Top-K retrieval handles this poorly.
• You’re prototyping and want to validate value before investing in infrastructure.

You don’t necessarily need vector search for exact queries like order codes, SKUs, or function names. Full-text search or SQL queries usually work better. Many systems use hybrid search, combining keywords and vectors.

9. Common mistakes

• Treating context window as your document budget. You forget to reserve space for the system prompt, conversation history, and output, so requests get truncated or error in production.
• Stuffing everything in the prompt without evaluating quality. The longer the prompt, the more easily the model overlooks details in the middle. Build a test set of questions with reference answers and measure on your data.
• Building a dedicated vector database too early. With tens of thousands of chunks, a vector index in PostgreSQL is usually sufficient, and you avoid running another system.
• Filtering permissions after retrieval. Taking top-K and then filtering by access rights can drop all valid results. Put access conditions in the query itself, like the WHERE tenant_id clause in section 5.
• Changing embedding models without re-indexing. Vectors from different models aren’t comparable.

Note: Long context and retrieval are not mutually exclusive. A common pattern is generous retrieval—fetching more chunks than before—then using the long context to let the model self-filter. This reduces the risk of missing information while keeping costs under control.

10. Conclusion

Longer context windows change the balance, but they don’t eliminate the need for retrieval. With a small, stable collection, “stuffing it all in the prompt” is usually simple and good enough. With large, frequently changing data, access requirements, or high volume, retrieval is still necessary. However, it doesn’t have to be a standalone vector database: a vector index in an existing database is often enough.

The practical approach is to measure first, then decide. Estimate your collection’s token count, determine access requirements and update frequency, then evaluate quality on real questions. When those criteria become code like ContextStrategySelector, your team has a clear, testable decision.