LLMs Need Evidence to Answer Correctly: RAG Explained with Examples

1. Overview

No matter how powerful an LLM is, it doesn’t inherently know the internal documents of your company—it only knows what gets fed into its input. This article explains Retrieval-Augmented Generation (RAG): why you need it, how retrieval works, and why answer quality depends almost entirely on the search step before text generation. By the end, you’ll understand the data flow in a RAG system and be able to write a simple retrieval pipeline in Java.

2. The Problem: LLMs Have No “Memory” of Private Data

Models are trained on public datasets and frozen at a point in time. They don’t know about the new refund policy your company issued last week, let alone the contents of some internal contract.

There are two ways to feed private information:

1. Stuff it straight into the prompt — only viable if the document is small enough to fit in the context window.
2. Retrieval — when the document set is large, the app selects only the most relevant passages and then feeds them into the prompt.

// Approach 1: stuff the entire document—only fine for short text
String prompt = """
    Based on the following policy, answer the customer's question.
    Policy: %s
    Question: %s
    """.formatted(fullPolicyText, userQuestion);

This approach is simple but doesn’t scale: context windows have token limits, and the more unrelated text you stuff in, the more the model tends to get lost in the middle. When you have thousands of pages of documents, you’re forced to use retrieval.

3. What is RAG and How It Works

RAG (Retrieval-Augmented Generation) splits the problem into two independent stages:

• Retrieval: find the right passages of text relevant to the question.
• Generation: the LLM uses those passages to generate an answer, complete with source citations.

User question
      │
      ▼
[Embedding model] – convert question to vector
      │
      ▼
[Vector database] – find semantically closest passages
      │
      ▼
[Original passages] – feed raw text into prompt
      │
      ▼
[LLM] – generate answer + source citations

The crucial point: the LLM doesn’t “look up” data directly in the vector database. It only receives plain text provided by the application after retrieval is complete. The vector database is just a search tool, not a place where “knowledge” the model reads is stored.

3.1. Embedding – Turn Questions into Numbers

An embedding model converts text into a vector of real numbers (e.g., 1536 dimensions). Two passages with similar meaning produce vectors close to each other in that space.

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

@Service
public class PolicyEmbeddingService {

    private final EmbeddingModel embeddingModel;

    public PolicyEmbeddingService(EmbeddingModel embeddingModel) {
        this.embeddingModel = embeddingModel;
    }

    public float[] embed(String text) {
        EmbeddingResponse response = embeddingModel.embedForResponse(List.of(text));
        return response.getResult().getOutput();
    }
}

embed() takes a text passage (for example, the question “What’s the hotel cancellation policy?”) and returns a float[] array—a vector representing its semantic meaning. This is what gets matched against vectors already stored in the vector database.

3.2. Vector Database – Find Semantically Similar Passages

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-pgvector-store-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

@Service
public class PolicyRetrievalService {

    private final VectorStore vectorStore;

    public PolicyRetrievalService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    public void indexPolicy(String policyId, String content) {
        Document document = new Document(content, Map.of("policyId", policyId));
        vectorStore.add(List.of(document));
    }

    public List<Document> search(String question) {
        SearchRequest request = SearchRequest.query(question).withTopK(3);
        return vectorStore.similaritySearch(request);
    }
}

• indexPolicy() saves a policy passage to the vector database, with metadata (policyId) so you can later know which document that text came from.
• search() takes the user’s question, automatically embeds it internally, then finds the 3 semantically closest passages (topK(3)).

search("What's the hotel cancellation policy?") results:
1. "Guests can cancel for free up to 24 hours before..."
2. "Non-refundable bookings have no cancellation..."
3. "Cancellation fees apply to group bookings of 5+ people..."

3.3. Generation – LLM Answers Based on Evidence

@Service
public class PolicyAnswerService {

    private final ChatClient chatClient;
    private final PolicyRetrievalService retrievalService;

    public PolicyAnswerService(ChatClient.Builder builder,
                                PolicyRetrievalService retrievalService) {
        this.chatClient = builder.build();
        this.retrievalService = retrievalService;
    }

    public String answer(String question) {
        List<Document> passages = retrievalService.search(question);

        String context = passages.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n---\n"));

        String prompt = """
            Base your answer only on the passages below. If there's insufficient information, clearly state that you don't know.
            Passages:
            %s

            Question: %s
            """.formatted(context, question);

        return chatClient.prompt(prompt).call().content();
    }
}

Notice the instruction “Base your answer only on the passages”—this is a key constraint to reduce hallucination: the model is asked to stick closely to evidence rather than make up additional information from its training knowledge.

Note: Without explicitly constraining “base your answer only on the context,” the model can still blend its background knowledge with retrieved content, leading to answers that sound plausible but diverge from the actual documents.

4. Why Bad Retrieval Means Bad Answers

This part is easy to overlook: LLMs are only good at interpreting what they receive; they have no way to independently verify whether that information is correct or complete.

Two common scenarios:

Scenario 1 – Retrieval returns outdated documents. If the vector database still indexes an old policy version (never updated or deleted), the system returns outdated text. The LLM reads it and answers with complete confidence—but that confidence is wrong.

Scenario 2 – Retrieval lacks crucial context. Suppose the original document is structured like this:

Article 5. Cancellation Policy
Guests can cancel for free up to 24 hours before check-in.

Article 5.1. Exception
For bookings during Lunar New Year holidays, free cancellation does not apply;
a fixed cancellation fee of 50% of the booking value applies.

If the chunking step splits Article 5 and Article 5.1 into separate chunks, and retrieval only grabs the Article 5 chunk (because the question doesn’t contain “Lunar New Year” or “exception”), the LLM will answer “you can cancel for free up to 24 hours”—sounds very plausible, but misses the critical exception condition.

@Test
void whenChunkMissingException_LLMMissesTheCondition() {
    // Simulate retrieval returning only 1 chunk, missing the exception
    List<Document> incompletePassages = List.of(
        new Document("Guests can cancel for free up to 24 hours before check-in.")
    );

    String answer = fakeAnswerService.answerFromPassages(
        "I booked during Lunar New Year. Is there a cancellation fee?",
        incompletePassages
    );

    // Answer will be wrong because it lacks evidence about the holiday exception
    assertThat(answer).doesNotContain("50%");
}

This test isn’t trying to catch the LLM being dumb—it illustrates that the problem lies with retrieval, not the model’s reasoning ability. If you provide enough evidence (both chunks), the answer will be correct even with a mid-tier model.

5. Chunking: The Key to “Just Right” Retrieval

Since the example above shows how wrong chunking has consequences, this is where you should invest the most when building RAG.

@Service
public class PolicyChunkingService {

    public List<String> splitByHeading(String document) {
        // Split on pattern "Article X" or "Article X.Y" to preserve parent-child context
        String[] rawChunks = document.split("(?=Article \\d+)");
        return Arrays.stream(rawChunks)
            .map(String::trim)
            .filter(chunk -> !chunk.isBlank())
            .toList();
    }
}

Instead of splitting on a fixed character count (e.g., every 500 characters—easy to cut between an article and its exception), this function splits on semantic boundaries (Article X). This keeps each chunk as a complete unit of information, reducing the risk of separating a rule from its exceptions.

@Test
void splitByHeading_ShouldKeepArticleAndExceptionTogether() {
    String document = """
        Article 5. Cancellation Policy
        Guests can cancel for free up to 24 hours.

        Article 5.1. Exception
        For Lunar New Year, the cancellation fee is fixed at 50%.
        """;

    List<String> chunks = chunkingService.splitByHeading(document);

    assertThat(chunks).hasSize(2);
    assertThat(chunks.get(1)).contains("Exception", "50%");
}

Note: Heading-based chunking only works well for structured documents (contracts, regulations, technical docs). For unstructured text (emails, chat logs), you need a different strategy—for example, split by paragraphs with overlap, or use semantic chunking based on sentence-to-sentence similarity.

6. When NOT to Use RAG

RAG isn’t a silver bullet:

• Small, fixed dataset: if your entire document set is just a few thousand tokens, stuffing it straight into the prompt is simpler and less error-prone than building a whole retrieval pipeline.
• Questions requiring complex multi-step reasoning where the answer doesn’t live directly in any single passage—RAG finds “passages similar to the question,” not “passages containing the reasoned answer.”
• Data changing constantly by the second (stock prices, real-time inventory): better to call an API directly (function calling) than constantly re-index the vector database.

7. Conclusion

LLMs don’t truly “know” your private data—they’re only good at interpreting provided evidence. RAG solves this by separating retrieval (finding the right passages) from generation (crafting the answer), with embedding models and vector databases as the bridge. The quality of your final answer depends almost entirely on how well you index and chunk your data—no amount of LLM firepower can save poor retrieval. If you’re building RAG, spend time validating your chunking and measuring retrieval accuracy before thinking about upgrading to a larger model.