EN
한국어FrançaisPortuguêsEspañolEnglishTiếng ViệtDeutsch简体中文日本語
Home / AI & Machine Learning / RAG

RAG from A to Z: retrieve documents then answer

Cris The Dev · 10/13/2026 · 10 min read · 110

1. Overview

LLMs only know what’s in their training data. They don’t know your store’s return policy or your company’s internal procedures. When asked about those things, they often make up answers. RAG (Retrieval-Augmented Generation) fixes this: the system retrieves relevant documents first, then lets the LLM answer based on those documents.

In this article, you’ll build a complete RAG pipeline using Spring Boot 3 and Spring AI. The pipeline includes document loading, chunking, embedding creation, search, prompt augmentation with sources, and testing. By the end, you’ll see common errors and when not to use RAG.

2. How RAG works

RAG has two separate phases.

The ingestion phase runs once or periodically:

  1. Read documents (Markdown, PDF, HTML…).
  2. Split into chunks of a few hundred tokens.
  3. Call an embedding model to turn each chunk into a numeric vector.
  4. Store the vector and chunk content in a vector store.

The query phase runs each time a user asks:

  1. Embed the question into a vector.
  2. Find chunks with the closest vectors (similarity search).
  3. Append those chunks to the prompt. This is the augmentation step.
  4. Send the prompt to the LLM to generate an answer. This is the generation step.

The core idea is very simple. The LLM doesn’t need to “memorize” your documents; it just needs to read the right passages directly in the prompt.

3. Project setup

Add the Spring AI BOM and OpenAI starter. This starter provides both the chat model and embedding model.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>1.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

Next, configure the model:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o-mini
spring.ai.openai.chat.options.temperature=0.1
spring.ai.openai.embedding.options.model=text-embedding-3-small

You should set temperature low. With RAG, the goal is to answer closely tied to the documents, not be creative.

Finally, declare two beans: a vector store and a ChatClient.

@Configuration
public class RagConfig {

    @Bean
    VectorStore vectorStore(EmbeddingModel embeddingModel) {
        return SimpleVectorStore.builder(embeddingModel).build();
    }

    @Bean
    ChatClient chatClient(ChatClient.Builder builder) {
        return builder.build();
    }
}

SimpleVectorStore stores all vectors in memory. It’s suitable for learning and demos. EmbeddingModel is auto-configured by Spring AI from the OpenAI starter. Each time you add a document, the store calls this model to create vectors.

Note: SimpleVectorStore loses all data when the application restarts and does linear search. For production environments, use PGvector, Qdrant, Elasticsearch… Spring AI has a starter for each type, and the VectorStore interface remains the same so the code below doesn’t need to change.

4. Loading documents

Suppose you have a file src/main/resources/docs/return-policy.md:

# Return Policy

Customers may return products within 30 days of purchase.
Products must have intact tags, labels, and receipt.
Refunds will be processed to your account within 5–7 business days.
Underwear and items discounted over 50% are not returnable.

4.1. Reading and chunking

@Component
public class DocumentIngestor implements ApplicationRunner {

    private final VectorStore vectorStore;
    private final ResourcePatternResolver resolver;

    public DocumentIngestor(VectorStore vectorStore, ResourcePatternResolver resolver) {
        this.vectorStore = vectorStore;
        this.resolver = resolver;
    }

    @Override
    public void run(ApplicationArguments args) throws IOException {
        // chunkSize=300 tokens, minChunkSizeChars=100, minChunkLengthToEmbed=5,
        // maxNumChunks=10000, keepSeparator=true
        var splitter = new TokenTextSplitter(300, 100, 5, 10_000, true);

        for (Resource file : resolver.getResources("classpath:docs/*.md")) {
            var reader = new TextReader(file);
            reader.getCustomMetadata().put("source", file.getFilename());

            List<Document> chunks = splitter.apply(reader.get());
            vectorStore.add(chunks);
            System.out.printf("Loaded %s: %d chunks%n", file.getFilename(), chunks.size());
        }
    }
}

This code does the following:

  • ApplicationRunner runs once after the context starts up, so documents are always loaded before the first request arrives.
  • TextReader reads a file into a Document. We attach metadata source as the filename to use when citing sources later.
  • TokenTextSplitter cuts text into chunks of about 300 tokens. Metadata is copied to each chunk.
  • vectorStore.add(...) calls the embedding model on each chunk and stores the vectors.

When starting up, you’ll see:

Loaded return-policy.md: 1 chunk
Loaded shipping.md: 3 chunks

4.2. Choosing chunk size

Chunk size directly affects answer quality:

  • Chunks too large: one chunk covers many topics, vectors become “diluted” so they don’t match precisely. Prompts also use more tokens.
  • Chunks too small: sentences lose context. For example, “within 30 days” ends up in one chunk while “return” is in another.

A range of 200–500 tokens is a good starting point for prose documents. For structured documents (FAQs, tables of contents), split by heading so each chunk is a complete idea.

5. Retrieving relevant documents

Retrieval takes just one call:

List<Document> docs = vectorStore.similaritySearch(
        SearchRequest.builder()
                .query("How long can I return something?")
                .topK(4)
                .similarityThreshold(0.5)
                .build());
  • query: the question. The vector store automatically embeds the question using the same model used at ingestion.
  • topK(4): get at most 4 nearest chunks.
  • similarityThreshold(0.5): exclude chunks with cosine similarity below 0.5. Without a threshold, the store always returns topK results, even if they’re completely unrelated to the question.

There’s no optimal threshold for all cases. You should log scores for a few dozen real questions, then choose a threshold that separates correct results from noise.

6. Building prompts and generating answers

6.1. Prompt builder

Separating prompt construction into a pure function makes testing easier:

public final class PromptBuilder {

    private static final String TEMPLATE = """
            You are a customer support assistant.
            Answer ONLY based on the CONTEXT below.
            If the context doesn't contain the information, reply exactly:
            "I couldn't find this information in the documents."
            At the end of your answer, cite your source in brackets, like [return-policy.md].

            CONTEXT:
            {context}

            QUESTION: {question}
            """;

    private PromptBuilder() {}

    public static String build(String question, List<Document> docs) {
        String context = docs.stream()
                .map(d -> "[" + d.getMetadata().getOrDefault("source", "unknown") + "]\n" + d.getText())
                .collect(Collectors.joining("\n---\n"));
        return TEMPLATE.replace("{context}", context)
                       .replace("{question}", question);
    }
}

Three points in the template are crucial:

  1. “Answer ONLY based on the CONTEXT” constrains the LLM to the documents, reducing hallucinations.
  2. A fixed refusal sentence gives the LLM a valid way out when documents lack information.
  3. Each chunk comes with its source name, so the LLM can cite it and users can verify.

Here we use replace instead of String.formatted. Reason: document content may contain % characters (e.g., “discounted 50%”), and formatted throws UnknownFormatConversionException on seeing those.

6.2. RagService

@Service
public class RagService {

    static final String NOT_FOUND = "I couldn't find this information in the documents.";

    private final VectorStore vectorStore;
    private final ChatClient chatClient;

    public RagService(VectorStore vectorStore, ChatClient chatClient) {
        this.vectorStore = vectorStore;
        this.chatClient = chatClient;
    }

    public String answer(String question) {
        List<Document> docs = vectorStore.similaritySearch(
                SearchRequest.builder()
                        .query(question)
                        .topK(4)
                        .similarityThreshold(0.5)
                        .build());

        if (docs == null || docs.isEmpty()) {
            return NOT_FOUND;
        }

        return chatClient.prompt()
                .user(PromptBuilder.build(question, docs))
                .call()
                .content();
    }
}

When no chunks are found, the service returns a refusal immediately without calling the LLM. This saves cost and completely prevents the LLM from making up an answer from general knowledge.

Finally, the controller:

@RestController
public class AskController {

    private final RagService ragService;

    public AskController(RagService ragService) {
        this.ragService = ragService;
    }

    @GetMapping("/ask")
    public String ask(@RequestParam String q) {
        return ragService.answer(q);
    }
}

Now you can try the API:

curl -G "http://localhost:8080/ask" --data-urlencode "q=How long can I return something?"
curl -G "http://localhost:8080/ask" --data-urlencode "q=Do you sell laptops?"
You can return within 30 days of purchase, as long as the product still has its tags, label, and receipt. [return-policy.md]
I couldn't find this information in the documents.

The first answer sticks to the documents and cites the source. The second doesn’t match any document, so the system refuses instead of making something up.

7. Writing tests

Don’t call the LLM in unit tests because it’s slow, costs money, and results are unpredictable. Instead, test two deterministic parts: the prompt builder and the refusal path when there are no documents.

class RagServiceTest {

    @Test
    void promptContainsContextSourceAndQuestion() {
        var doc = new Document("Return within 30 days.", Map.of("source", "return-policy.md"));

        String prompt = PromptBuilder.build("How long to return?", List.of(doc));

        assertThat(prompt)
                .contains("[return-policy.md]")
                .contains("Return within 30 days.")
                .contains("QUESTION: How long to return?")
                .doesNotContain("{context}", "{question}");
    }

    @Test
    void percentSignInDocumentDoesNotBreakPrompt() {
        var doc = new Document("Items discounted over 50% are not returnable.", Map.of("source", "return-policy.md"));

        assertThat(PromptBuilder.build("Sale items?", List.of(doc)))
                .contains("over 50% are");
    }

    @Test
    void returnsNotFoundWithoutCallingLlmWhenNoDocuments() {
        VectorStore vectorStore = mock(VectorStore.class);
        ChatClient chatClient = mock(ChatClient.class);
        when(vectorStore.similaritySearch(any(SearchRequest.class))).thenReturn(List.of());

        String answer = new RagService(vectorStore, chatClient).answer("Do you sell laptops?");

        assertThat(answer).isEqualTo(RagService.NOT_FOUND);
        verifyNoInteractions(chatClient);
    }
}

The third test verifies two things. First, the answer is the correct refusal. Second, ChatClient is never called at all. If someone accidentally removes the isEmpty() check later, the test will fail.

To evaluate retrieval quality, you can write a separate integration test. Prepare a set of questions with their expected source filenames, then check whether that file appears in the top K results. This test suite should run periodically, not on every build.

8. Simplify with QuestionAnswerAdvisor

Once you understand each step, you can use Spring AI’s built-in advisor. This advisor encapsulates retrieval and prompt building. First, add the dependency:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-advisors-vector-store</artifactId>
</dependency>
String answer = chatClient.prompt()
        .advisors(QuestionAnswerAdvisor.builder(vectorStore)
                .searchRequest(SearchRequest.builder().topK(4).similarityThreshold(0.5).build())
                .build())
        .user("How long can I return something?")
        .call()
        .content();

The advisor automatically embeds the question, finds chunks, and inserts them into the prompt using a default template. This is convenient, but you have less control over source formatting and early refusal. So the manual approach in section 6 is still valuable when you need those requirements.

9. Common mistakes and when not to use RAG

Switching embedding models without re-ingesting data. Vectors from different models are not comparable. Each time you switch models, you must re-embed all documents.

Not setting similarityThreshold. The vector store always returns topK results. The LLM then receives garbage context and answers confidently but incorrectly.

Re-ingesting documents on every startup with a persistent store. With PGvector, the ApplicationRunner above will create duplicates on each deploy. You should assign a fixed ID to each chunk (e.g., hash of content and filename) or delete by source before re-ingesting.

Putting too many chunks in the prompt. Increasing topK to 20 doesn’t make answers better. LLMs tend to miss information buried in long context, and costs rise with token count.

Blindly trusting document content. If documents come from users, they might contain instruction-like passages (prompt injection). You need to control which data sources are allowed into the store.

Note: RAG isn’t suitable for every problem. Questions requiring aggregation across all data—like “how many orders were returned in September?”—should become SQL queries. Similarity search only returns a few chunks and can’t count or aggregate. If your entire knowledge base is just a few pages, feeding it straight into the prompt is often simpler and more accurate than building a whole RAG pipeline.

10. Conclusion

RAG decouples knowledge from the model. Documents live in the vector store, and the LLM reads only the relevant bits for each question. In this article, you’ve built both phases: ingestion (read, chunk, embed, store) and query (search with threshold, augment prompt with sources, call LLM). You also wrote tests for prompt construction and the refusal path.

From this foundation, you can expand in several directions: replace SimpleVectorStore with PGvector, chunk by heading, filter by metadata (e.g., only search in one department’s documents), or add a reranking step before feeding chunks into the prompt.

ragspring-aillmvector-storespring-boot
App by the authorAlterTurn photos into postage stamps, collected in a stamp bookiOS · Android

More on this topic

RAG · 10/05/2026 · 8 min read · 112 · video

From Found Passages to Grounded Answers in RAG

Finding relevant passages doesn't mean you have an answer. This guide covers rerank pipelines, hybrid search, and grounding so LLMs can answer with evidence and gracefully refuse when data is insufficient.