EN
EspañolPortuguêsFrançaisEnglish한국어Tiếng ViệtDeutsch简体中文日本語
Home / AI & Machine Learning / RAG

From Found Passages to Grounded Answers in RAG

Cris The Dev · 10/05/2026 · 8 min read · 112
This article has a short video:YouTubeTikTok

1. Overview

Retrieval-Augmented Generation (RAG) often stops at the step of “find the top-k nearest passages” then throws them straight into the prompt. This approach easily causes hallucinations because vector search always returns something nearest, even when no passage actually answers the question. This article breaks down the next step: turning candidates into trustworthy evidence through hybrid search, reranking, and a refusal mechanism when data is insufficient. By the end, you’ll build a small Spring Boot pipeline that walks through all these steps, complete with tests to verify the behavior.

2. The Problem: Nearest Doesn’t Mean Right

Imagine you have a chatbot helping employees look up refund policies when flights are cancelled. Vector search will always return the 30 passages “nearest” to the question, even if your collection has no passage about cancellations at all. If the system feeds the top-3 passages straight to an LLM, the model will try to “make up” a plausible answer based on passages about hotel bookings instead.

Two problems need solving:

  1. Filtering: among dozens of candidates, which passages are actually useful for the question.
  2. Controlled refusal: when no passage is good enough, the system must say “not found” instead of guessing.

3. Overall Pipeline

Question
  → Vector search (semantic)         ─┐
  → Keyword search (BM25/exact match) ├─→ Merge (hybrid) → Rerank → Confidence threshold → Evidence → LLM

Each step narrows the quantity and raises the quality: from dozens of raw candidates down to a few passages with clear evidence and source citations.

4. Basic Data Model

public record Passage(
        String id,
        String text,
        String source,
        double score
) {
    public Passage withScore(double newScore) {
        return new Passage(id, text, source, newScore);
    }
}

Passage represents an indexed text chunk, with source for citations and score for comparison/sorting. withScore lets you create a copy with a new score after each rerank step without mutating the original object.

5. Hybrid Search: Combining Semantic and Keyword

Embeddings excel at capturing meaning (e.g., “flight cancelled” ↔ “travel disruption”), but easily miss exact tokens like policy codes POL-2024-07 or proper names. Keyword search (BM25) compensates for this weakness.

public interface VectorSearchClient {
    List<Passage> search(String query, int topK);
}

public interface KeywordSearchClient {
    List<Passage> search(String query, int topK);
}

@Service
public class HybridSearchService {

    private final VectorSearchClient vectorClient;
    private final KeywordSearchClient keywordClient;

    public HybridSearchService(VectorSearchClient vectorClient,
                                KeywordSearchClient keywordClient) {
        this.vectorClient = vectorClient;
        this.keywordClient = keywordClient;
    }

    public List<Passage> search(String query) {
        List<Passage> semantic = vectorClient.search(query, 20);
        List<Passage> keyword = keywordClient.search(query, 20);

        Map<String, Passage> merged = new LinkedHashMap<>();
        for (Passage p : semantic) {
            merged.put(p.id(), p);
        }
        for (Passage p : keyword) {
            merged.merge(p.id(), p, (existing, incoming) ->
                    existing.withScore(existing.score() + incoming.score()));
        }
        return new ArrayList<>(merged.values());
    }
}

merge sums the scores when a passage appears in both results — a strong signal that the passage is both semantically relevant and matches key exact terms. The result is a merged candidate list, not yet sorted by final usefulness.

6. Reranking: Filter with a Pairwise Comparison Model

Vector search scores each passage independently against the question (bi-encoder), so it doesn’t “read” deep relationships between them. A reranker (usually a cross-encoder) receives the question and passage together, scoring them more accurately but slower — so only apply it to a small pool of candidates already filtered at coarse granularity.

public interface Reranker {
    List<Passage> rerank(String query, List<Passage> candidates, int topN);
}

@Component
public class CrossEncoderReranker implements Reranker {

    private final CrossEncoderClient client; // call external rerank model

    public CrossEncoderReranker(CrossEncoderClient client) {
        this.client = client;
    }

    @Override
    public List<Passage> rerank(String query, List<Passage> candidates, int topN) {
        return candidates.stream()
                .map(p -> p.withScore(client.relevanceScore(query, p.text())))
                .sorted(Comparator.comparingDouble(Passage::score).reversed())
                .limit(topN)
                .toList();
    }
}

client.relevanceScore returns a true relevance score (e.g., 0–1) between the question and each passage. After sorting in descending order, keep only topN — usually 3 to 5, short enough to fit in a prompt without diluting context.

Note: Don’t rerank the entire corpus of thousands of passages — cross-encoders are tens to hundreds of times slower than vector search. Always limit your candidate input (20–50) before reranking.

7. Building Evidence with Citations for the LLM

After reranking, you have a few high-quality passages. Next, package them into clear, sourced context so the LLM can cite them in its answer instead of speaking without sources.

@Component
public class EvidenceBuilder {

    public String build(List<Passage> topPassages) {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < topPassages.size(); i++) {
            Passage p = topPassages.get(i);
            sb.append("[Source %d - %s]%n%s%n%n"
                    .formatted(i + 1, p.source(), p.text()));
        }
        return sb.toString();
    }
}

Each passage is numbered and tagged with its source (source). When spliced into the system prompt, you ask the LLM to answer only from these passages and cite the source number — this lets users verify the evidence instead of blindly trusting the answer.

[Source 1 - policy-2024-07.pdf]
Employees are reimbursed 100% of incurred hotel costs if the flight is cancelled due to weather...

[Source 2 - policy-2024-07.pdf]
Applicability: the flight must be cancelled; does not apply to delays under 6 hours...

8. Handling No-Answer Cases

This step is easiest to overlook. Every collection has “nearest neighbors” — even when no passage truly answers the question. Without checking, the LLM will use unrelated context to infer a plausible-sounding but wrong answer.

Simple solution: set a confidence threshold based on the highest rerank score.

@Service
public class AnswerService {

    private static final double CONFIDENCE_THRESHOLD = 0.55;

    private final HybridSearchService searchService;
    private final Reranker reranker;
    private final EvidenceBuilder evidenceBuilder;
    private final LlmClient llmClient;

    public AnswerService(HybridSearchService searchService,
                          Reranker reranker,
                          EvidenceBuilder evidenceBuilder,
                          LlmClient llmClient) {
        this.searchService = searchService;
        this.reranker = reranker;
        this.evidenceBuilder = evidenceBuilder;
        this.llmClient = llmClient;
    }

    public String answer(String question) {
        List<Passage> candidates = searchService.search(question);
        List<Passage> topPassages = reranker.rerank(question, candidates, 4);

        boolean hasEvidence = !topPassages.isEmpty()
                && topPassages.get(0).score() >= CONFIDENCE_THRESHOLD;

        if (!hasEvidence) {
            return "I didn't find a clear policy that answers this question. "
                    + "Please contact HR to confirm.";
        }

        String context = evidenceBuilder.build(topPassages);
        return llmClient.generate(question, context);
    }
}

If the best passage after reranking scores below CONFIDENCE_THRESHOLD, the system replies with refusal instead of calling the LLM to generate an unsupported answer. The specific threshold should be tuned with real data — too low causes hallucinations, too high refuses answerable questions.

9. Behavior Testing

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.assertj</groupId>
    <artifactId>assertj-core</artifactId>
    <scope>test</scope>
</dependency>
class AnswerServiceTest {

    @Test
    void shouldAbstainWhenBestScoreBelowThreshold() {
        HybridSearchService search = mock(HybridSearchService.class);
        Reranker reranker = mock(Reranker.class);
        EvidenceBuilder builder = new EvidenceBuilder();
        LlmClient llm = mock(LlmClient.class);

        Passage weakMatch = new Passage("p1", "General hotel booking policy",
                "hotel-faq.pdf", 0.3);

        when(search.search(anyString())).thenReturn(List.of(weakMatch));
        when(reranker.rerank(anyString(), anyList(), anyInt()))
                .thenReturn(List.of(weakMatch));

        AnswerService service = new AnswerService(search, reranker, builder, llm);
        String result = service.answer("How do I get reimbursed if my flight is cancelled?");

        assertThat(result).contains("didn't find");
        verifyNoInteractions(llm);
    }

    @Test
    void shouldAnswerWhenConfidenceIsHigh() {
        HybridSearchService search = mock(HybridSearchService.class);
        Reranker reranker = mock(Reranker.class);
        EvidenceBuilder builder = new EvidenceBuilder();
        LlmClient llm = mock(LlmClient.class);

        Passage strongMatch = new Passage("p2", "100% reimbursement if flight is cancelled",
                "policy-2024-07.pdf", 0.82);

        when(search.search(anyString())).thenReturn(List.of(strongMatch));
        when(reranker.rerank(anyString(), anyList(), anyInt()))
                .thenReturn(List.of(strongMatch));
        when(llm.generate(anyString(), anyString())).thenReturn("You are eligible for 100% reimbursement [Source 1].");

        AnswerService service = new AnswerService(search, reranker, builder, llm);
        String result = service.answer("How do I get reimbursed if my flight is cancelled?");

        assertThat(result).contains("Source 1");
        verify(llm).generate(anyString(), anyString());
    }
}

The first test checks refusal behavior when the score is low — llm is never called, saving cost and avoiding hallucination. The second confirms that with strong evidence, the system calls the LLM and the answer includes source citations.

10. Common Pitfalls

  • Skipping hybrid search, using only pure vector: easily misses exact matches on codes, proper names — things embeddings don’t distinguish well.
  • Reranking the entire dataset instead of a filtered subset: slow and wasteful.
  • No confidence threshold: always answer even when context is unrelated, causing hallucinations that users find hard to detect because the answer “sounds” confident.
  • Too-long context: stuffing 30 original passages into the prompt instead of 3–5 reranked ones dilutes the signal and inflates token cost.

Note: If your domain has stable vocabulary with few technical terms (e.g., casual chat), hybrid search may be redundant — pure vector search still works and is simpler to operate.

11. Conclusion

RAG only becomes trustworthy when retrieval stops at the right moment: merging semantic and keyword results, reranking to pick truly useful passages, and refusing when evidence is insufficient. The small pipeline in this article — HybridSearchService, Reranker, EvidenceBuilder, AnswerService — is a skeleton you can plug real models into (embeddings, BM25 index, cross-encoder) to get a cited, truthful assistant instead of a guessing machine.

ragllmspring-bootsearchjava
App by the authorGlowmi · Lantern NightA cozy Mid-Autumn match-3 puzzleiOS · Android

More on this topic