Embeddings for Semantic Search: Detailed Guide with Examples

1. Overview

When you ask a chatbot “My flight was cancelled, will I get my hotel refund?”, the RAG system needs to find the exact policy section about “trip disruption” — even though your question doesn’t contain any words that exactly match the original text. This article explains the embedding mechanism that makes this possible: turning text into vectors, matching by semantic distance, and storing them so the LLM can still trace back to the original text. After reading, you’ll be able to write a simple semantic search pipeline in Java yourself, complete with tests proving it works correctly.

2. What is Embedding

An embedding model takes a piece of text and returns a vector — a sequence of real numbers, typically a few hundred to a few thousand dimensions. This vector isn’t “compressed keywords” but rather a point in high-dimensional space, where text segments with similar meanings will have vectors that lie close to each other.

public record EmbeddingVector(float[] values) {
    public int dimension() {
        return values.length;
    }
}

EmbeddingVector is just a wrapper around float[], but having it helps you avoid passing raw float[] arrays around and mixing them up between different models — a very easy mistake to make as your system grows.

2.1. Why You Shouldn’t Think of Embeddings as a Clearly Labeled “Map”

On a real map, you have longitude and latitude axes with clear labels like “Hanoi”, “Da Nang”. With embeddings, each individual dimension doesn’t carry a label like “hotel”, “flight”, “refund”. Meaning only emerges through the collective pattern across the entire vector. That’s why you can’t “read” a single dimension to know what it represents — you can only compare one full vector against another.

This matters because it explains why you shouldn’t arbitrarily trim a vector (for example, keeping only 100 of 1536 dimensions to save memory) — doing so breaks the pattern and search accuracy drops sharply.

3. Matching by Semantic Distance

Two vectors that are “close” to each other are usually measured by cosine similarity — the closer to 1, the more similar in meaning.

public final class VectorMath {

    public static double cosineSimilarity(float[] a, float[] b) {
        if (a.length != b.length) {
            throw new IllegalArgumentException(
                "Vector dimension mismatch: %d vs %d".formatted(a.length, b.length));
        }
        double dot = 0, normA = 0, normB = 0;
        for (int i = 0; i < a.length; i++) {
            dot += a[i] * b[i];
            normA += a[i] * a[i];
            normB += b[i] * b[i];
        }
        return dot / (Math.sqrt(normA) * Math.sqrt(normB));
    }
}

• Line if (a.length != b.length): catches immediately the common mistake — comparing vectors from two different models (e.g., one 768-dimensional, one 1536-dimensional) causes hard-to-debug runtime errors if not checked early.
• dot, normA, normB: compute the dot product and magnitude of each vector according to the standard cosine formula.

Note: two models with the same number of dimensions doesn’t mean they’re compatible. Model A and Model B both output 1536-dimensional vectors, but their semantic spaces are different — cross-comparing them will give meaningless results even though the code runs without error.

4. Simulating an Embedding Model for Practice Without Needing Real API Calls

To make the example runnable immediately, I use a fake EmbeddingClient based on keyword frequency across “topics” (travel, password, billing). In production you’d replace it with OpenAI, Cohere, or an open-source model, but the interface stays the same.

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

public interface EmbeddingClient {
    EmbeddingVector embed(String text);
}

public class KeywordEmbeddingClient implements EmbeddingClient {

    private static final List<String> TRAVEL_WORDS =
        List.of("flight", "cancelled", "hotel", "trip", "airport");
    private static final List<String> PASSWORD_WORDS =
        List.of("password", "login", "account", "reset", "forgot");
    private static final List<String> BILLING_WORDS =
        List.of("refund", "payment", "invoice", "charge", "fee");

    @Override
    public EmbeddingVector embed(String text) {
        String normalized = text.toLowerCase();
        float travel = score(normalized, TRAVEL_WORDS);
        float password = score(normalized, PASSWORD_WORDS);
        float billing = score(normalized, BILLING_WORDS);
        return new EmbeddingVector(new float[] { travel, password, billing });
    }

    private float score(String text, List<String> keywords) {
        long matches = keywords.stream().filter(text::contains).count();
        return matches / (float) keywords.size();
    }
}

KeywordEmbeddingClient uses only 3 dimensions (travel/password/billing) for easy observation — real embeddings use hundreds of dimensions learned from data, but the core idea “text with similar meaning → vectors close together” is identical.

5. Storing Embeddings with Metadata So the LLM Can Retrieve the Original Text

Vectors only help you locate the text; the LLM still needs the actual words to generate an answer. So each record in the vector store must link the embedding with the original chunk and metadata.

public record DocumentChunk(
    String chunkId,
    String documentId,
    String text,
    String section,
    LocalDate effectiveDate,
    int version,
    EmbeddingVector embedding
) {}

• chunkId and documentId: let you retrieve the full document if needed or update to a newer version.
• section, effectiveDate, version: help you filter before semantic matching — for example, search only in policies still in effect, skip outdated versions.
• text: must be stored alongside; the LLM can’t “read” a vector — it needs actual words to generate an answer.

public class InMemoryVectorStore {

    private final List<DocumentChunk> chunks = new ArrayList<>();

    public void add(DocumentChunk chunk) {
        chunks.add(chunk);
    }

    public List<DocumentChunk> findNearest(EmbeddingVector query, int topK) {
        return chunks.stream()
            .sorted(Comparator.comparingDouble(
                (DocumentChunk c) -> -VectorMath.cosineSimilarity(
                    query.values(), c.embedding().values())))
            .limit(topK)
            .toList();
    }
}

findNearest sorts chunks by cosine similarity in descending order (the minus sign reverses the comparator) and returns the topK nearest results — this is the “retrieval” step in RAG.

6. Complete Example: Different Wording But Correct Meaning

public class SemanticSearchDemo {

    public static void main(String[] args) {
        EmbeddingClient embedder = new KeywordEmbeddingClient();
        InMemoryVectorStore store = new InMemoryVectorStore();

        store.add(new DocumentChunk(
            "chunk-1", "policy-travel", "Refund policy when a flight is cancelled or trip is disrupted.",
            "Travel Disruption", LocalDate.of(2026, 1, 1), 2,
            embedder.embed("Refund policy when a flight is cancelled or trip is disrupted.")
        ));

        store.add(new DocumentChunk(
            "chunk-2", "policy-account", "Guide to resetting password when you forget your login account.",
            "Account Recovery", LocalDate.of(2025, 6, 1), 1,
            embedder.embed("Guide to resetting password when you forget your login account.")
        ));

        String question = "My flight was cancelled, will I get my hotel booking refunded?";
        EmbeddingVector queryVector = embedder.embed(question);

        List<DocumentChunk> results = store.findNearest(queryVector, 1);
        System.out.println(results.get(0).text());
    }
}

Running this program prints:

Refund policy when a flight is cancelled or trip is disrupted.

The question doesn’t contain the words “policy” or “trip”, but because the question and text both lie close in semantic space (high travel dimension for both), the system picks the right chunk — completely ignoring the password reset chunk.

7. Verification Through Tests

class SemanticSearchTest {

    private final EmbeddingClient embedder = new KeywordEmbeddingClient();
    private final InMemoryVectorStore store = new InMemoryVectorStore();

    @BeforeEach
    void setUp() {
        store.add(new DocumentChunk(
            "chunk-1", "policy-travel",
            "Refund policy when a flight is cancelled or trip is disrupted.",
            "Travel Disruption", LocalDate.of(2026, 1, 1), 2,
            embedder.embed("Refund policy when a flight is cancelled or trip is disrupted.")
        ));
        store.add(new DocumentChunk(
            "chunk-2", "policy-account",
            "Guide to resetting password when you forget your login account.",
            "Account Recovery", LocalDate.of(2025, 6, 1), 1,
            embedder.embed("Guide to resetting password when you forget your login account.")
        ));
    }

    @Test
    void shouldMatchTravelPolicyEvenWithoutSharedWords() {
        String question = "My flight was cancelled, will I get my hotel booking refunded?";
        EmbeddingVector queryVector = embedder.embed(question);

        List<DocumentChunk> results = store.findNearest(queryVector, 1);

        assertThat(results)
            .hasSize(1)
            .first()
            .extracting(DocumentChunk::documentId)
            .isEqualTo("policy-travel");
    }

    @Test
    void shouldRejectComparingDifferentDimensions() {
        float[] a = { 1f, 0f, 0f };
        float[] b = { 1f, 0f };

        assertThatThrownBy(() -> VectorMath.cosineSimilarity(a, b))
            .isInstanceOf(IllegalArgumentException.class)
            .hasMessageContaining("dimension mismatch");
    }
}

The first test asserts the core behavior: a question worded differently from the original text still finds the right chunk. The second test ensures you don’t accidentally compare two incompatible vectors — this mistake is easy to make when switching embedding models without re-embedding all old data.

8. Common Mistakes and When Not to Use

• Switching embedding models but not re-embedding old data: old and new vectors don’t lie in the same space, matching silently gives wrong results — no exception will tell you.
• Chunks too long or too short: long chunks blend many ideas together, reducing accuracy; very short chunks lose context. Usually 200–500 tokens is a good starting point, then tune based on your domain.
• Forgetting to store metadata: if you only save the vector without documentId/version, you can’t delete or update the right document when policies change.
• Using embeddings for data requiring exact matching: if you need to find exactly order code ORD-2026-00123, full-text search or exact key indexing still beats embeddings — embeddings excel at semantic search, not exact string search.

Note: with small datasets (a few hundred chunks), the InMemoryVectorStore above works fine. As you scale to millions of chunks, you’ll need a specialized vector database (pgvector, Pinecone, Milvus, etc.) with approximate nearest neighbor (ANN) indexing to avoid comparing every vector sequentially.

9. Conclusion

Embeddings turn text into vectors so machines can compare “meaning” instead of just matching keywords — that’s the foundation of every RAG system. The key points to remember: query and documents must use the same compatible encoder, each vector must come with metadata to retrieve the original text, and matching dimensionality doesn’t guarantee two models are compatible. Starting from the Java example above, you can replace KeywordEmbeddingClient with a real embedding model (via Spring AI or direct API) while keeping everything else — cosine similarity, vector store, metadata — unchanged.