Tokenizer in Hugging Face: Principles and How to Use in Java

1. Overview

When working with NLP models on Hugging Face, the first step is always tokenization — converting raw text into a sequence of numbers that the model understands. This article explains how tokenizers work, common token splitting strategies (BPE, WordPiece, SentencePiece), and how to call HuggingFaceTokenizer directly from Java via the DJL library without needing to run Python. By the end, you’ll build a TokenizerService in Spring Boot to encode/decode text, control padding/truncation, and write tests for it.

2. What is a Tokenizer and Why It Matters

A tokenizer does two things: splits a sentence into tokens (which can be words, subwords, or characters), then maps each token to an integer ID in the model’s vocabulary. The model never “sees” text — it only sees this sequence of IDs.

For example, the sentence "Order service xử lý thanh toán" with the bert-base-uncased tokenizer might produce:

tokens: ["order", "service", "xu", "##ly", "thanh", "toan"]
ids:    [2344, 2326, 1045, 2135, 7899, 8821]

The notation ##ly means this token is concatenated directly to the previous token (subword). This is why the number of tokens is often more than the number of words — important when calculating API call costs or the max_length limit of a model.

Note: Each pretrained model comes with its own tokenizer, trained at the same time as the model on the same vocabulary. Using the wrong tokenizer (for example, BERT’s tokenizer for a GPT model) will produce meaningless IDs, causing the model to predict completely wrong without raising an error.

3. Common Tokenization Strategies

3.1. BPE (Byte-Pair Encoding)

Used in GPT-2, RoBERTa. Starts with characters, repeatedly merges the most frequently occurring character pairs into a new token, until reaching the desired vocabulary size.

3.2. WordPiece

Used in BERT. Similar to BPE but selects pairs to merge based on statistical likelihood rather than raw frequency. Subword tokens are marked with ## to indicate they’re concatenated to the previous word.

3.3. SentencePiece

Used in T5, XLNet, ALBERT. Treats whitespace as an ordinary character (represented as ▁), so it handles well for languages without clear word boundaries like Japanese and Chinese.

You don’t need to implement these algorithms yourself — Hugging Face has already packaged them in a tokenizer.json file that comes with each model; your job is just to load and call it.

4. Adding Dependencies

To use the tokenizer directly in Java (without going through a Python server), add the DJL Hugging Face Tokenizers library:

<dependency>
    <groupId>ai.djl.huggingface</groupId>
    <artifactId>tokenizers</artifactId>
    <version>0.28.0</version>
</dependency>

This library reads Hugging Face’s standard tokenizer.json file directly, without needing to run a complex JVM via JNI.

5. Basic Text Encoding

import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
import ai.djl.huggingface.tokenizers.Encoding;

import java.util.Arrays;

public class TokenizerDemo {

    public static void main(String[] args) throws Exception {
        HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-uncased");

        Encoding encoding = tokenizer.encode("Order service handles payment");

        System.out.println(Arrays.toString(encoding.getIds()));
        System.out.println(Arrays.toString(encoding.getTokens()));
    }
}

• HuggingFaceTokenizer.newInstance(...): loads the tokenizer from Hugging Face Hub (or local cache) by model name.
• encode(...): returns an Encoding object containing ids, tokens, and attentionMask.
• The first run downloads the tokenizer.json file to ~/.djl.ai/cache; subsequent runs use the cache, so no network is needed.

Sample output:

[101, 2344, 2326, 16024, 7909, 102]
[[CLS], order, service, handles, payment, [SEP]]

Note the two special tokens [CLS] (start of sentence) and [SEP] (end of sentence) are automatically inserted — this is a BERT convention, not an error.

6. Decoding Back

long[] ids = encoding.getIds();
String decoded = tokenizer.decode(ids);
System.out.println(decoded);

[CLS] order service handles payment [SEP]

decode is useful when you need to debug: see exactly what the model “sees” after tokenization, or when processing text generation output from a generative model.

7. Padding, Truncation, and Batching

When sending multiple sentences to a model at once, the sentences must have equal length (padding) and not exceed the model’s limit (truncation).

import java.util.Map;

HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance(
        "bert-base-uncased",
        Map.of(
            "padding", "true",
            "truncation", "true",
            "maxLength", "16"
        ));

Encoding[] batch = tokenizer.batchEncode(new String[] {
        "Order placed successfully",
        "Payment failed due to insufficient balance"
});

for (Encoding e : batch) {
    System.out.println(Arrays.toString(e.getIds()));
    System.out.println(Arrays.toString(e.getAttentionMask()));
}

• padding=true: shorter sentences are padded with [PAD] tokens (ID 0) to match the longest sentence in the batch.
• truncation=true + maxLength=16: sentences longer than 16 tokens are truncated.
• attentionMask: a sequence of 1s and 0s marking which tokens are real (1) or padding (0) — the model uses this mask to ignore the padding when calculating attention.

[101, 2344, 2872, 4652, 102, 0, 0, 0]
[1, 1, 1, 1, 1, 0, 0, 0]

Note: If you forget to set attentionMask when feeding data to the model, the padding portion could be included in attention calculations, causing incorrect output without any runtime error.

8. Integration with Spring Boot

Wrap the tokenizer as a shared bean to avoid reloading it with each request (loading the tokenizer from Hub/cache takes 100–300ms, which shouldn’t be done in the request thread).

import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer;
import org.springframework.stereotype.Service;

import java.io.IOException;
import java.util.Map;

@Service
public class TokenizerService {

    private final HuggingFaceTokenizer tokenizer;

    public TokenizerService() throws IOException {
        this.tokenizer = HuggingFaceTokenizer.newInstance(
                "bert-base-uncased",
                Map.of("padding", "true", "truncation", "true", "maxLength", "128"));
    }

    public long[] encode(String text) {
        return tokenizer.encode(text).getIds();
    }

    public String decode(long[] ids) {
        return tokenizer.decode(ids);
    }

    public int countTokens(String text) {
        return tokenizer.encode(text).getTokens().length;
    }
}

• The constructor initializes the tokenizer exactly once when the bean is created; Spring manages the singleton lifecycle.
• countTokens: useful when you need to check input length limits before calling the model (for example, Order description must not exceed 512 tokens).

Use it in another service:

@RestController
@RequestMapping("/tokens")
public class TokenController {

    private final TokenizerService tokenizerService;

    public TokenController(TokenizerService tokenizerService) {
        this.tokenizerService = tokenizerService;
    }

    @GetMapping("/count")
    public int count(@RequestParam String text) {
        return tokenizerService.countTokens(text);
    }
}

Calling GET /tokens/count?text=Order service xử lý thanh toán will return the number of tokens the BERT tokenizer creates for that sentence.

9. Writing Tests

import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;

class TokenizerServiceTest {

    @Test
    void shouldEncodeAndDecodeRoundTrip() throws Exception {
        TokenizerService service = new TokenizerService();

        String original = "payment failed";
        long[] ids = service.encode(original);
        String decoded = service.decode(ids);

        assertThat(ids).isNotEmpty();
        assertThat(decoded).contains("payment").contains("failed");
    }

    @Test
    void shouldCountMoreTokensThanWordsForSubwordSplit() throws Exception {
        TokenizerService service = new TokenizerService();

        int tokenCount = service.countTokens("unbelievable circumstances");

        // "unbelievable" is often split into multiple subwords
        assertThat(tokenCount).isGreaterThan(2);
    }
}

• Test 1 verifies round-trip encode → decode preserves the original content (case and punctuation may be lost due to the uncased tokenizer).
• Test 2 confirms subword behavior: long, uncommon words are split into more tokens than word count by whitespace.

10. Common Pitfalls and When Not to Use

• Mixing tokenizers between different model families: the tokenizer for bert-base-uncased is incompatible with gpt2. Always load the tokenizer matching the model name you use for inference.
• Not accounting for attention mask when padding: results in incorrect model output without an exception, very hard to debug. Always pass attentionMask along with inputIds to the model.
• Loading tokenizer with each request: wastes time and unnecessary I/O. Always declare a singleton bean like TokenizerService above.
• No network in production environment: the first time loading a tokenizer requires internet to fetch the file from Hugging Face Hub. For offline environments, you need to download tokenizer.json beforehand and load it from a local path using HuggingFaceTokenizer.newInstance(Path.of(...)).
• Counting tokens for non-Hugging Face models (e.g., GPT-4 via OpenAI API): GPT’s BPE tokenizer differs from Hugging Face’s standard tokenizer. In this case, use the provider’s own token counting library, not HuggingFaceTokenizer.
• Vietnamese text with diacritics: some tokenizers trained primarily on English may split Vietnamese words suboptimally (too many unfamiliar subwords). If Vietnamese processing is primary, choose a model/tokenizer fine-tuned for Vietnamese (e.g., vinai/phobert-base) instead of base BERT.

11. Conclusion

Tokenizers are the foundational step that determines how the model “sees” text — understanding the BPE/WordPiece/SentencePiece mechanisms helps you avoid hard-to-debug errors like vocabulary mismatches or incorrect attention masks. With the DJL library, you can encode/decode directly in Java without setting up a separate Python server, integrating cleanly into Spring Boot as a regular bean. Remember: always use the correct tokenizer for your model, cache the tokenizer as a singleton, and carefully check padding/truncation before feeding data to the model.