Running LLMs Locally: When to Self-Host Models

1. Overview

Calling an LLM provider’s API in the cloud is the fastest way to add AI to your application. Yet more and more teams want to run models on their own infrastructure—whether that’s a laptop, an internal server, or rented GPUs.

In this article, we’ll explore what “running LLMs locally” actually means and why this trend matters. Then you’ll set up a local model, call it from a Spring Boot application, and write tests for the integration. At the end, we’ll discuss the criteria for deciding when to self-host and when not to.

2. What Does “Running LLMs Locally” Mean?

2.1. Three Core Components

A self-hosted LLM system typically has three layers:

Model weights: the weight files of the model. You typically download open-weight models from public repositories.
Inference runtime: software that loads weights into RAM or VRAM and generates tokens. Examples are Ollama, llama.cpp, or vLLM.
API layer: an HTTP endpoint for applications to call. Many runtimes provide an OpenAI-compatible API, so your client code barely needs to change.

Your application only sees the third layer. This is crucial because it lets you switch between local and cloud later without rewriting your business logic.

2.2. Quantization and Memory

The deciding factor for “can my hardware run this” is memory. You can estimate memory for weights roughly like this:

Weight memory ≈ number of parameters × bytes per parameter

16-bit (FP16): 2 bytes / parameter
8-bit:         1 byte / parameter
4-bit:       ~0.5 bytes / parameter

Beyond weights, you also need space for the KV cache. This grows with context length and the number of parallel requests. Quantization is a technique that reduces numerical precision—for example, from 16-bit down to 4-bit—to make the model much smaller. In return, answer quality drops a little. For many tasks like classification, summarization, or information extraction, this trade-off is usually acceptable.

3. Why Is Self-Hosting LLMs Getting Attention?

There are four main drivers:

Privacy and compliance: customer data, contracts, or source code never leaves your infrastructure.
Predictable costs: cloud APIs charge per token. With steady, high volume, a fixed GPU machine might be cheaper.
Latency and offline capability: no internet round-trip, no third-party rate limits, works in air-gapped environments.
Improving open-weight models: smaller models are now good enough for many specialized tasks. You can also fine-tune them for your domain.

Note: “Local” doesn’t have to mean your laptop. A GPU cluster in your own VPC still counts as self-hosting. The core point is you control the model and data.

4. Setting Up a Local Model with Ollama

Ollama is the simplest way to get started. It packages the runtime, manages models, and provides an HTTP API. You can run it with Docker Compose:

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    restart: unless-stopped

volumes:
  ollama-data:

The ollama-data volume persists downloaded models, so you don’t reload them every time the container restarts. Port 11434 is Ollama’s default API port.

Next, download a model and try it out:

docker compose up -d
docker compose exec ollama ollama pull llama3.2

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "Summary: order delivered 3 days late, customer wants a refund."}],
  "stream": false
}'

You can replace llama3.2 with any model that fits your hardware. With "stream": false, the result is a single JSON response like:

{
  "model": "llama3.2",
  "message": { "role": "assistant", "content": "Customer requests a refund because the order was delivered 3 days late." },
  "done": true,
  ...
}

If your machine has an NVIDIA GPU, install NVIDIA Container Toolkit and give the container GPU access. Otherwise, the model runs on CPU. This works but is much slower.

5. Calling a Local Model from Spring Boot

The example here is a real scenario: drafting customer support replies. Message data stays entirely within your internal infrastructure.

5.1. Dependencies and Configuration

You only need RestClient, which comes with Spring Web. No extra AI libraries needed:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

Declare the configuration in application.properties:

local-llm.base-url=http://localhost:11434
local-llm.model=llama3.2
local-llm.timeout=60s

Set timeout generously. A local model, especially on CPU, might take tens of seconds to generate a long response.

5.2. Properties and RestClient

@ConfigurationProperties(prefix = "local-llm")
public record LocalLlmProperties(String baseUrl, String model, Duration timeout) {}

@Configuration
@EnableConfigurationProperties(LocalLlmProperties.class)
public class LlmClientConfig {

    @Bean
    RestClient ollamaRestClient(RestClient.Builder builder, LocalLlmProperties props) {
        HttpClient httpClient = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(5))
                .build();
        JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(httpClient);
        factory.setReadTimeout(props.timeout());

        return builder
                .baseUrl(props.baseUrl())
                .requestFactory(factory)
                .build();
    }
}

The two timeout types here serve different purposes. Connect timeout is short (5 seconds) to quickly detect when Ollama isn’t running. Read timeout is long to wait for the model to finish generating.

5.3. DTOs and Service

public record ChatMessage(String role, String content) {}

public record ChatRequest(String model, List<ChatMessage> messages, boolean stream) {}

@JsonIgnoreProperties(ignoreUnknown = true)
public record ChatResponse(ChatMessage message, boolean done) {}

@Service
public class SupportReplyService {

    private static final String SYSTEM_PROMPT =
            "You are a customer support representative. Answer politely and concisely.";

    private final RestClient ollama;
    private final String model;

    public SupportReplyService(RestClient ollamaRestClient, LocalLlmProperties props) {
        this.ollama = ollamaRestClient;
        this.model = props.model();
    }

    public String draftReply(String customerMessage) {
        ChatRequest request = new ChatRequest(model, List.of(
                new ChatMessage("system", SYSTEM_PROMPT),
                new ChatMessage("user", customerMessage)
        ), false);

        ChatResponse response = ollama.post()
                .uri("/api/chat")
                .contentType(MediaType.APPLICATION_JSON)
                .body(request)
                .retrieve()
                .body(ChatResponse.class);

        if (response == null || response.message() == null) {
            throw new IllegalStateException("LLM did not return content");
        }
        return response.message().content().strip();
    }
}

The flow of draftReply works like this:

The service combines a system message to guide tone, then adds the customer’s message as a user message.
stream is set to false, so Ollama returns the complete response in one message.
@JsonIgnoreProperties ignores stats the service doesn’t use, like processing time or token count.
Empty responses become clear exceptions instead of cryptic NullPointerExceptions.

6. Integration Testing

Don’t make unit tests depend on a real model—that’s slow and unpredictable. Instead, use MockRestServiceServer to verify that the service sends the right request and parses the response correctly:

class SupportReplyServiceTest {

    private MockRestServiceServer server;
    private SupportReplyService service;

    @BeforeEach
    void setUp() {
        RestClient.Builder builder = RestClient.builder().baseUrl("http://localhost:11434");
        server = MockRestServiceServer.bindTo(builder).build();
        var props = new LocalLlmProperties("http://localhost:11434", "llama3.2", Duration.ofSeconds(60));
        service = new SupportReplyService(builder.build(), props);
    }

    @Test
    void givenCustomerMessage_whenDraftReply_thenReturnsAssistantContent() {
        server.expect(requestTo("http://localhost:11434/api/chat"))
                .andExpect(method(HttpMethod.POST))
                .andExpect(jsonPath("$.model").value("llama3.2"))
                .andExpect(jsonPath("$.stream").value(false))
                .andExpect(jsonPath("$.messages[1].content").value("Where is my order?"))
                .andRespond(withSuccess("""
                        {"model":"llama3.2",
                         "message":{"role":"assistant","content":"  We are checking on your order.  "},
                         "done":true,
                         "total_duration":123456}
                        """, MediaType.APPLICATION_JSON));

        String reply = service.draftReply("Where is my order?");

        assertThat(reply).isEqualTo("We are checking on your order.");
        server.verify();
    }
}

This test verifies four things:

The request goes to the right endpoint /api/chat using POST.
The body has the right model name and stream=false.
The customer message is in the second message.
The service ignores the unknown total_duration field and trims whitespace from the reply.

To verify answer quality, create a separate test suite that runs against a real model, using Testcontainers for example. This suite should run on a schedule, not in every build.

7. Keeping the Flexibility to Switch Between Local and Cloud

Ollama also supports OpenAI-compatible endpoints at /v1/chat/completions. If you build your client to this standard, switching between local and cloud is just a config change per profile:

# application-local.properties
llm.base-url=http://localhost:11434/v1
llm.model=llama3.2

# application-cloud.properties
llm.base-url=https://api.your-provider.example/v1
llm.model=your-cloud-model
llm.api-key=${LLM_API_KEY}

Put LLM calls behind an interface like ReplyDrafter. That way you can use local models for sensitive data and cloud models for quality-critical tasks, without your business logic knowing where the model is.

8. When to and When Not to Self-Host

8.1. When to Self-Host

You have sensitive data or regulations forbid sending data to third parties.
High, steady volume makes per-token costs higher than owning a GPU, even including operational effort.
Narrow, repetitive tasks like ticket classification, field extraction from invoices, or log summarization. Smaller models often excel at these.
You need offline capability, in air-gapped environments, or at the edge.
You need heavy fine-tuning or full control over model versions to keep results stable over time.

8.2. When Not to Self-Host

You need the best reasoning for complex tasks. Large cloud models still outperform here.
Low or bursty volume. Idle GPUs most of the time will almost certainly cost more than pay-per-token.
Your team lacks experience managing GPUs, monitoring, auto-scaling, and patching runtime security.
You’re still validating an idea. Prove value with cloud APIs first, then optimize costs.

A simple cost comparison:

Self-hosting cost / month = hardware (depreciation or rental) + power/hosting + operations hours
Cloud cost / month         = tokens per month × price per token

Only self-host if it's clearly cheaper at your real load, not at hoped-for load.

8.3. Common Mistakes

Warning: Ollama listens only on localhost by default and has no authentication. If you expose the port to the network, put it behind an authenticated reverse proxy. Never expose it directly to the internet.

Gut-feel evaluation: trying a few prompts and declaring it ready for production. You need a fixed benchmark dataset for your actual task.
Forgetting KV cache: the model fits in VRAM but runs out of memory with long context or parallel requests.
Timeout too short: the first call after startup waits for the model to load, which is slow. Warm up the model when your app starts.
Wrong runtime for production: Ollama is convenient for dev and moderate load. For many concurrent users, consider throughput-optimized runtimes like vLLM.

9. Conclusion

Self-hosting LLMs isn’t “cheaper” or “better” by default. It’s a trade-off: you get data control, predictable costs, and low latency, but you take on operational responsibility and usually accept smaller models.

In this article, you’ve set up Ollama with Docker and called it from Spring Boot using RestClient. You’ve written integration tests without a real model and learned how to design for easy local-to-cloud switching via config. The most practical approach is often using both: local models for sensitive data and repetitive tasks, cloud for what needs peak capability.