Zero-Cost AI Stack: Detailed Explanation with Examples

1. Overview

Many people think building an AI system—agent, RAG, internal chatbot—costs thousands of dollars monthly because you have to pay API fees to OpenAI or Anthropic. That’s not the case: there’s a stack of 8 layers, each with a free or self-hosted version, enough for you to build and run a complete AI system for zero cost. This article goes through each layer of that stack, explains its role, with concrete code examples to help you understand how to connect them. After reading, you’ll know which tool to pick for each layer and understand where the real limits of “free” lie.

2. Why “$0” Does Not Mean Free Forever

Before diving into details, one thing needs to be clear: “$0” here means software/license cost, not unlimited infrastructure cost. Vercel’s free tier has bandwidth limits, Supabase’s free tier has storage limits, running Ollama locally consumes your machine’s CPU/RAM. This stack is suitable for prototyping, learning, demoing to small clients, not for scaling to millions of users.

Note: If your actual traffic exceeds the free tier, you’ll still have to pay—just for infrastructure (compute, storage) instead of paying the LLM vendor per token. This is an important distinction to understand before committing to this stack for real production.

3. Frontend Layer: Next.js or Streamlit

The interface layer doesn’t need to be complex. If you want beautiful UI, use Next.js deployed on Vercel’s free tier. If you just need a quick demo for an AI pipeline, Streamlit is much faster because you don’t need to write a separate frontend.

# app.py
import streamlit as st
import requests

st.title("Internal Document Q&A Assistant")

question = st.text_input("Enter your question")

if st.button("Ask") and question:
    response = requests.post(
        "http://localhost:8000/ask",
        json={"question": question},
        timeout=30,
    )
    st.write(response.json()["answer"])

This code creates an input field and a button. When you click the button, it calls the backend API (running locally) and displays the answer. Run with streamlit run app.py, and Streamlit will host on localhost:8501 automatically. No need to write React, no need to build pipelines—perfect for validating ideas before investing in Next.js.

4. Agent Orchestration Layer: LangGraph

When your system is not just “call LLM once and return result” but needs multiple steps (retrieve → synthesize → check → answer), you need an orchestrator. LangGraph models an agent as a stateful graph, allowing branching and loops.

<!-- pom.xml doesn't apply here because LangGraph is a Python library -->

pip install langgraph langchain-ollama

from langgraph.graph import StateGraph, END
from typing import TypedDict

class AgentState(TypedDict):
    question: str
    answer: str

def retrieve_step(state: AgentState) -> AgentState:
    state["answer"] = f"Searching for documents matching: {state['question']}"
    return state

def generate_step(state: AgentState) -> AgentState:
    state["answer"] += " -> synthesized answer"
    return state

graph = StateGraph(AgentState)
graph.add_node("retrieve", retrieve_step)
graph.add_node("generate", generate_step)
graph.set_entry_point("retrieve")
graph.add_edge("retrieve", "generate")
graph.add_edge("generate", END)

app = graph.compile()
result = app.invoke({"question": "What is the warranty policy?", "answer": ""})
print(result["answer"])

Searching for documents matching: What is the warranty policy? -> synthesized answer

StateGraph defines a processing flow with two nodes: retrieve and generate, each taking state and returning new state. set_entry_point specifies the starting node, add_edge connects the steps. When you call app.invoke(...), LangGraph runs sequentially through the graph and returns the final state. Everything runs locally, no paid API calls.

5. RAG Pipeline Layer: Chroma and Qdrant

RAG (Retrieval-Augmented Generation) needs a vector store to save embeddings and search by similarity. Chroma runs embedded, no separate server needed—suitable for prototyping.

pip install chromadb

import chromadb

client = chromadb.PersistentClient(path="./chroma_data")
collection = client.get_or_create_collection(name="policies")

collection.add(
    documents=["Product warranty is 12 months from purchase date.",
               "Exchange available within 7 days if box is intact."],
    ids=["doc1", "doc2"],
)

results = collection.query(query_texts=["warranty policy"], n_results=1)
print(results["documents"])

[['Product warranty is 12 months from purchase date.']]

PersistentClient saves data to disk instead of just RAM, so data persists across restarts. collection.add automatically computes embeddings for documents (Chroma has a default embedding function). collection.query finds the document closest in vector space to the question. For larger datasets, you can switch to Qdrant running locally via Docker—similar API but optimized for mid-scale.

6. LLM Layer: Ollama Running Models Locally

This is the core layer that helps you escape API bills. Ollama lets you run Llama, Mistral, Gemma right on your machine with no request limits.

ollama pull llama3.1
ollama run llama3.1 "Summarize the warranty policy in one sentence."

The product comes with a 12-month warranty from the purchase date.

The ollama pull command downloads the model to your machine (the model is quantized to run on typical CPUs/GPUs). The ollama run command starts the model and sends a prompt, returning the result directly in the terminal. Ollama also exposes a REST API at localhost:11434, so you can integrate it into Python or Java code.

If your backend is Spring Boot, calling Ollama via RestClient is straightforward:

import org.springframework.web.client.RestClient;

public class OllamaClient {

    private final RestClient restClient = RestClient.create("http://localhost:11434");

    public String generate(String model, String prompt) {
        var response = restClient.post()
                .uri("/api/generate")
                .body(new OllamaRequest(model, prompt, false))
                .retrieve()
                .body(OllamaResponse.class);
        return response.response();
    }

    record OllamaRequest(String model, String prompt, boolean stream) {}
    record OllamaResponse(String response) {}
}

OllamaClient wraps HTTP calls to Ollama. Spring 6’s RestClient replaces the old RestTemplate, with shorter syntax. OllamaRequest sets stream = false to receive the full response at once instead of streaming token-by-token—easier to handle in demo examples.

7. Tool Use via MCP (Model Context Protocol)

An agent that only talks to LLM can’t do much practically. MCP is a communication standard that helps agents call external tools—read files, query databases, call GitHub APIs—under a unified interface, without writing separate integrations for each tool.

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/data/docs"]
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"]
    }
  }
}

This config file declares two MCP servers: one for filesystem (read/write files in /data/docs), one for GitHub (create issues, read PRs). Your agent (LangGraph, Claude Code, or any MCP-compatible client) only needs to connect to these servers and call tools by standard schema, no need to write separate API wrappers for each service.

Note: Running MCP servers locally via npx doesn’t cost license fees, but some servers require access tokens (e.g., GitHub token)—you still need to get those tokens from the original service.

8. Code Agent: Aider

When you want an agent to write and edit code in your repo, Aider is an open-source CLI tool that works well with local models via Ollama.

pip install aider-chat
aider --model ollama/llama3.1 --file src/main/java/OrderService.java

This command opens Aider, specifies using model llama3.1 via Ollama (not via a paid API), and pre-loads the file OrderService.java into context. From there you type requests in Vietnamese or English (‘add validation for negative quantities’), Aider generates diffs and auto-commits if you approve.

9. Data Layer: SQLite, DuckDB, Supabase

Not every AI system needs a Postgres cluster. SQLite is enough for small structured storage, DuckDB is powerful for analyzing file-based data (CSV, Parquet) without needing a server.

import duckdb

con = duckdb.connect()
result = con.execute("""
    SELECT category, COUNT(*) AS total
    FROM read_csv_auto('orders.csv')
    GROUP BY category
    ORDER BY total DESC
""").fetchall()
print(result)

[('electronics', 120), ('books', 45), ('clothing', 30)]

DuckDB reads CSV files directly via read_csv_auto, no ETL or separate database import needed. SQL queries run nearly instantly on hundreds of thousands of rows—enough for most internal analytics use cases. When you need a backend with built-in REST API and auth, Supabase’s free tier provides managed Postgres with realtime and storage.

10. Deployment: Docker, Cloudflare Workers, Hugging Face

Containerizing with Docker helps you deploy consistently between your dev machine and free hosting.

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  app:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      - ollama

volumes:
  ollama_data:

This file defines two services: ollama runs the LLM model, app is your backend, dependent on ollama for inference calls. docker compose up starts both simultaneously, the ollama_data volume keeps the pulled model across container restarts. When you need a public demo, push the frontend to Cloudflare Workers (edge computing, generous free tier) and a small model to Hugging Face Spaces so others can test without configuring their machines.

11. Putting the Whole Stack Together

The real execution flow: user types a question on Streamlit → LangGraph receives the request, calls Chroma to retrieve relevant context → embeds context into prompt, sends to Ollama → if tools are needed (read files, query DB), agent calls via MCP → result returns to UI. Everything runs on one machine or container, no requests leave to paid APIs.

def answer_question(question: str) -> str:
    docs = collection.query(query_texts=[question], n_results=2)
    context = "\n".join(docs["documents"][0])
    prompt = f"Based on the following information:\n{context}\n\nAnswer: {question}"
    import ollama
    response = ollama.generate(model="llama3.1", prompt=prompt)
    return response["response"]

This function is the ‘heart’ of RAG: fetch context from Chroma, embed it in the prompt, call Ollama to generate the final answer. This is the most basic retrieve-then-generate pattern, no complex framework needed if your use case is simple.

12. Writing Tests for Data Processing

The easiest part to test in a RAG pipeline is usually the text chunking step before embedding. If your Java backend handles this step, a simple unit test helps avoid off-by-one errors when splitting.

import java.util.List;
import java.util.ArrayList;

public class DocumentChunker {

    public List<String> chunk(String text, int maxLength) {
        List<String> chunks = new ArrayList<>();
        for (int i = 0; i < text.length(); i += maxLength) {
            chunks.add(text.substring(i, Math.min(i + maxLength, text.length())));
        }
        return chunks;
    }
}

import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;

class DocumentChunkerTest {

    @Test
    void shouldSplitTextIntoFixedSizeChunks() {
        DocumentChunker chunker = new DocumentChunker();

        List<String> chunks = chunker.chunk("Product warranty is 12 months for all products", 10);

        assertThat(chunks).hasSizeGreaterThan(1);
        assertThat(chunks.get(0)).hasSize(10);
        assertThat(String.join("", chunks)).isEqualTo("Product warranty is 12 months for all products");
    }
}

This test asserts three things: long text is split into multiple chunks, the first chunk has the correct length, and joining all chunks back gives the exact original text—no characters lost. This is the kind of silent bug that easily happens when you write chunking logic without thorough testing.

13. Common Pitfalls and When Not to Use This Stack

• Local models are much slower than cloud APIs without GPU acceleration. On typical CPUs, a single llama3.1 8B request can take 5-15 seconds—acceptable for demos, not for products needing instant response.
• Free tiers have silent rate limits. Vercel, Supabase, Cloudflare all cap requests per month; exceeding the limit can silently kill your app if you don’t monitor the dashboard.
• Embedded Chroma is not suitable for multi-instance setups. If you scale your backend to multiple pods, each with its own Chroma file, you’ll get inconsistency—then you need Qdrant server or a managed vector store.
• Local MCP servers increase attack surface. A filesystem MCP server lets agents read/write files—if accidentally exposed to the internet, this is a serious security hole.

Note: If you’re building a product with SLA, need consistent low latency, or expect high traffic from day one, this zero-cost stack should only be used for prototyping. When going live for real, consider paying for managed LLM APIs or dedicated GPU instances to ensure service quality.

14. Conclusion

The zero-cost stack is not a marketing trick—it’s a real combination of open-source tools and free tiers, enough for you to build an AI system with RAG, agents, tool use, and complete deployment without needing a credit card. The real barrier is not money, but understanding how to connect each layer, knowing the limits of free tiers, and knowing when to switch to paid infrastructure for quality. Start with this stack to learn and validate ideas, then upgrade layer by layer as your product truly needs to scale.