Function Calling and Tool Use with Spring AI

1. Overview

An LLM only knows what’s in its training data. It has no idea where your order ORD-1001 is. Function calling (also called tool use) solves exactly this problem. You describe some functions to the model. The model decides when to call which function and with what arguments. Your application executes the function and returns the result to the model.

This article walks through the mechanism at the JSON level first, then builds a customer support assistant with Spring AI. The assistant can look up order status and cancel orders. By the end, you’ll know how to declare tools, pass context safely, handle errors, and write tests without calling the actual model.

2. How Function Calling Works

2.1. The Ask-Call-Respond Loop

The most common misunderstanding: the model doesn’t run your code. It only returns a request to call a function, as structured JSON. A conversation turn with tools goes like this:

1. The application sends the user’s question, along with a list of tools (name, description, JSON Schema of parameters).
2. The model returns tool_calls instead of a final answer.
3. The application runs the corresponding function and sends the result back as a message with role tool.
4. The model reads the result. It may call additional tools, or write a final answer.

2.2. Looking at Raw JSON

The format below follows OpenAI’s style. Other providers use similar structures, just different field names. The request sent up has a tools section:

{
  "model": "gpt-4o-mini",
  "messages": [
    { "role": "user", "content": "Where is my order ORD-1001?" }
  ],
  "tools": [{
    "type": "function",
    "function": {
      "name": "getOrderStatus",
      "description": "Look up the status and estimated delivery date of an order",
      "parameters": {
        "type": "object",
        "properties": { "orderId": { "type": "string" } },
        "required": ["orderId"]
      }
    }
  }]
}

The model doesn’t answer right away. It requests a function call:

{
  "role": "assistant",
  "tool_calls": [{
    "id": "call_abc123",
    "type": "function",
    "function": { "name": "getOrderStatus", "arguments": "{\"orderId\":\"ORD-1001\"}" }
  }]
}

The application runs the function and sends back the result, with the correct tool_call_id:

{ "role": "tool", "tool_call_id": "call_abc123",
  "content": "{\"orderId\":\"ORD-1001\",\"status\":\"SHIPPED\",\"eta\":\"2026-10-30\"}" }

Now the model can write a final answer, such as: “Order ORD-1001 has been handed to the carrier and should arrive on Oct 30.”

Two things to remember. First, arguments is a string of JSON generated by the model. This string can have wrong types or missing fields. Second, the quality of the description determines whether the model chooses the right tool.

Writing this loop yourself isn’t hard, but it’s verbose. Spring AI does it for you.

3. Setting Up Spring AI

Add the BOM and starter for OpenAI. If using a different provider, just swap the starter; the tool code stays the same.

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>1.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>
</dependencies>

Configure the API key and model:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o-mini
spring.ai.openai.chat.options.temperature=0.2

Low temperature helps the model choose tools more consistently. For lookup tasks, you don’t need “creativity”.

4. Declaring Tools with @Tool

We’ll use an in-memory order repository for simplicity:

public record Order(String id, String customerId, String status, String eta) {}

public record OrderStatusReply(String orderId, String status, String eta) {}

A tool class is a normal bean. Each method marked with @Tool becomes a tool:

@Component
public class OrderTools {

    private final Map<String, Order> orders = new ConcurrentHashMap<>(Map.of(
        "ORD-1001", new Order("ORD-1001", "CUS-7", "SHIPPED", "2026-10-30"),
        "ORD-1002", new Order("ORD-1002", "CUS-7", "PENDING", "2026-11-02")
    ));

    @Tool(description = "Look up the status and estimated delivery date of an order")
    public OrderStatusReply getOrderStatus(
            @ToolParam(description = "Order ID, format ORD-xxxx") String orderId,
            ToolContext context) {
        Order order = findOwnedOrder(orderId, context);
        return new OrderStatusReply(order.id(), order.status(), order.eta());
    }

    @Tool(description = "Cancel an order that hasn't been handed to the carrier yet. Only call when the customer has explicitly confirmed they want to cancel")
    public String cancelOrder(
            @ToolParam(description = "Order ID, format ORD-xxxx") String orderId,
            @ToolParam(description = "Cancellation reason provided by the customer", required = false) String reason,
            ToolContext context) {
        Order order = findOwnedOrder(orderId, context);
        if (!"PENDING".equals(order.status())) {
            throw new IllegalStateException(
                "Order " + orderId + " is in status " + order.status() + " and cannot be cancelled");
        }
        orders.put(orderId, new Order(order.id(), order.customerId(), "CANCELLED", null));
        return "Cancelled order " + orderId;
    }

    private Order findOwnedOrder(String orderId, ToolContext context) {
        String customerId = (String) context.getContext().get("customerId");
        Order order = orders.get(orderId);
        if (order == null || !order.customerId().equals(customerId)) {
            throw new IllegalArgumentException("Order " + orderId + " not found");
        }
        return order;
    }
}

A few points worth noting:

• @Tool(description = …) is exactly the description from section 2.2. By default, the tool name is the method name.
• @ToolParam describes each parameter. required = false removes the parameter from the required list in the schema.
• A parameter of type ToolContext does not appear in the schema sent to the model. The model can’t see it or fill it. We use it to pass customerId from the login session. This way the model can’t “make up” someone else’s customer ID.
• The return type OrderStatusReply is serialized to JSON by Spring AI before sending back to the model.
• For orders not belonging to the current customer, we return “not found” instead of “no permission”. This prevents leaking that the order exists.

5. Calling Tools via ChatClient

@Service
public class SupportAssistant {

    private final ChatClient chatClient;
    private final OrderTools orderTools;

    public SupportAssistant(ChatClient.Builder builder, OrderTools orderTools) {
        this.chatClient = builder
            .defaultSystem("""
                You are a customer support assistant for our store.
                Only answer based on data retrieved from tools, do not guess order status.
                Before cancelling an order, ask the customer to confirm.""")
            .build();
        this.orderTools = orderTools;
    }

    public String answer(String customerId, String question) {
        return chatClient.prompt()
            .user(question)
            .tools(orderTools)
            .toolContext(Map.of("customerId", customerId))
            .call()
            .content();
    }
}

.tools(orderTools) scans every @Tool method in the bean and sends their definitions to the model. When the model returns tool_calls, Spring AI automatically does three things: parse the arguments, call the right method, then send the result back. This loop repeats until the model responds with text. .content() returns only the final answer.

Try it with a simple controller:

@RestController
@RequestMapping("/support")
public class SupportController {

    private final SupportAssistant assistant;

    public SupportController(SupportAssistant assistant) {
        this.assistant = assistant;
    }

    @PostMapping
    public String ask(@RequestBody String question, Principal principal) {
        return assistant.answer(principal.getName(), question);
    }
}

When customer CUS-7 asks “Where is my order ORD-1001?”, you get a result like:

Your order ORD-1001 has been handed to the carrier and should arrive on Oct 30, 2026.

The exact wording will differ between runs. The data always comes from the tool.

6. Looking Inside: ToolCallback and Schema

Spring AI converts each @Tool method into a ToolCallback. You can create them yourself to see what the model actually receives:

ToolCallback[] callbacks = ToolCallbacks.from(new OrderTools());
for (ToolCallback cb : callbacks) {
    System.out.println(cb.getToolDefinition().name());
    System.out.println(cb.getToolDefinition().inputSchema());
}

Output (abbreviated):

cancelOrder
{"type":"object","properties":{"orderId":{"type":"string","description":"Order ID, format ORD-xxxx"},
 "reason":{"type":"string","description":"Cancellation reason provided by the customer"}},"required":["orderId"],...}
getOrderStatus
{"type":"object","properties":{"orderId":{"type":"string","description":"Order ID, format ORD-xxxx"}},
 "required":["orderId"],...}

As mentioned in section 4, ToolContext doesn’t appear in the schema. When the model picks the wrong tool, print the schema like above to check first.

7. Testing Without Calling the Model

Calling the real model in unit tests is slow, costs money, and gives unstable results. Instead, test two things: the schema that the model receives and the behavior of the tool when it gets an arguments string exactly like the model would send.

class OrderToolsTest {

    private final ToolCallback[] callbacks = ToolCallbacks.from(new OrderTools());
    private final ToolContext customer7 = new ToolContext(Map.of("customerId", "CUS-7"));

    private ToolCallback tool(String name) {
        return Arrays.stream(callbacks)
            .filter(cb -> cb.getToolDefinition().name().equals(name))
            .findFirst().orElseThrow();
    }

    @Test
    void schemaShouldNotExposeToolContext() {
        String schema = tool("getOrderStatus").getToolDefinition().inputSchema();

        assertThat(schema).contains("\"orderId\"").doesNotContain("context");
    }

    @Test
    void shouldReturnStatusAsJson() {
        String result = tool("getOrderStatus")
            .call("{\"orderId\":\"ORD-1001\"}", customer7);

        assertThat(result).contains("\"status\":\"SHIPPED\"", "\"eta\":\"2026-10-30\"");
    }

    @Test
    void shouldHideOrdersOfOtherCustomers() {
        ToolContext stranger = new ToolContext(Map.of("customerId", "CUS-99"));

        assertThatThrownBy(() -> tool("getOrderStatus").call("{\"orderId\":\"ORD-1001\"}", stranger))
            .hasRootCauseInstanceOf(IllegalArgumentException.class)
            .hasRootCauseMessage("Order ORD-1001 not found");
    }

    @Test
    void shouldRefuseToCancelShippedOrder() {
        assertThatThrownBy(() -> tool("cancelOrder").call("{\"orderId\":\"ORD-1001\"}", customer7))
            .isInstanceOf(ToolExecutionException.class)
            .hasRootCauseMessage("Order ORD-1001 is in status SHIPPED and cannot be cancelled");
    }

    @Test
    void shouldCancelPendingOrder() {
        String result = tool("cancelOrder")
            .call("{\"orderId\":\"ORD-1002\",\"reason\":\"Ordered by mistake\"}", customer7);

        assertThat(result).contains("Cancelled order ORD-1002");
    }
}

MethodToolCallback wraps any exception from your method in ToolExecutionException. So these tests check the root cause instead of the original exception type. This test suite runs in milliseconds and checks exactly the part you control.

Model behavior (does it pick the right tool, does it ask for confirmation before cancelling) is a different kind of test. Run it separately with the real model, not in unit tests.

8. Handling Errors from Tools

What happens when cancelOrder throws an exception during a real run? By default (spring.ai.tools.throw-exception-on-error=false), Spring AI does not fail the request. It takes the exception message and sends it back to the model as a tool result. The model reads the message and explains to the customer:

I'm sorry, order ORD-1001 has already been handed to the carrier so it can't be cancelled.
Would you like me to guide you through the return process when you receive it?

This behavior is very convenient, but comes with two consequences you should know:

• The exception message will be read by the model and usually relayed to the user. Don’t throw exceptions with stack traces, SQL, or internal information.
• If you want infrastructure errors (database down, for example) to bubble up, set:

spring.ai.tools.throw-exception-on-error=true

A reasonable approach: business logic errors (order already shipped, order not found) go to the model, while infrastructure errors bubble up for monitoring to catch.

9. Common Mistakes and When Not to Use Function Calling

Warning: Treat the arguments generated by the model as untrusted user input. Prompt injection can make the model call cancelOrder with any order ID. All permission checks must be in your tool code, based on ToolContext or security context, not just instructions in the system prompt.

Common mistakes:

• Vague descriptions. Writing “Get order” is hard for the model to distinguish from “Get order history”. Describe clearly what the tool does, when to use it, and when not to.
• Too many tools in one request. Each tool definition costs tokens, and the model chooses worse with a long list. Only pass tools relevant to the current use case.
• Tools with side effects and no confirmation step. For operations like cancelling orders, refunds, or sending emails, consider splitting into two steps. The first tool just prepares and returns a summary. The real operation runs only after the user clicks confirm on the UI, not just because the model “thinks” the customer agreed.
• Lack of idempotency. The model might call the same tool twice. Design cancelOrder so the second call doesn’t cause harm.
• Returning too much data. Dumping 500 lines of order history into context wastes tokens and confuses the model. Filter and summarize in your tool.
• Forgetting that each tool call is another round-trip to the model. A question needing three tools might take a few seconds. Measure latency before putting it in a flow that needs fast responses.

When not to use function calling:

• The processing flow is fixed and known in advance (always look up the order first, then answer). Call your service directly instead, then put the result in the prompt. This is cheaper, faster, and more predictable.
• You just need the model to return structured data. Use structured output (e.g. .call().entity(MyRecord.class)) instead of simulating it with a tool.
• Sensitive, irreversible operations (transferring money, deleting data) without a human confirmation step.

10. Conclusion

Function calling is not magic. The model only returns a JSON request to call a function; your application actually executes it. Spring AI makes execution simpler: @Tool and @ToolParam generate the schema, ChatClient.tools(...) handles the call-response loop, and ToolContext passes session data that the model can’t access.

The most important thing is drawing the trust boundary correctly. Write clear descriptions and check permissions right in the tool. For operations with side effects, require user confirmation. Finally, test tools via ToolCallback with real JSON strings to know for sure your code works, regardless of what the model says.