Implementing Tool Result Validation and Error Recovery in LLM Function Calling Pipelines

Function calling is one of the most powerful patterns in modern LLM development — and one of the most fragile in production. Your agent asks for a customer's order history, the API returns a 429, or the response schema has drifted since last quarter, and suddenly your otherwise-impressive pipeline is hallucinating refunds or stuck in a silent retry loop. You've seen the demos where everything works. This lesson is about what you do when it doesn't.

The core problem is that function calling pipelines sit at the intersection of two fundamentally unreliable systems: LLMs that can request tool calls with malformed arguments, and external services that return unexpected responses. Between them sits your validation and recovery layer — and in most production codebases, that layer is either absent or an afterthought bolted on after the first incident. By the end of this lesson, you'll know how to build it properly from the start.

We're going to work through a realistic scenario: an order management agent that can look up order status, apply discounts, and escalate issues to a support queue. Along the way you'll build a complete validation and error recovery system, including schema validation for tool arguments, result contract checking, typed error taxonomies, retry strategies with backoff, and graceful degradation patterns that keep your agent useful even when tools fail.

What you'll learn:

• How to validate LLM-generated tool call arguments before execution
• How to define and enforce result contracts for tool return values
• How to build a typed error taxonomy that drives different recovery behaviors
• How to implement retry logic, fallback strategies, and graceful degradation
• How to feed error context back to the model to enable self-correction

Prerequisites

You should be comfortable with the basics of function calling and tool use with LLMs, including how to define tool schemas, send tool results back in the message thread, and manage a basic agentic loop. Familiarity with Pydantic and Python's asyncio will also help, though the patterns here translate to any language.

Understanding the Failure Surface

Before you write a single line of validation code, you need a clear map of where things actually break. In a function calling pipeline, failures cluster into three distinct zones.

Zone 1: LLM argument generation. The model produces a tool call, but the arguments are wrong. The order_id field contains "ORD-" with no number. The discount_percent is 150. The required customer_email field is missing entirely. These aren't rare edge cases — they happen regularly, especially when the model is operating under a complex system prompt, the tool schema is ambiguous, or the user's request was underspecified.

Zone 2: Tool execution. The arguments were valid, but the tool itself failed. The downstream API is rate-limited, the database connection timed out, a third-party service returned a 500, or the record simply doesn't exist. This zone is almost entirely about external system reliability, not the LLM.

Zone 3: Result contract violation. The tool ran successfully from an infrastructure standpoint, but the result it returned doesn't match what the LLM expects to receive. A field changed its type from string to integer. A new required field appeared. The API started paginating a previously flat response. Your wrapper code has a bug that occasionally returns None instead of an empty list.

Key insight: Most error handling in function calling tutorials only addresses Zone 2 — the "tool threw an exception" case. Zones 1 and 3 are equally important and more insidious, because the pipeline often continues silently with bad data rather than raising an error.

Each zone calls for different tools: argument validation lives before execution, result contract checking lives after, and retry/recovery logic spans both. Let's build each layer.

Defining Your Tool Schemas and Contracts

Good validation starts with precise schemas. The tool definition you send to the LLM describes what arguments it should provide. The result contract you define separately describes what your code will return. These are related but distinct.

Here's our order management agent's tool suite, starting with clean schema definitions:

from pydantic import BaseModel, Field, field_validator, model_validator
from typing import Optional, Literal
from enum import Enum
import re

# --- Argument schemas (validate what the LLM sends) ---

class GetOrderStatusArgs(BaseModel):
    order_id: str = Field(
        ...,
        description="Order ID in format ORD-XXXXXXXX",
        pattern=r"^ORD-[A-Z0-9]{8}$"
    )

    @field_validator("order_id")
    @classmethod
    def normalize_order_id(cls, v: str) -> str:
        # Accept lowercase and normalize
        return v.upper().strip()


class ApplyDiscountArgs(BaseModel):
    order_id: str = Field(..., pattern=r"^ORD-[A-Z0-9]{8}$")
    discount_percent: float = Field(..., ge=0, le=100)
    reason: Literal["loyalty", "error_compensation", "promotional"] = Field(
        ...,
        description="Reason code for the discount"
    )

    @model_validator(mode="after")
    def cap_compensation_discount(self) -> "ApplyDiscountArgs":
        # Business rule: error compensation capped at 25%
        if self.reason == "error_compensation" and self.discount_percent > 25:
            raise ValueError(
                f"Error compensation discounts cannot exceed 25%. "
                f"Requested: {self.discount_percent}%"
            )
        return self


class EscalateIssueArgs(BaseModel):
    order_id: str = Field(..., pattern=r"^ORD-[A-Z0-9]{8}$")
    issue_summary: str = Field(..., min_length=10, max_length=500)
    priority: Literal["low", "medium", "high", "urgent"] = "medium"
    customer_email: Optional[str] = Field(
        None,
        pattern=r"^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$"
    )


# --- Result contracts (validate what your tools return) ---

class OrderStatus(str, Enum):
    PROCESSING = "processing"
    SHIPPED = "shipped"
    DELIVERED = "delivered"
    CANCELLED = "cancelled"
    REFUND_PENDING = "refund_pending"


class OrderStatusResult(BaseModel):
    order_id: str
    status: OrderStatus
    last_updated: str  # ISO 8601
    tracking_number: Optional[str] = None
    estimated_delivery: Optional[str] = None
    line_items: list[dict] = Field(default_factory=list)


class DiscountResult(BaseModel):
    success: bool
    order_id: str
    new_total: float = Field(..., ge=0)
    discount_applied: float = Field(..., ge=0)
    confirmation_code: str


class EscalationResult(BaseModel):
    ticket_id: str
    estimated_response_hours: int = Field(..., ge=1, le=72)
    assigned_team: str

Notice that the argument schemas encode business logic, not just data types. Pydantic's validators catch the "discount_percent=150" case and the "error_compensation at 40%" case before either of them ever reaches your actual discount service. This is cheap insurance.

Tip: Keep your argument schemas and result contracts in a separate tool_contracts.py module. Both your tool definitions (sent to the LLM) and your runtime validation code should derive from this single source of truth. Schema drift between what the LLM is told and what you actually validate is a common source of confusing bugs.

Building the Argument Validation Layer

Now let's wire these schemas into a validation layer that sits between the LLM response and your tool execution. The key design decision is: what do you do when argument validation fails? There are two options:

1. Raise an error and halt — appropriate for critical business logic violations
2. Return a structured error to the model — appropriate for recoverable misunderstandings

Option 2 is almost always better, because it lets the model self-correct. When you tell the model "the discount_percent must be between 0 and 100, you provided 150", it can immediately retry with the right value.

import json
from typing import Any, Type
from pydantic import BaseModel, ValidationError

# Map tool names to their argument schemas
TOOL_ARG_SCHEMAS: dict[str, Type[BaseModel]] = {
    "get_order_status": GetOrderStatusArgs,
    "apply_discount": ApplyDiscountArgs,
    "escalate_issue": EscalateIssueArgs,
}


class ToolValidationError(Exception):
    """Raised when tool arguments fail validation."""
    def __init__(self, tool_name: str, errors: list[dict], raw_args: dict):
        self.tool_name = tool_name
        self.errors = errors
        self.raw_args = raw_args
        super().__init__(f"Validation failed for {tool_name}: {errors}")


def validate_tool_args(tool_name: str, raw_args: dict) -> BaseModel:
    """
    Validate and parse tool arguments against their schema.
    Returns the parsed model on success, raises ToolValidationError on failure.
    """
    schema = TOOL_ARG_SCHEMAS.get(tool_name)
    if schema is None:
        raise ToolValidationError(
            tool_name,
            [{"type": "unknown_tool", "msg": f"No schema registered for tool '{tool_name}'"}],
            raw_args
        )

    try:
        return schema.model_validate(raw_args)
    except ValidationError as e:
        # Transform Pydantic's error format into something useful for the LLM
        errors = [
            {
                "field": ".".join(str(loc) for loc in err["loc"]),
                "message": err["msg"],
                "invalid_value": err.get("input"),
            }
            for err in e.errors()
        ]
        raise ToolValidationError(tool_name, errors, raw_args)


def format_validation_error_for_llm(error: ToolValidationError) -> str:
    """
    Format a validation error as a tool result message that helps the
    model understand what went wrong and how to fix it.
    """
    error_lines = []
    for err in error.errors:
        field = err.get("field", "unknown")
        msg = err.get("message", "Invalid value")
        val = err.get("invalid_value", "<not provided>")
        error_lines.append(f"  - Field '{field}': {msg} (received: {repr(val)})")

    return json.dumps({
        "status": "validation_error",
        "tool": error.tool_name,
        "message": "Your tool call arguments failed validation. Please correct them and try again.",
        "errors": error.errors,
        "hint": "\n".join(error_lines)
    })

Now plug this into the message loop. Here's how the integration looks in practice:

import openai

client = openai.OpenAI()


def run_tool_call(tool_name: str, raw_args: dict) -> str:
    """
    Central dispatcher for tool execution with full validation.
    Returns a JSON string suitable for use as a tool result message.
    """
    # Step 1: Validate arguments
    try:
        validated_args = validate_tool_args(tool_name, raw_args)
    except ToolValidationError as e:
        # Return structured error — don't raise, let the model self-correct
        return format_validation_error_for_llm(e)

    # Step 2: Execute the tool with validated args
    try:
        if tool_name == "get_order_status":
            result = get_order_status(validated_args)
        elif tool_name == "apply_discount":
            result = apply_discount(validated_args)
        elif tool_name == "escalate_issue":
            result = escalate_issue(validated_args)
        else:
            return json.dumps({"status": "error", "message": f"Unknown tool: {tool_name}"})

        # Step 3: Validate the result contract
        return validate_and_serialize_result(tool_name, result)

    except Exception as e:
        return handle_tool_execution_error(tool_name, e)

Classifying and Handling Execution Errors

When a tool actually runs and fails, not all failures are equal. A database timeout is temporary and should be retried. A "record not found" response is permanent and should be surfaced to the user. A permissions error indicates a configuration problem that no amount of retrying will fix.

Building a typed error taxonomy is the foundation of intelligent recovery. Here's a practical one for an order management domain:

from enum import Enum
from dataclasses import dataclass
from typing import Optional


class ToolErrorCategory(Enum):
    # Transient — retry is appropriate
    RATE_LIMITED = "rate_limited"
    TIMEOUT = "timeout"
    TEMPORARY_UNAVAILABLE = "temporary_unavailable"

    # Permanent — retry won't help
    NOT_FOUND = "not_found"
    PERMISSION_DENIED = "permission_denied"
    INVALID_STATE = "invalid_state"  # e.g., can't discount a cancelled order

    # Configuration — needs human intervention
    MISCONFIGURED = "misconfigured"
    SCHEMA_MISMATCH = "schema_mismatch"

    # Unknown
    UNEXPECTED = "unexpected"


@dataclass
class ToolError:
    category: ToolErrorCategory
    message: str
    retry_after_seconds: Optional[int] = None
    user_facing_message: Optional[str] = None
    technical_detail: Optional[str] = None

    @property
    def is_retryable(self) -> bool:
        return self.category in {
            ToolErrorCategory.RATE_LIMITED,
            ToolErrorCategory.TIMEOUT,
            ToolErrorCategory.TEMPORARY_UNAVAILABLE,
        }

    def to_llm_message(self) -> str:
        """Produce a message the LLM can act on."""
        base = {
            "status": "tool_error",
            "error_type": self.category.value,
            "message": self.message,
        }
        if self.retry_after_seconds and self.is_retryable:
            base["retry_guidance"] = (
                f"This is a temporary error. You may retry after "
                f"{self.retry_after_seconds} seconds."
            )
        if self.user_facing_message:
            base["user_message"] = self.user_facing_message
        return json.dumps(base)


def classify_exception(exc: Exception) -> ToolError:
    """
    Map raw exceptions from your tool implementations to typed ToolErrors.
    Extend this as you add new tools and discover new failure modes.
    """
    import httpx

    exc_str = str(exc).lower()

    if isinstance(exc, httpx.TimeoutException):
        return ToolError(
            category=ToolErrorCategory.TIMEOUT,
            message="The external service did not respond in time.",
            retry_after_seconds=5,
            user_facing_message="We're having trouble reaching our order system. Please try again."
        )

    if isinstance(exc, httpx.HTTPStatusError):
        status = exc.response.status_code
        if status == 429:
            retry_after = int(exc.response.headers.get("Retry-After", 10))
            return ToolError(
                category=ToolErrorCategory.RATE_LIMITED,
                message=f"Rate limited by upstream API.",
                retry_after_seconds=retry_after,
            )
        if status == 404:
            return ToolError(
                category=ToolErrorCategory.NOT_FOUND,
                message="The requested resource does not exist.",
                user_facing_message="We couldn't find that order. Please double-check the order ID."
            )
        if status == 403:
            return ToolError(
                category=ToolErrorCategory.PERMISSION_DENIED,
                message="Access denied by upstream API.",
                technical_detail=str(exc),
            )
        if status >= 500:
            return ToolError(
                category=ToolErrorCategory.TEMPORARY_UNAVAILABLE,
                message=f"Upstream service error (HTTP {status}).",
                retry_after_seconds=15,
            )

    if "cannot apply discount" in exc_str or "invalid state" in exc_str:
        return ToolError(
            category=ToolErrorCategory.INVALID_STATE,
            message=str(exc),
            user_facing_message=str(exc),
        )

    return ToolError(
        category=ToolErrorCategory.UNEXPECTED,
        message="An unexpected error occurred.",
        technical_detail=str(exc),
    )

Warning: Don't expose raw exception messages to the LLM — or through the LLM to users. Stack traces, database error messages, and internal API responses can contain sensitive information. Always filter through a layer like classify_exception that produces controlled, safe output.

Implementing Retry Logic with Exponential Backoff

For transient errors, you want retry logic, but it needs to be smarter than a simple loop. The retry should be transparent to the LLM for network-level transients (just retry the tool call), but visible to the LLM when retries are exhausted (so it can reason about what to tell the user).

import asyncio
import time
import logging
from functools import wraps
from typing import Callable, TypeVar, Awaitable

logger = logging.getLogger(__name__)

T = TypeVar("T")


async def with_retry(
    func: Callable[..., Awaitable[T]],
    *args,
    max_attempts: int = 3,
    base_delay: float = 1.0,
    max_delay: float = 30.0,
    jitter: bool = True,
    tool_name: str = "unknown",
    **kwargs,
) -> T:
    """
    Execute an async function with exponential backoff retry.
    Only retries on ToolErrors that are marked as retryable.
    """
    import random

    last_error: Optional[ToolError] = None

    for attempt in range(1, max_attempts + 1):
        try:
            return await func(*args, **kwargs)

        except Exception as exc:
            tool_error = classify_exception(exc)

            if not tool_error.is_retryable:
                logger.warning(
                    f"Non-retryable error in {tool_name} (attempt {attempt}): "
                    f"{tool_error.category.value} — {tool_error.message}"
                )
                raise  # Re-raise immediately, no point retrying

            last_error = tool_error

            if attempt == max_attempts:
                logger.error(
                    f"All {max_attempts} attempts failed for {tool_name}. "
                    f"Last error: {tool_error.message}"
                )
                break

            # Calculate delay with exponential backoff
            delay = min(base_delay * (2 ** (attempt - 1)), max_delay)

            # Honor Retry-After if provided by the upstream API
            if tool_error.retry_after_seconds:
                delay = max(delay, tool_error.retry_after_seconds)

            # Add jitter to avoid thundering herd problems
            if jitter:
                delay += random.uniform(0, delay * 0.2)

            logger.info(
                f"Retrying {tool_name} in {delay:.1f}s "
                f"(attempt {attempt}/{max_attempts})"
            )
            await asyncio.sleep(delay)

    # Exhausted retries — raise a final ToolError
    raise Exception(f"Tool {tool_name} failed after {max_attempts} attempts: {last_error.message}")


# Example: wrapping a real tool function
async def get_order_status_with_retry(args: GetOrderStatusArgs) -> OrderStatusResult:
    return await with_retry(
        _call_order_api,
        args.order_id,
        max_attempts=3,
        base_delay=1.0,
        tool_name="get_order_status"
    )

Notice the jitter. When you have many concurrent agents all hitting the same API and getting rate-limited at the same moment, a pure exponential backoff causes them all to retry at the same time — the "thundering herd" problem. Jitter spreads those retries out. This is especially relevant if you're orchestrating parallel LLM calls at scale.

Validating Tool Results Against Contracts

Zone 3 failures — where tools run successfully but return unexpected data — are subtle. Your apply_discount function might start returning discount_applied as a percentage string ("15%") instead of a float (0.15) after an upstream API change, and your agent will pass that directly to the LLM, which might handle it correctly sometimes and hallucinate the monetary impact other times.

TOOL_RESULT_SCHEMAS: dict[str, Type[BaseModel]] = {
    "get_order_status": OrderStatusResult,
    "apply_discount": DiscountResult,
    "escalate_issue": EscalationResult,
}


def validate_and_serialize_result(tool_name: str, raw_result: Any) -> str:
    """
    Validate a tool's return value against its result contract,
    then serialize it to JSON for the LLM.
    """
    schema = TOOL_RESULT_SCHEMAS.get(tool_name)

    if schema is None:
        # No contract defined — pass through with a warning
        logger.warning(f"No result contract for tool '{tool_name}' — skipping validation")
        return json.dumps(raw_result if isinstance(raw_result, dict) else {"result": str(raw_result)})

    try:
        # Coerce and validate — Pydantic will try type coercion before failing
        validated = schema.model_validate(
            raw_result if isinstance(raw_result, dict) else raw_result.__dict__
        )
        return validated.model_dump_json()

    except ValidationError as e:
        # The tool returned something that violates its contract
        # This is a serious issue — log it prominently and return a safe error
        logger.error(
            f"RESULT CONTRACT VIOLATION in {tool_name}: {e.errors()}. "
            f"Raw result: {raw_result}"
        )

        # Don't silently pass bad data to the LLM
        return json.dumps({
            "status": "internal_error",
            "message": (
                "The tool returned data in an unexpected format. "
                "This has been logged for investigation. "
                "Please inform the user that this operation could not be completed."
            )
        })

Note: Result contract violations should trigger alerts in your monitoring system. Unlike validation errors or transient network failures, they indicate a breaking change somewhere in your infrastructure — a new API version, a changed schema, or a bug in your tool implementation. Treat them like test failures: they need investigation, not just retries. This connects directly to your LLM observability setup.

Feeding Error Context Back to the Model

This is where the loop closes. You've validated arguments, classified errors, and validated results. Now you need to actually put the error information back into the conversation in a way that enables the model to recover intelligently.

The OpenAI function calling protocol requires you to send a tool role message with the result (even if that "result" is an error). Here's a complete agentic loop that ties everything together:

from openai.types.chat import ChatCompletionMessage, ChatCompletionToolParam
from typing import Optional
import json

TOOLS: list[ChatCompletionToolParam] = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Look up the current status of an order by its order ID.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "description": "Order ID in format ORD-XXXXXXXX (e.g., ORD-A1B2C3D4)"
                    }
                },
                "required": ["order_id"]
            }
        }
    },
    # ... (apply_discount and escalate_issue definitions follow same pattern)
]

MAX_TOOL_ITERATIONS = 8  # Prevent infinite loops
MAX_CONSECUTIVE_ERRORS = 3  # Bail out if the model keeps generating bad calls


async def run_agent(user_message: str) -> str:
    """
    Run the order management agent with full validation and error recovery.
    Returns the final text response to the user.
    """
    messages = [
        {
            "role": "system",
            "content": (
                "You are an order management assistant. You have access to tools to look up "
                "order status, apply discounts, and escalate issues. When a tool returns a "
                "validation error, correct your arguments and try again. When a tool returns "
                "a permanent error (not_found, invalid_state), explain the situation to the user "
                "instead of retrying. When a tool returns an internal_error, apologize and suggest "
                "the user contact support directly."
            )
        },
        {"role": "user", "content": user_message}
    ]

    consecutive_errors = 0

    for iteration in range(MAX_TOOL_ITERATIONS):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            tools=TOOLS,
            tool_choice="auto"
        )

        message = response.choices[0].message

        # No tool calls — the model has a final answer
        if not message.tool_calls:
            return message.content or ""

        # Append the assistant's message (with tool calls) to history
        messages.append(message)

        # Process each tool call in this response
        for tool_call in message.tool_calls:
            tool_name = tool_call.function.name
            call_id = tool_call.id

            # Parse the arguments (the LLM sends these as a JSON string)
            try:
                raw_args = json.loads(tool_call.function.arguments)
            except json.JSONDecodeError:
                # The model generated malformed JSON — extremely rare but possible
                result_content = json.dumps({
                    "status": "validation_error",
                    "message": "Your tool call contained malformed JSON arguments.",
                    "received": tool_call.function.arguments[:200]
                })
                consecutive_errors += 1
            else:
                # Full validation and execution pipeline
                result_content = await run_tool_call_async(tool_name, raw_args)

                # Track consecutive errors to detect stuck loops
                result_data = json.loads(result_content)
                if result_data.get("status") in ("validation_error", "tool_error", "internal_error"):
                    consecutive_errors += 1
                else:
                    consecutive_errors = 0

            # Append the tool result to the message history
            messages.append({
                "role": "tool",
                "tool_call_id": call_id,
                "content": result_content
            })

        # Safety valve: too many consecutive errors means something is fundamentally broken
        if consecutive_errors >= MAX_CONSECUTIVE_ERRORS:
            messages.append({
                "role": "user",
                "content": (
                    "[SYSTEM: Multiple consecutive tool errors detected. "
                    "Please stop attempting tool calls and respond to the user directly, "
                    "acknowledging that you're experiencing technical difficulties.]"
                )
            })
            # Force a final response without tools
            final_response = client.chat.completions.create(
                model="gpt-4o",
                messages=messages,
                tool_choice="none"
            )
            return final_response.choices[0].message.content or ""

    # Hit the iteration limit
    return (
        "I wasn't able to complete this request after multiple attempts. "
        "Please contact our support team directly for assistance."
    )

The consecutive error tracking is particularly important. Without it, a model that generates the wrong order_id format repeatedly will consume your entire token budget and return nothing useful. This kind of agentic loop design guardrail is what separates production systems from demos.

Warning: The MAX_TOOL_ITERATIONS limit isn't just about cost — it's a safety mechanism. A loop without a ceiling is a denial-of-service vulnerability in your own system. Set it conservatively and monitor how often real requests approach the limit; if they're regularly hitting 6-7 iterations, your tool schemas or system prompt likely need tuning.

Graceful Degradation Strategies

Sometimes you can't recover — the data just isn't there, or the service is down. Graceful degradation means your agent remains useful even when it can't do everything requested.

The key insight is that the model's system prompt should define explicit fallback behaviors for different error categories:

SYSTEM_PROMPT = """
You are an order management assistant with access to the following tools.

## Error Handling Guidelines

**validation_error**: Your tool call arguments were incorrect. Read the error message carefully, 
correct the specific field(s) mentioned, and try again with the corrected arguments. 
Do not ask the user for clarification unless the error indicates information is genuinely missing.

**not_found**: The requested resource doesn't exist. Do not retry. Tell the user the order 
couldn't be found and ask them to verify the order ID.

**invalid_state**: The operation is not permitted in the current state (e.g., discounting 
a cancelled order). Do not retry. Explain the constraint to the user clearly.

**rate_limited / timeout / temporary_unavailable**: These are temporary. If you receive one, 
wait a moment and try once more. If it fails again, acknowledge the service disruption 
and offer to help the user with what you can, or suggest they try again shortly.

**internal_error**: Something went wrong on our end. Do not retry. Apologize to the user 
and direct them to contact support at support@example.com or call 1-800-ORDERS-1.

**permission_denied / misconfigured**: These indicate a system problem. Do not retry. 
Apologize, explain you're unable to complete this action, and escalate via the 
escalate_issue tool if appropriate.

## Fallback Capabilities
Even when tools fail, you can:
- Explain order status terminology to customers
- Describe our return and refund policies from your training
- Provide support contact information
- Help draft messages for the customer to send to support
"""

Giving the model explicit, category-specific instructions is far more reliable than hoping it infers the right behavior from a generic error message. These instructions also serve as documentation for your team about what each error type means. This connects to the broader topic of prompt engineering fundamentals — your system prompt is doing real behavioral programming here.

Hands-On Exercise

Build a validation layer for a financial data pipeline agent. The agent has two tools: get_stock_price(ticker: str, date: str) and calculate_portfolio_value(holdings: list[dict]).

Requirements:

1. Define Pydantic argument schemas for both tools. The ticker must be 1-5 uppercase letters. The date must be a valid ISO date string that is not in the future. Each holding in the list must have ticker (str), shares (positive float), and purchase_price (positive float).

2. Define result contracts: get_stock_price returns {ticker, price, date, currency, market_status} where market_status is one of "open", "closed", "holiday". calculate_portfolio_value returns {total_value, holdings_count, currency, as_of_date}.

3. Implement the error taxonomy for this domain. What error categories make sense for a stock data API? (Hint: think about what happens when you request a date when markets were closed, or a ticker that was delisted.)

4. Write a validate_holding_list function that validates each entry in the holdings list and returns a list of per-item errors if any are invalid, rather than failing on the first invalid holding.

5. Test your system by deliberately passing: an invalid ticker ("TOOLONGNAME"), a future date, a negative share count, and a valid request. Verify that errors are returned in a format the LLM can act on, and that valid requests pass through cleanly.

Stretch goal: Implement a "partial success" result contract for calculate_portfolio_value that can still return a value when some tickers fail to fetch their price, noting which holdings were excluded from the calculation.

Common Mistakes & Troubleshooting

Mistake 1: Treating all errors as tool failure messages If your database throws a ConnectionRefusedError and you send the raw exception message to the LLM, two bad things happen: you leak implementation details, and the LLM will often try to tell the user about database connection errors — which is unhelpful and potentially embarrassing. Always classify before surfacing.

Mistake 2: Validating arguments but not results This is extremely common. Teams add Pydantic validation for LLM inputs because they know those are unreliable, but trust their own tool implementations implicitly. In practice, result contract violations from API drift or bugs in your tool code cause subtle, hard-to-debug failures where the agent confidently reports incorrect information.

Mistake 3: Returning errors that are too vague "An error occurred" tells the model nothing actionable. "The field 'order_id' must match the pattern ORD-XXXXXXXX. You provided 'ORD-123'. Please reformat and try again." gives the model everything it needs to self-correct. The more specific your error messages, the higher your first-call resolution rate.

Mistake 4: Validating too strictly without coercion If the model provides discount_percent as the string "15" instead of the integer 15, Pydantic's default behavior in v2 is to coerce it. Accept this gift. Reserve hard validation failures for genuinely wrong values, not type formatting differences. The goal is to handle real failures, not to punish the model for minor formatting variations.

Mistake 5: No circuit breaker for repeated failures Without the consecutive error tracking shown in the agentic loop above, a misconfigured tool or a bad model prompt can drain your API budget before you notice. Monitor your average iteration count per conversation — a sudden increase is a leading indicator of a validation or schema problem.

Tip: Add structured logging to your run_tool_call function that records the tool name, argument validation outcome, execution outcome, result validation outcome, and total latency on every call. This data is invaluable for debugging and for building dashboards that show you which tools are flaky. See LLM observability patterns for a complete approach to this.

Troubleshooting: The model keeps generating the same invalid arguments Check your error message format. If the model is seeing {"status": "validation_error", "errors": [{"loc": ["order_id"], "msg": "String should match pattern..."}]}, it might not connect loc: ["order_id"] to the field name. Rewrite to "Field 'order_id': must match pattern ORD-XXXXXXXX. You provided: 'ORD-123'" and you'll typically see self-correction on the next attempt.

Troubleshooting: Validation passes but the LLM ignores tool results If the model generates a tool call, gets a result, and then responds as if it hadn't, check your token budget. Long conversation histories with many tool calls can push the actual tool results near or past the context window, causing them to be truncated. Keep your result payloads lean — return only what the model needs, not entire API responses.

Troubleshooting: Inconsistent behavior on identical inputs If validation sometimes passes and sometimes fails for the same input, check for non-determinism in your validator logic — especially anything involving the current time (e.g., "date must not be in the future" validators) or external state. Make sure your validators are pure functions.

Summary & Next Steps

You've built a complete validation and error recovery layer for an LLM function calling pipeline. The architecture has three core layers working together: argument validation catches bad inputs from the LLM before they reach your tools, result contract validation catches schema drift and bugs in your tool implementations, and a typed error taxonomy drives different recovery strategies for different failure modes. The agentic loop ties everything together with consecutive error tracking to prevent runaway loops.

The key mental model to carry forward is this: your pipeline exists in an adversarial environment where both the LLM and the external services it calls are unreliable in different ways. The validation layer is not defensive programming — it's the actual reliability mechanism. Without it, you're hoping two unreliable systems happen to agree.

Where to go next:

• For agents that need to reason across multiple tool calls and maintain state, explore structured agent memory patterns to understand how to persist validation context across sessions.

• To protect against adversarial inputs that could manipulate your tool calls through user messages, read up on guardrails and safety layers and prompt injection defense.

• For evaluating whether your error recovery is actually working in production — and not just in your unit tests — build an LLM-as-judge evaluation pipeline that tests your agent's behavior across a battery of failure scenarios.

• Once your validation layer is solid, look at how structured output patterns can further constrain model behavior and reduce the surface area for argument validation failures in the first place.