Single-agent LLM systems hit their limits fast. Learn how to design production-grade multi-agent architectures with explicit task delegation contracts, surgical context transfer, and principled result aggregation — including full Python implementations and observability patterns.

Imagine you're building an AI-powered investment research platform. A user asks: "Should I rebalance my portfolio given the recent Fed rate decision, my current holdings, and my tax situation this year?" That question spans at least three distinct domains — macroeconomic analysis, portfolio optimization, and tax strategy — and no single LLM agent can handle all of it well. If you throw the whole thing at one GPT-4o instance with an enormous system prompt trying to be everything at once, you'll get mediocre output across all three dimensions and a context window that's hemorrhaging tokens and money.
The solution isn't a bigger model. It's a better architecture. Multi-agent systems let you decompose complex tasks across specialized agents, each optimized for a specific slice of the problem, and then stitch their outputs back together into a coherent result. But "just call multiple agents" is not a protocol — it's a wish. What separates a production multi-agent system from a research demo is the rigor of three things: how tasks get delegated (who decides what goes where, with what instructions), how context gets transferred (what information travels with the task, in what format, with what guarantees), and how results get aggregated (how partial outputs are combined, reconciled, and delivered as a unified response).
By the end of this lesson, you'll know how to implement all three layers with production-grade code. You'll understand the design trade-offs involved, avoid the classic failure modes that kill multi-agent systems in real deployments, and have a working reference architecture you can adapt immediately.
What you'll learn:
You should be comfortable with:
aiohttp or similar)You'll also benefit from familiarity with the agentic loop design patterns covered elsewhere in this learning path.
Before writing a line of code, you need a mental model that holds up under real complexity. The pattern that works in production is the orchestrator-agent topology: a central orchestrator that understands the overall task and delegates to specialist agents, each of which operates within a well-defined scope.
This is distinct from two other common patterns you'll encounter:
Fully sequential chains (Agent A → Agent B → Agent C) are simple but brittle. Each agent in the chain has to consume and re-emit the full context, which causes drift, token waste, and cascading failures. If Agent B fails, the whole chain stops.
Peer-to-peer meshes (any agent can call any other agent) are flexible but quickly become debugging nightmares. When something goes wrong, you have no clear accountability chain, and coordination overhead scales quadratically.
The orchestrator-agent topology hits the sweet spot: the orchestrator maintains the global task plan and state, while agents maintain only local task state. Agents don't know about each other. They receive a task envelope, do their job, and return a result envelope. The orchestrator does all the routing, context management, and aggregation.
User Request
│
▼
┌─────────────────┐
│ Orchestrator │ ← Owns the task plan, routes, aggregates
│ (LLM + logic) │
└────────┬────────┘
│ Delegates tasks (with context envelopes)
┌────┴─────────┬─────────────┐
▼ ▼ ▼
┌────────┐ ┌─────────┐ ┌──────────┐
│Macro │ │Portfolio│ │Tax │
│Analysis│ │Agent │ │Strategy │
│Agent │ │ │ │Agent │
└────────┘ └─────────┘ └──────────┘
│ │ │
└──────────────┴─────────────┘
│
Result Envelopes
│
▼
┌─────────────┐
│ Aggregator │ ← May be part of orchestrator
└─────────────┘
│
▼
Final Response
Key insight
The orchestrator doesn't have to be an LLM — and often shouldn't be for the routing layer. Use an LLM where you need judgment (task decomposition, conflict resolution, final synthesis). Use deterministic code where you need reliability (routing, schema validation, retry logic). The best production systems mix both.
The most common failure in multi-agent systems is implicit contracts: the orchestrator sends a vague task description, the agent makes assumptions, and the result doesn't fit what the orchestrator expected. This is exactly the kind of problem that disappears in demos (where everything is cherry-picked) and destroys production systems.
The solution is to define a Task Envelope — a typed structure that carries everything an agent needs to execute its task, unambiguously. Here's the schema:
from pydantic import BaseModel, Field
from typing import Any, Optional, Literal
from datetime import datetime
import uuid
class TaskContext(BaseModel):
"""Minimal context slice relevant to this specific task."""
user_id: str
session_id: str
original_request: str
relevant_facts: dict[str, Any] = Field(default_factory=dict)
upstream_results: dict[str, Any] = Field(default_factory=dict)
constraints: list[str] = Field(default_factory=list)
class TaskEnvelope(BaseModel):
"""The contract between orchestrator and agent."""
task_id: str = Field(default_factory=lambda: str(uuid.uuid4()))
agent_role: str # e.g., "macro_analysis", "portfolio_optimization"
task_description: str # Specific, scoped instruction
context: TaskContext
output_schema: dict[str, Any] # JSON Schema the agent must conform to
priority: Literal["low", "medium", "high", "critical"] = "medium"
timeout_seconds: int = 30
max_tokens: int = 1500
created_at: datetime = Field(default_factory=datetime.utcnow)
parent_task_id: Optional[str] = None # For hierarchical task trees
allow_tool_use: bool = True
metadata: dict[str, Any] = Field(default_factory=dict)
The critical fields here are output_schema and task_description. The output schema tells the agent exactly what format its result must take — not vaguely, but as a concrete JSON Schema that you'll validate against. The task description is scoped, not open-ended. Compare these two:
Bad (implicit): "Analyze the macroeconomic situation."
Good (explicit): "Given the Fed rate decision described in context.relevant_facts['fed_announcement'], assess: (1) the likely impact on 10-year Treasury yields over the next 90 days, (2) implications for growth vs. value equity allocation, and (3) one specific risk factor that might invalidate this analysis. Output must conform to the provided schema."
The difference isn't just stylistic. The explicit version gives the agent a bounded task it can actually complete, produces output in a predictable shape, and surfaces uncertainty explicitly. The vague version gives you a fluent essay that's nearly impossible to programmatically aggregate with other results.
Context transfer is where most multi-agent systems quietly leak both money and coherence. The naive approach is to pass the entire conversation history to every agent. That's expensive (each agent re-processes the full history), introduces irrelevant information that distracts the agent, and often leaks context from one domain into another where it doesn't belong.
The right approach is surgical context extraction: the orchestrator maintains the full context, but each agent receives only the slice it needs. This requires deliberate design.
class ContextManager:
"""
Manages the global task context and produces minimal slices
for individual agents.
"""
def __init__(self, session_id: str, user_id: str):
self.session_id = session_id
self.user_id = user_id
self._full_context: dict[str, Any] = {}
self._completed_results: dict[str, "ResultEnvelope"] = {}
self._fact_index: dict[str, str] = {} # fact_key -> source_task_id
def ingest_user_context(self, structured_data: dict[str, Any]) -> None:
"""Store structured facts extracted from user input."""
self._full_context.update(structured_data)
for key in structured_data:
self._fact_index[key] = "user_input"
def register_result(self, result: "ResultEnvelope") -> None:
"""Store a completed agent result and index its outputs."""
self._completed_results[result.task_id] = result
for key, value in result.extracted_facts.items():
self._full_context[key] = value
self._fact_index[key] = result.task_id
def build_context_slice(
self,
agent_role: str,
required_facts: list[str],
upstream_task_ids: list[str]
) -> TaskContext:
"""
Build a minimal context envelope for a specific agent.
Only includes facts the agent actually needs.
"""
relevant_facts = {}
for fact_key in required_facts:
if fact_key in self._full_context:
relevant_facts[fact_key] = self._full_context[fact_key]
upstream_results = {}
for task_id in upstream_task_ids:
if task_id in self._completed_results:
result = self._completed_results[task_id]
# Pass only the structured output, not raw LLM text
upstream_results[task_id] = {
"agent_role": result.agent_role,
"summary": result.summary,
"structured_output": result.structured_output,
"confidence": result.confidence_score
}
return TaskContext(
user_id=self.user_id,
session_id=self.session_id,
original_request=self._full_context.get("original_request", ""),
relevant_facts=relevant_facts,
upstream_results=upstream_results,
constraints=self._full_context.get("user_constraints", [])
)
Notice that when we pass upstream results to an agent, we pass the structured output — not the raw LLM-generated text. This is a crucial discipline. If Agent B receives Agent A's result as a wall of narrative prose, it'll re-process it through its own lens and potentially contradict it without flagging the conflict. If it receives structured fields with typed values, it has something it can actually act on deterministically.
Warning
Be especially careful about context contamination when agents have overlapping domains. If your tax agent receives full portfolio optimization results, it may try to give portfolio advice instead of staying in its lane. Scope the context slice aggressively. It's better to give an agent slightly less context and have it ask for more (via a tool call back to the orchestrator) than to pollute its reasoning with irrelevant information.
Each specialized agent follows the same interface regardless of what it does internally. This is what makes the system composable:
from openai import AsyncOpenAI
import json
import asyncio
from typing import Optional
class ResultEnvelope(BaseModel):
task_id: str
agent_role: str
status: Literal["success", "partial", "failed", "escalated"]
structured_output: dict[str, Any]
summary: str # 1-2 sentence human-readable summary
extracted_facts: dict[str, Any] # Facts others can build on
confidence_score: float # 0.0-1.0
warnings: list[str] = Field(default_factory=list)
error_detail: Optional[str] = None
input_tokens: int = 0
output_tokens: int = 0
latency_ms: float = 0.0
created_at: datetime = Field(default_factory=datetime.utcnow)
class SpecialistAgent:
"""
Base class for all specialist agents. Handles the
protocol layer; subclasses implement domain logic.
"""
def __init__(self, role: str, system_prompt: str, client: AsyncOpenAI):
self.role = role
self.system_prompt = system_prompt
self.client = client
async def execute(self, envelope: TaskEnvelope) -> ResultEnvelope:
import time
start = time.monotonic()
try:
result = await self._run_with_timeout(envelope)
result.latency_ms = (time.monotonic() - start) * 1000
return result
except asyncio.TimeoutError:
return ResultEnvelope(
task_id=envelope.task_id,
agent_role=self.role,
status="failed",
structured_output={},
summary="Agent timed out.",
extracted_facts={},
confidence_score=0.0,
error_detail=f"Exceeded {envelope.timeout_seconds}s timeout",
latency_ms=(time.monotonic() - start) * 1000
)
except Exception as e:
return ResultEnvelope(
task_id=envelope.task_id,
agent_role=self.role,
status="failed",
structured_output={},
summary="Agent execution failed.",
extracted_facts={},
confidence_score=0.0,
error_detail=str(e),
latency_ms=(time.monotonic() - start) * 1000
)
async def _run_with_timeout(self, envelope: TaskEnvelope) -> ResultEnvelope:
return await asyncio.wait_for(
self._execute_core(envelope),
timeout=envelope.timeout_seconds
)
async def _execute_core(self, envelope: TaskEnvelope) -> ResultEnvelope:
"""Core execution logic — override this in subclasses for custom behavior."""
user_message = self._build_user_message(envelope)
response = await self.client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": self.system_prompt},
{"role": "user", "content": user_message}
],
response_format={"type": "json_object"},
max_tokens=envelope.max_tokens,
temperature=0.2 # Low temperature for consistent structured output
)
raw_output = response.choices[0].message.content
parsed = json.loads(raw_output)
# Validate against expected schema
self._validate_output(parsed, envelope.output_schema)
return ResultEnvelope(
task_id=envelope.task_id,
agent_role=self.role,
status="success",
structured_output=parsed,
summary=parsed.get("summary", ""),
extracted_facts=parsed.get("extracted_facts", {}),
confidence_score=parsed.get("confidence_score", 0.8),
warnings=parsed.get("warnings", []),
input_tokens=response.usage.prompt_tokens,
output_tokens=response.usage.completion_tokens
)
def _build_user_message(self, envelope: TaskEnvelope) -> str:
context_str = json.dumps(envelope.context.model_dump(), indent=2)
schema_str = json.dumps(envelope.output_schema, indent=2)
return f"""Task: {envelope.task_description}
Context:
{context_str}
Required Output Schema:
{schema_str}
IMPORTANT: Your entire response must be valid JSON conforming exactly to the schema above.
Include a "confidence_score" (0.0-1.0), a "summary" (1-2 sentences),
"extracted_facts" (key facts others can build on), and "warnings" (any caveats).
"""
def _validate_output(self, output: dict, schema: dict) -> None:
"""Validate output against JSON schema. Raises ValueError on failure."""
import jsonschema
try:
jsonschema.validate(instance=output, schema=schema)
except jsonschema.ValidationError as e:
raise ValueError(f"Agent output failed schema validation: {e.message}")
Tip
Setting temperature to 0.1–0.2 for structured-output agents is not just a convention — it's a correctness strategy. Higher temperatures increase the chance the model will deviate from your JSON schema, invent fields, or add commentary outside the structure. Save the creativity for synthesis tasks where variation is an asset, not a liability.
The orchestrator has the hardest job in the system. It needs to:
Here's a production-grade orchestrator implementation:
from dataclasses import dataclass, field
from enum import Enum
import asyncio
class ExecutionStrategy(Enum):
PARALLEL = "parallel"
SEQUENTIAL = "sequential"
CONDITIONAL = "conditional"
@dataclass
class TaskNode:
"""Node in the task execution DAG."""
envelope: TaskEnvelope
agent: SpecialistAgent
dependencies: list[str] = field(default_factory=list) # task_ids this depends on
strategy: ExecutionStrategy = ExecutionStrategy.PARALLEL
class Orchestrator:
def __init__(self, client: AsyncOpenAI, context_manager: ContextManager):
self.client = client
self.context_manager = context_manager
self.agents: dict[str, SpecialistAgent] = {}
self.task_registry: dict[str, TaskNode] = {}
self.results: dict[str, ResultEnvelope] = {}
def register_agent(self, agent: SpecialistAgent) -> None:
self.agents[agent.role] = agent
async def decompose_request(self, user_request: str) -> list[dict]:
"""
Use an LLM to decompose the user request into agent tasks.
Returns a list of task specifications.
"""
available_agents = list(self.agents.keys())
decomposition_prompt = f"""You are a task routing system.
Available specialist agents: {json.dumps(available_agents)}
User request: {user_request}
Decompose this request into specific tasks for the available agents.
Return a JSON array where each element has:
- "agent_role": one of the available agents
- "task_description": specific, scoped instruction for that agent
- "required_facts": list of fact keys this agent needs from context
- "depends_on": list of agent_roles that must complete first (for sequential deps)
- "can_run_parallel": boolean
Only include agents that are genuinely needed for this request.
"""
response = await self.client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": decomposition_prompt}],
response_format={"type": "json_object"},
temperature=0
)
result = json.loads(response.choices[0].message.content)
return result.get("tasks", [])
async def execute_plan(
self,
task_specs: list[dict],
user_context: dict[str, Any]
) -> dict[str, ResultEnvelope]:
"""
Execute the task plan, respecting dependencies and maximizing parallelism.
"""
self.context_manager.ingest_user_context(user_context)
# Build task nodes
task_nodes = self._build_task_dag(task_specs)
# Execute in topological order, parallelizing where possible
completed = set()
all_results = {}
while len(completed) < len(task_nodes):
# Find tasks whose dependencies are all satisfied
ready = [
node for task_id, node in task_nodes.items()
if task_id not in completed
and all(dep in completed for dep in node.dependencies)
]
if not ready:
# Circular dependency or all remaining tasks have unfulfilled deps
break
# Execute ready tasks in parallel
tasks = [self._execute_single_task(node) for node in ready]
batch_results = await asyncio.gather(*tasks, return_exceptions=True)
for node, result in zip(ready, batch_results):
task_id = node.envelope.task_id
if isinstance(result, Exception):
result = ResultEnvelope(
task_id=task_id,
agent_role=node.envelope.agent_role,
status="failed",
structured_output={},
summary="Unexpected error during execution.",
extracted_facts={},
confidence_score=0.0,
error_detail=str(result)
)
all_results[task_id] = result
self.context_manager.register_result(result)
completed.add(task_id)
return all_results
async def _execute_single_task(self, node: TaskNode) -> ResultEnvelope:
"""Execute a single task node, building its context slice first."""
# Rebuild context slice now that upstream results may be available
upstream_task_ids = [
tid for tid, n in self.task_registry.items()
if n.envelope.agent_role in node.dependencies
]
updated_context = self.context_manager.build_context_slice(
agent_role=node.envelope.agent_role,
required_facts=node.envelope.metadata.get("required_facts", []),
upstream_task_ids=upstream_task_ids
)
# Update the envelope with fresh context
updated_envelope = node.envelope.model_copy(
update={"context": updated_context}
)
return await node.agent.execute(updated_envelope)
def _build_task_dag(self, task_specs: list[dict]) -> dict[str, TaskNode]:
"""Build the task dependency graph from decomposition output."""
nodes = {}
role_to_task_id = {}
for spec in task_specs:
agent_role = spec["agent_role"]
if agent_role not in self.agents:
continue
task_id = str(uuid.uuid4())
role_to_task_id[agent_role] = task_id
envelope = TaskEnvelope(
task_id=task_id,
agent_role=agent_role,
task_description=spec["task_description"],
context=TaskContext(
user_id=self.context_manager.user_id,
session_id=self.context_manager.session_id,
original_request=self.context_manager._full_context.get(
"original_request", ""
)
),
output_schema=self._get_output_schema(agent_role),
metadata={"required_facts": spec.get("required_facts", [])}
)
nodes[task_id] = TaskNode(
envelope=envelope,
agent=self.agents[agent_role],
dependencies=[], # Will resolve below
strategy=(
ExecutionStrategy.PARALLEL
if spec.get("can_run_parallel", True)
else ExecutionStrategy.SEQUENTIAL
)
)
# Resolve dependencies: convert role names to task IDs
for spec in task_specs:
agent_role = spec["agent_role"]
if agent_role not in role_to_task_id:
continue
task_id = role_to_task_id[agent_role]
nodes[task_id].dependencies = [
role_to_task_id[dep_role]
for dep_role in spec.get("depends_on", [])
if dep_role in role_to_task_id
]
self.task_registry = nodes
return nodes
def _get_output_schema(self, agent_role: str) -> dict:
"""Return the expected JSON schema for a given agent role."""
# In production, these would be stored in a registry
base_schema = {
"type": "object",
"required": ["summary", "confidence_score", "extracted_facts", "warnings"],
"properties": {
"summary": {"type": "string"},
"confidence_score": {"type": "number", "minimum": 0, "maximum": 1},
"extracted_facts": {"type": "object"},
"warnings": {"type": "array", "items": {"type": "string"}}
}
}
return base_schema
Aggregation is the step that most tutorials skip. They show you how to run three agents, print their results, and call it done. But in a real application, the user asked one question and expects one answer — not three separate essays. Aggregation is where the multi-agent system either proves its worth or falls apart.
There are three distinct aggregation challenges you need to handle:
1. Conflict resolution: What happens when Agent A says "increase bond allocation" and Agent B says "decrease bond allocation"? You need to surface the conflict, not silently pick one.
2. Confidence weighting: A result with a 0.95 confidence score should contribute more to the final synthesis than one with 0.4. You need a principled weighting mechanism.
3. Partial failure handling: What if one of three agents failed? Can you still produce a useful result? Often yes — but you need to be explicit about what's missing.
class AggregationResult(BaseModel):
final_response: str
contributing_agents: list[str]
failed_agents: list[str]
conflicts_detected: list[dict[str, Any]]
overall_confidence: float
caveats: list[str]
raw_agent_results: dict[str, Any] # For debugging/audit
class ResultAggregator:
def __init__(self, client: AsyncOpenAI):
self.client = client
def detect_conflicts(
self,
results: dict[str, ResultEnvelope]
) -> list[dict]:
"""
Heuristically detect potential conflicts across agent results.
In production, this can be rule-based or LLM-assisted.
"""
conflicts = []
successful = {
k: v for k, v in results.items()
if v.status in ("success", "partial")
}
# Look for explicit conflict markers in extracted facts
all_facts = {}
for task_id, result in successful.items():
for fact_key, fact_value in result.extracted_facts.items():
if fact_key in all_facts:
existing_value, existing_source = all_facts[fact_key]
if existing_value != fact_value:
conflicts.append({
"fact_key": fact_key,
"value_a": existing_value,
"source_a": existing_source,
"value_b": fact_value,
"source_b": task_id
})
else:
all_facts[fact_key] = (fact_value, task_id)
return conflicts
def compute_overall_confidence(
self,
results: dict[str, ResultEnvelope],
total_expected: int
) -> float:
"""
Weighted confidence score accounting for failures.
"""
successful = [r for r in results.values() if r.status == "success"]
partial = [r for r in results.values() if r.status == "partial"]
if not successful and not partial:
return 0.0
# Weight: successes count fully, partials at 0.5
weighted_confidence = sum(r.confidence_score for r in successful)
weighted_confidence += sum(r.confidence_score * 0.5 for r in partial)
# Penalize for missing agents
completeness_factor = len(results) / total_expected
denominator = len(successful) + len(partial) * 0.5
if denominator == 0:
return 0.0
avg_confidence = weighted_confidence / denominator
return round(avg_confidence * completeness_factor, 3)
async def synthesize(
self,
results: dict[str, ResultEnvelope],
original_request: str,
total_expected_agents: int
) -> AggregationResult:
"""
Use an LLM to synthesize agent results into a coherent final response.
"""
successful_results = {
k: v for k, v in results.items()
if v.status in ("success", "partial")
}
failed_results = {
k: v for k, v in results.items()
if v.status in ("failed", "escalated")
}
conflicts = self.detect_conflicts(results)
overall_confidence = self.compute_overall_confidence(
results, total_expected_agents
)
# Build synthesis context (compact — only structured outputs)
synthesis_context = {
agent_id: {
"role": result.agent_role,
"summary": result.summary,
"output": result.structured_output,
"confidence": result.confidence_score,
"warnings": result.warnings
}
for agent_id, result in successful_results.items()
}
failed_roles = [v.agent_role for v in failed_results.values()]
synthesis_prompt = f"""You are synthesizing results from multiple specialist agents
into a single coherent response for the user.
Original user request: {original_request}
Agent results:
{json.dumps(synthesis_context, indent=2)}
Detected conflicts: {json.dumps(conflicts, indent=2)}
Failed agents (data unavailable): {failed_roles}
Instructions:
1. Synthesize the agent results into a direct, actionable response to the user
2. Where agents agree, state the consensus confidently
3. Where conflicts exist, explicitly surface them and explain the trade-offs
4. If agents failed, acknowledge what information is missing and how it affects completeness
5. Do not add information not present in the agent results
6. Keep the response focused and practical
Respond in JSON with keys: "response", "caveats" (list of strings)"""
llm_response = await self.client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": synthesis_prompt}],
response_format={"type": "json_object"},
temperature=0.3
)
synthesis = json.loads(llm_response.choices[0].message.content)
return AggregationResult(
final_response=synthesis["response"],
contributing_agents=[r.agent_role for r in successful_results.values()],
failed_agents=failed_roles,
conflicts_detected=conflicts,
overall_confidence=overall_confidence,
caveats=synthesis.get("caveats", []),
raw_agent_results={
k: v.model_dump() for k, v in results.items()
}
)
Key insight
The synthesis LLM is acting as an editor and integrator, not as an analyst. Its job is to weave together structured inputs that other agents have already validated and reasoned over. This division of labor is what prevents the classic problem of a single "super-agent" drifting between roles and losing coherence.
Here's how the full system looks end-to-end for the investment research use case:
async def handle_investment_query(
user_request: str,
user_id: str,
portfolio_data: dict,
tax_data: dict
) -> AggregationResult:
client = AsyncOpenAI()
session_id = str(uuid.uuid4())
# Initialize context manager
ctx_manager = ContextManager(session_id=session_id, user_id=user_id)
# Pre-load user context
user_context = {
"original_request": user_request,
"portfolio": portfolio_data,
"tax_situation": tax_data,
"user_constraints": ["risk_tolerance: moderate", "investment_horizon: 5yr"]
}
# Create specialist agents
macro_agent = SpecialistAgent(
role="macro_analysis",
system_prompt="""You are a macroeconomic analyst specializing in Fed policy
and its effects on asset classes. You reason from first principles,
cite specific data points, and always quantify uncertainty.""",
client=client
)
portfolio_agent = SpecialistAgent(
role="portfolio_optimization",
system_prompt="""You are a portfolio analyst. You analyze allocation efficiency,
rebalancing triggers, and risk-adjusted return expectations.
You always work from provided data, not assumptions.""",
client=client
)
tax_agent = SpecialistAgent(
role="tax_strategy",
system_prompt="""You are a tax strategy specialist focused on investment tax
efficiency. You identify tax-loss harvesting opportunities, wash-sale risks,
and capital gains optimization strategies. You flag when professional
tax advice is required.""",
client=client
)
# Initialize orchestrator
orchestrator = Orchestrator(client=client, context_manager=ctx_manager)
orchestrator.register_agent(macro_agent)
orchestrator.register_agent(portfolio_agent)
orchestrator.register_agent(tax_agent)
# Decompose and execute
task_specs = await orchestrator.decompose_request(user_request)
results = await orchestrator.execute_plan(task_specs, user_context)
# Aggregate
aggregator = ResultAggregator(client=client)
final_result = await aggregator.synthesize(
results=results,
original_request=user_request,
total_expected_agents=len(task_specs)
)
return final_result
A multi-agent system that you can't observe is a liability in production. When something goes wrong — and it will — you need to know which agent failed, what context it received, what it produced, how long it took, and how much it cost. Instrument everything from the start, not as an afterthought.
The key observability dimensions for multi-agent systems are:
Trace correlation: Every task needs a trace_id that links back to the originating user request. When you're looking at a weird result, you need to be able to pull all agent calls for that request in one query. The LLM observability patterns covered elsewhere in this path apply here, with the addition that you need to track the parent-child relationship between orchestrator calls and agent calls.
Cost attribution: Multi-agent systems can get expensive fast. Each agent call is a separate API call, and your orchestrator's decomposition + synthesis calls add overhead on top. Build token counting into your ResultEnvelope from day one (we did this above with input_tokens and output_tokens). Aggregate these per-request and alert when a request's total cost exceeds a threshold. For more on managing this systematically, see the guide on cost optimization strategies.
Latency breakdown: Log each agent's latency_ms and the orchestrator's overall latency separately. If you're seeing slow responses, you need to know whether it's one slow agent in a parallel batch (which you might be able to swap out) or the synthesis step (which might need a faster model).
def emit_telemetry(
request_id: str,
results: dict[str, ResultEnvelope],
aggregation: AggregationResult,
total_latency_ms: float
) -> None:
"""Emit structured telemetry for the full request."""
total_input_tokens = sum(r.input_tokens for r in results.values())
total_output_tokens = sum(r.output_tokens for r in results.values())
telemetry = {
"request_id": request_id,
"timestamp": datetime.utcnow().isoformat(),
"total_latency_ms": total_latency_ms,
"total_input_tokens": total_input_tokens,
"total_output_tokens": total_output_tokens,
"agent_count": len(results),
"success_count": sum(1 for r in results.values() if r.status == "success"),
"failed_count": sum(1 for r in results.values() if r.status == "failed"),
"overall_confidence": aggregation.overall_confidence,
"conflicts_detected": len(aggregation.conflicts_detected),
"per_agent": [
{
"agent_role": r.agent_role,
"status": r.status,
"latency_ms": r.latency_ms,
"input_tokens": r.input_tokens,
"output_tokens": r.output_tokens,
"confidence": r.confidence_score
}
for r in results.values()
]
}
# In production: send to your observability platform (Langfuse, Datadog, etc.)
print(json.dumps(telemetry, indent=2))
Note
If you're running this at scale, consider parallel LLM call patterns for the orchestration layer itself. When you have many concurrent user requests, you need async patterns that go beyond basic asyncio.gather — proper semaphores, rate limit backpressure, and connection pooling all matter.
Build a multi-agent system for the following scenario: a customer support escalation analyzer that handles incoming support tickets for a software product.
Your system should have:
Implementation requirements:
Define TaskEnvelope and ResultEnvelope schemas with output schemas specific to each agent role.
Implement the ContextManager with a build_context_slice method. The technical diagnosis agent should receive the full ticket text. The resolution routing agent should receive outputs from both upstream agents (sentiment/urgency AND technical diagnosis).
Make sentiment/urgency and technical diagnosis run in parallel, then feed their results to the resolution routing agent sequentially.
Implement conflict detection: if the urgency score is "critical" but the resolution router recommends "auto-resolve," flag this as a conflict requiring human review.
Add telemetry emission that records per-agent token counts and latencies.
Stretch goal: Add a retry mechanism to the agent base class that retries once on schema validation failure, using the validation error message as additional context in the retry prompt.
The most common issue: agents receive narrative summaries of upstream results rather than structured fields. This causes the downstream agent to re-interpret the prose and potentially introduce drift or hallucinations. Always extract and pass structured facts, not summaries.
If your dependency graph has a cycle (Agent A depends on B, B depends on A), asyncio.gather won't deadlock — it'll just never execute the cycle. Build cycle detection into _build_task_dag:
def _detect_cycles(self, nodes: dict[str, TaskNode]) -> bool:
"""DFS-based cycle detection in the task DAG."""
visited = set()
path = set()
def dfs(task_id: str) -> bool:
visited.add(task_id)
path.add(task_id)
for dep in nodes[task_id].dependencies:
if dep not in visited:
if dfs(dep):
return True
elif dep in path:
return True
path.discard(task_id)
return False
for task_id in nodes:
if task_id not in visited:
if dfs(task_id):
return True
return False
The decomposition step is an LLM call with no ground truth. In production, you should validate decomposition outputs against a whitelist of known agent roles and expected task patterns. If the decomposition produces an unknown agent role, fail fast rather than silently skipping it.
If two of three agents succeed, you must decide: produce a partial answer with caveats, or block until all agents complete (possibly via retry)? The answer depends on your domain. For the investment research case, missing tax strategy is a material gap — you should surface it prominently. Build this logic into your aggregator explicitly.
If an agent can itself spawn sub-agents (hierarchical multi-agent systems), you're one bug away from an infinite recursion loop that runs up a large API bill before hitting any timeout. Always maintain a depth counter in the task envelope and refuse to spawn sub-agents beyond a maximum depth (typically 2-3 levels).
Warning
Multi-agent systems can silently accumulate large token costs in production, especially if agents are verbose and pass each other's full outputs downstream. Build budget guardrails at the per-request level: estimate total tokens before dispatching the full plan, and refuse or downgrade the plan if it exceeds your budget threshold. This integrates naturally with the LLM router architecture pattern — cheaper agents for simpler subtasks.
When an agent consistently fails schema validation, the typical causes are:
Fix #1 and #2 through schema design and prompt hardening. Fix #3 by moving the JSON schema requirement to the system prompt rather than the user message.
If your relevant_facts dict grows without bound as agents produce results, you'll eventually hit context window limits. Implement a pruning strategy: after each agent completes, extract only the specific fact keys the remaining agents need. Mark consumed facts so they can be pruned from the live context.
You now have a production-ready architecture for multi-agent handoff protocols. The core disciplines are:
TaskEnvelope and ResultEnvelope schemas with output schemas make the interface between orchestrator and agents deterministic and validatableContextManager extracts minimal, relevant context slices rather than passing the full state to every agentThe architecture presented here is intentionally framework-agnostic — it uses pure Python and the OpenAI SDK directly, so you can understand every moving part. If you're working with LangChain or LlamaIndex, the same conceptual layers apply; those frameworks provide some of the scaffolding but impose their own abstractions that you'll need to work around for fine-grained control.
Where to go next:
The hardest part of multi-agent systems isn't the code — it's the discipline of maintaining explicit contracts between components when LLMs make it tempting to just "prompt your way through" the coordination layer. The systems that hold up in production are the ones where every agent knows exactly what it's supposed to do, what it's supposed to return, and nothing more.
Building with LLMs