Wicked Smart Data
LearnInsightsAboutContact
Sign InLet's Build
LearnInsightsAboutContact
Sign InLet's Build
Wicked Smart Data

Intelligence, automation, and expert execution — plus an elite library of free knowledge. We turn complexity into competitive advantage.

Start a conversation

Platform

  • Learning Paths
  • Insights
  • RSS Feed

Company

  • About
  • Contact
  • Work With Us

Legal

  • Privacy Policy
  • Terms of Service

© 2026 Wicked Smart Data. All rights reserved.

Intelligence · Automation · Advantage

All Insights
AI & Machine Learning

Implementing LLM-Powered Document Routing: Classifying and Directing Incoming Files to Specialized Processing Pipelines Based on Content Type and Business Rules

Most organizations waste enormous time manually triaging incoming documents. This expert-level lesson walks you through building a production-grade LLM routing system that classifies documents, applies business rules, and dispatches files automatically — with the architecture, prompt design, and observability to make it reliable.

🔥 Expert25 min readNov 18, 2026Updated Nov 18, 2026
Implementing LLM-Powered Document Routing: Classifying and Directing Incoming Files to Specialized Processing Pipelines Based on Content Type and Business Rules
On this page
  • Introduction
  • Prerequisites
  • The Architecture Before the Code
  • Layer 1: Document Ingestion and Text Extraction
  • Extracting Text from Common Formats
  • Layer 2: Designing the Classification Prompt
  • Defining Your Document Taxonomy
  • Writing the Classification Prompt
  • Calling the LLM and Parsing the Response
  • Layer 3: The Rule Engine
  • Layer 4: Dispatch and Observability
  • Putting the Pipeline Together
  • Improving Accuracy with Few-Shot Examples
  • Handling the Hard Cases
  • Multilingual Documents
  • Split Documents and Attachments
  • Confidence Calibration Over Time
  • Security and Adversarial Considerations
  • Hands-On Exercise
  • Common Mistakes & Troubleshooting
  • Summary & Next Steps

Implementing LLM-Powered Document Routing: Classifying and Directing Incoming Files to Specialized Processing Pipelines Based on Content Type and Business Rules

Introduction

Imagine your company receives 500 documents per day: vendor invoices, customer contracts, regulatory filings, support tickets, internal memos, and the occasional spreadsheet someone emailed as a PDF "because it looked nicer." Right now, a human — probably a junior analyst or a shared-mailbox owner — triages that inbox, decides what goes where, and manually hands work off to the right team. It's slow, error-prone, and deeply unsatisfying work for anyone smart enough to have a better use of their time.

LLM-powered document routing replaces that manual triage layer with an intelligent classification engine that reads incoming files, understands their content and context, applies your business rules, and dispatches each document to the correct downstream pipeline — whether that's an invoice processing workflow, a legal review queue, a data extraction job, or an escalation path. When it's built well, it handles the routine 95% automatically while surfacing the genuinely ambiguous cases for human review. When it's built poorly, it misroutes sensitive contracts to the wrong team and nobody notices for three weeks.

By the end of this lesson, you'll know how to build the right version. We'll cover the full architectural picture — from document ingestion and content extraction through classification prompt design, structured output parsing, routing logic, confidence scoring, and failure handling.

What you'll learn:

  • How to design a multi-tier classification system that separates document type identification from business-rule application
  • How to write classification prompts that return structured, machine-parseable routing decisions with confidence scores
  • How to handle ambiguous, multilingual, and malformed documents without crashing your pipeline
  • How to instrument your routing system so you can measure accuracy and improve it over time
  • How to integrate LLM classification with deterministic rule engines for a production-grade hybrid architecture

Prerequisites

You should be comfortable with Python and basic API calls. You should have some exposure to working with LLMs programmatically — if you haven't done that yet, read Integrating AI APIs into Business Workflows: Call OpenAI, Claude, and Gemini Programmatically first. You should also understand why structured output matters for automation — we'll use it heavily throughout this lesson.

Basic knowledge of queuing systems (even just conceptually) will help, but isn't required.


The Architecture Before the Code

Before writing a single line of Python, you need a mental model of how this system actually works. Most failed routing implementations skip this step and end up with a giant prompt that tries to do everything at once. That approach breaks on edge cases and is impossible to debug.

A well-designed routing system has four distinct layers:

Layer 1: Ingestion and extraction. Documents arrive — via email attachment, S3 upload, API push, shared drive, or watched folder. Text is extracted from PDFs, Word files, images, and spreadsheets. This layer is deterministic: convert the file, extract the text, normalize encoding.

Layer 2: LLM classification. The extracted text (or a representative sample of it) is sent to an LLM with a carefully designed prompt. The LLM returns a structured JSON object containing: the document type, a confidence score, key extracted metadata, and any flags or anomalies it noticed.

Layer 3: Rule engine. The classification output is fed into a deterministic rule engine. This is where your business logic lives — not the LLM. The rule engine maps document types to pipeline destinations, applies confidence thresholds, checks for mandatory metadata fields, and decides whether a document goes to auto-processing, manual review, or rejection.

Layer 4: Dispatch. The rule engine's output triggers the actual routing — dropping a message onto a queue, calling a webhook, writing a record to a database, or spawning a downstream job.

The key architectural principle here is separation of concerns: the LLM is responsible for understanding documents, the rule engine is responsible for applying business policy. Never put business rules inside your prompt if you can help it. Business rules change frequently; prompts are painful to version and test. Keep them separate.

Key insight

The LLM's job is to describe the document accurately. Your rule engine's job is to decide what to do with it. Conflating these two responsibilities produces fragile systems where changing a business rule requires re-testing your entire classification prompt.


Layer 1: Document Ingestion and Text Extraction

You can't classify text you can't read. Document extraction is unsexy but critical, and it has more gotchas than most people expect.

Extracting Text from Common Formats

import pdfplumber
import docx
import openpyxl
from pathlib import Path

def extract_text(file_path: str, max_chars: int = 8000) -> dict:
    """
    Extract text from a document and return a structured result.
    max_chars limits extraction to control LLM token costs.
    """
    path = Path(file_path)
    suffix = path.suffix.lower()
    
    result = {
        "file_name": path.name,
        "file_type": suffix,
        "text": "",
        "page_count": None,
        "extraction_method": None,
        "extraction_warnings": []
    }
    
    try:
        if suffix == ".pdf":
            with pdfplumber.open(file_path) as pdf:
                result["page_count"] = len(pdf.pages)
                pages_text = []
                for page in pdf.pages[:20]:  # Cap at 20 pages for extraction
                    page_text = page.extract_text()
                    if page_text:
                        pages_text.append(page_text)
                result["text"] = "\n\n".join(pages_text)[:max_chars]
                result["extraction_method"] = "pdfplumber"
                
                if not result["text"].strip():
                    result["extraction_warnings"].append("PDF appears to be image-only; OCR required")
                    
        elif suffix in (".docx", ".doc"):
            doc = docx.Document(file_path)
            paragraphs = [p.text for p in doc.paragraphs if p.text.strip()]
            result["text"] = "\n".join(paragraphs)[:max_chars]
            result["extraction_method"] = "python-docx"
            
        elif suffix in (".xlsx", ".xls"):
            wb = openpyxl.load_workbook(file_path, read_only=True, data_only=True)
            sheet_texts = []
            for sheet_name in wb.sheetnames[:3]:  # First 3 sheets only
                ws = wb[sheet_name]
                rows = []
                for row in ws.iter_rows(max_row=50, values_only=True):
                    row_text = " | ".join(str(cell) for cell in row if cell is not None)
                    if row_text.strip():
                        rows.append(row_text)
                sheet_texts.append(f"[Sheet: {sheet_name}]\n" + "\n".join(rows))
            result["text"] = "\n\n".join(sheet_texts)[:max_chars]
            result["extraction_method"] = "openpyxl"
            
        elif suffix in (".txt", ".csv", ".md"):
            with open(file_path, encoding="utf-8", errors="replace") as f:
                result["text"] = f.read()[:max_chars]
            result["extraction_method"] = "plain_text"
            
        else:
            result["extraction_warnings"].append(f"Unsupported file type: {suffix}")
            
    except Exception as e:
        result["extraction_warnings"].append(f"Extraction error: {str(e)}")
    
    return result

Notice the max_chars=8000 cap. This is a deliberate cost-control decision — you don't need to send an entire 80-page contract to the LLM to classify it as a vendor agreement. The document type is usually determinable from the first few pages. If you're unfamiliar with the token math behind this, the tradeoffs between sending full documents versus excerpts are covered in depth in the lesson on tokens, context windows, and input limits.

Warning

Scanned PDFs with no embedded text are a common failure mode. pdfplumber will return an empty string, your LLM will receive nothing useful, and your classifier will produce garbage. Build detection for this case — check if extracted text is under 100 characters for a multi-page PDF — and route those documents to an OCR pre-processing step (Tesseract, AWS Textract, or Google Document AI) before classification.


Layer 2: Designing the Classification Prompt

This is where the real craft lives. A mediocre classification prompt works fine on clean documents and fails spectacularly on anything unusual. A well-designed prompt handles ambiguity gracefully, extracts metadata you'll actually use, and signals its own uncertainty.

Defining Your Document Taxonomy

Before writing the prompt, you need to define your classification schema. This is a business architecture decision, not a technical one. Work with the people who process documents to identify:

  • The top-level document types in your domain
  • The routing destination for each type
  • The metadata fields that are mandatory for each type
  • The edge cases and boundary conditions (what's the difference between a "vendor invoice" and a "purchase order"?)

Here's a realistic taxonomy for a mid-size company's accounts payable and contracts team:

DOCUMENT_TAXONOMY = {
    "VENDOR_INVOICE": {
        "description": "Bill from a vendor requesting payment for goods or services",
        "required_metadata": ["vendor_name", "invoice_number", "total_amount", "due_date"],
        "routing_destination": "ap_pipeline",
        "examples": ["NET 30 invoice from Acme Corp for $12,400", "Tax invoice #INV-2024-0033"]
    },
    "PURCHASE_ORDER": {
        "description": "Internal document authorizing a purchase from a vendor",
        "required_metadata": ["po_number", "vendor_name", "total_value"],
        "routing_destination": "procurement_pipeline",
        "examples": ["PO #PO-2024-0145 to Dell Technologies for laptop equipment"]
    },
    "VENDOR_CONTRACT": {
        "description": "Legal agreement governing a commercial relationship with a vendor",
        "required_metadata": ["vendor_name", "contract_type", "effective_date", "expiry_date"],
        "routing_destination": "legal_review_queue",
        "examples": ["Master Service Agreement", "Software License Agreement"]
    },
    "CUSTOMER_CONTRACT": {
        "description": "Legal agreement governing a commercial relationship with a customer",
        "required_metadata": ["customer_name", "contract_value", "effective_date"],
        "routing_destination": "sales_ops_pipeline",
        "examples": ["Statement of Work", "Master Subscription Agreement"]
    },
    "COMPLIANCE_FILING": {
        "description": "Regulatory or compliance document requiring submission or retention",
        "required_metadata": ["regulatory_body", "filing_type", "reporting_period"],
        "routing_destination": "compliance_queue",
        "examples": ["SOC 2 audit report", "GDPR DPA", "Tax filing"]
    },
    "SUPPORT_TICKET": {
        "description": "Customer complaint, request for help, or issue report",
        "required_metadata": ["customer_name", "issue_summary", "urgency"],
        "routing_destination": "support_pipeline",
        "examples": ["Email about broken feature", "Billing dispute from customer"]
    },
    "INTERNAL_MEMO": {
        "description": "Internal communication not requiring external action",
        "required_metadata": ["author", "subject"],
        "routing_destination": "archive_pipeline",
        "examples": ["All-hands meeting notes", "Policy update announcement"]
    },
    "UNKNOWN": {
        "description": "Document does not fit any known category",
        "required_metadata": [],
        "routing_destination": "manual_review_queue",
        "examples": []
    }
}

Writing the Classification Prompt

Now build the prompt. We're going to use a system prompt plus a user message structure, which gives us better control over model behavior. The fundamentals of why this structure works are explained in building effective system prompts for business applications.

import json

def build_classification_system_prompt(taxonomy: dict) -> str:
    """Build a system prompt that encodes the document taxonomy."""
    
    taxonomy_description = []
    for doc_type, config in taxonomy.items():
        if doc_type == "UNKNOWN":
            continue
        examples_str = "; ".join(config["examples"]) if config["examples"] else "varies"
        taxonomy_description.append(
            f"- **{doc_type}**: {config['description']}. Examples: {examples_str}"
        )
    
    taxonomy_text = "\n".join(taxonomy_description)
    
    return f"""You are a document classification expert for an enterprise document routing system.

Your task is to analyze the text content of business documents and return a structured JSON classification.

## Document Categories
{taxonomy_text}
- **UNKNOWN**: The document does not clearly fit any category above.

## Instructions
1. Read the document text carefully.
2. Identify the most likely document type from the categories above.
3. Extract any metadata fields that are present in the text.
4. Assign a confidence score between 0.0 and 1.0.
5. Note any anomalies, red flags, or reasons for uncertainty.

## Output Format
Return ONLY valid JSON with this exact structure — no prose, no markdown fences:
{{
  "document_type": "<TYPE_FROM_TAXONOMY>",
  "confidence": <0.0-1.0>,
  "extracted_metadata": {{
    "field_name": "extracted_value"
  }},
  "anomalies": ["<list of anomaly descriptions>"],
  "classification_rationale": "<1-2 sentence explanation of your reasoning>",
  "requires_human_review": <true|false>,
  "review_reason": "<reason if requires_human_review is true, else null>"
}}

## Confidence Score Guidelines
- 0.9–1.0: Unambiguous. Document is clearly this type.
- 0.7–0.9: High confidence. Minor ambiguity but classification is reliable.
- 0.5–0.7: Moderate confidence. Document has mixed signals; consider human review.
- Below 0.5: Low confidence. Classify as UNKNOWN and flag for manual review.

Always prefer UNKNOWN over a forced guess when documents are genuinely ambiguous.
"""
def build_classification_user_message(extraction_result: dict) -> str:
    """Build the user message containing the document content."""
    
    warnings_str = ""
    if extraction_result["extraction_warnings"]:
        warnings_str = f"\n[SYSTEM NOTE: Extraction warnings: {'; '.join(extraction_result['extraction_warnings'])}]"
    
    return f"""Please classify the following document.

File name: {extraction_result['file_name']}
File type: {extraction_result['file_type']}
Page count: {extraction_result.get('page_count', 'unknown')}
{warnings_str}

--- DOCUMENT TEXT BEGIN ---
{extraction_result['text'] if extraction_result['text'] else '[NO TEXT EXTRACTED - possible scanned document]'}
--- DOCUMENT TEXT END ---
"""

Tip

Including the file name in the classification prompt is more valuable than it sounds. A file named MSA_AcmeCorp_2024_FINAL_SIGNED.pdf is a strong signal that overrides an ambiguous document body. The LLM will use that context. Just make sure your confidence scoring guidelines account for cases where the filename is misleading (someone named an invoice contract.pdf).

Calling the LLM and Parsing the Response

from openai import OpenAI
import json
import re

client = OpenAI()

def classify_document(extraction_result: dict, taxonomy: dict) -> dict:
    """
    Send document to LLM for classification and parse the result.
    Returns a structured classification result.
    """
    system_prompt = build_classification_system_prompt(taxonomy)
    user_message = build_classification_user_message(extraction_result)
    
    try:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_message}
            ],
            temperature=0.0,  # Classification needs determinism
            response_format={"type": "json_object"},  # JSON mode
            max_tokens=1000
        )
        
        raw_output = response.choices[0].message.content
        classification = json.loads(raw_output)
        
        # Validate the returned document_type is in our taxonomy
        if classification.get("document_type") not in taxonomy:
            classification["document_type"] = "UNKNOWN"
            classification["anomalies"] = classification.get("anomalies", []) + [
                f"LLM returned unknown document type: {classification.get('document_type')}"
            ]
            classification["requires_human_review"] = True
            classification["review_reason"] = "Invalid document type returned by classifier"
        
        # Enforce confidence threshold
        confidence = float(classification.get("confidence", 0))
        if confidence < 0.5 and classification["document_type"] != "UNKNOWN":
            classification["requires_human_review"] = True
            classification["review_reason"] = (
                classification.get("review_reason") or 
                f"Low confidence score: {confidence:.2f}"
            )
        
        # Attach usage stats for cost tracking
        classification["_meta"] = {
            "model": response.model,
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
            "file_name": extraction_result["file_name"]
        }
        
        return classification
        
    except json.JSONDecodeError as e:
        return {
            "document_type": "UNKNOWN",
            "confidence": 0.0,
            "extracted_metadata": {},
            "anomalies": [f"JSON parse error: {str(e)}"],
            "classification_rationale": "Classification failed due to malformed LLM output",
            "requires_human_review": True,
            "review_reason": "LLM returned non-parseable output",
            "_meta": {"file_name": extraction_result["file_name"], "error": str(e)}
        }
    
    except Exception as e:
        return {
            "document_type": "UNKNOWN",
            "confidence": 0.0,
            "extracted_metadata": {},
            "anomalies": [f"Classification error: {str(e)}"],
            "classification_rationale": "Classification failed due to system error",
            "requires_human_review": True,
            "review_reason": f"System error: {str(e)}",
            "_meta": {"file_name": extraction_result["file_name"], "error": str(e)}
        }

Notice temperature=0.0. Classification is a task that benefits from maximum determinism. You're not asking the model to be creative; you're asking it to make a consistent judgment. The implications of temperature on model behavior are explained in depth in the lesson on understanding AI model temperature and parameters.


Layer 3: The Rule Engine

The classification step tells you what the document is. The rule engine tells you what to do with it. These are separate concerns and they should live in separate code.

from dataclasses import dataclass
from typing import Optional, Callable
from enum import Enum

class RoutingAction(Enum):
    AUTO_PROCESS = "auto_process"
    MANUAL_REVIEW = "manual_review"
    REJECT = "reject"
    ESCALATE = "escalate"

@dataclass
class RoutingDecision:
    action: RoutingAction
    destination: str
    priority: int  # 1=low, 2=normal, 3=high, 4=urgent
    reason: str
    document_type: str
    confidence: float
    extracted_metadata: dict
    flags: list

class DocumentRouter:
    """
    Deterministic rule engine that converts LLM classification output
    into concrete routing decisions.
    """
    
    def __init__(self, taxonomy: dict, confidence_thresholds: dict = None):
        self.taxonomy = taxonomy
        self.confidence_thresholds = confidence_thresholds or {
            "auto_process": 0.85,
            "manual_review": 0.5,
            # Below 0.5 -> UNKNOWN -> manual_review regardless
        }
        self._custom_rules: list[Callable] = []
    
    def add_rule(self, rule_fn: Callable) -> None:
        """Register a custom business rule function."""
        self._custom_rules.append(rule_fn)
    
    def route(self, classification: dict) -> RoutingDecision:
        """Apply routing rules to a classification result."""
        
        doc_type = classification.get("document_type", "UNKNOWN")
        confidence = float(classification.get("confidence", 0))
        metadata = classification.get("extracted_metadata", {})
        anomalies = classification.get("anomalies", [])
        requires_review = classification.get("requires_human_review", False)
        flags = list(anomalies)
        
        # --- Rule 1: Explicit human review flag from classifier ---
        if requires_review:
            return RoutingDecision(
                action=RoutingAction.MANUAL_REVIEW,
                destination="manual_review_queue",
                priority=2,
                reason=classification.get("review_reason", "Classifier flagged for review"),
                document_type=doc_type,
                confidence=confidence,
                extracted_metadata=metadata,
                flags=flags
            )
        
        # --- Rule 2: Unknown document type ---
        if doc_type == "UNKNOWN":
            return RoutingDecision(
                action=RoutingAction.MANUAL_REVIEW,
                destination="manual_review_queue",
                priority=2,
                reason="Document type could not be determined",
                document_type=doc_type,
                confidence=confidence,
                extracted_metadata=metadata,
                flags=flags
            )
        
        # --- Rule 3: Confidence below auto-process threshold ---
        if confidence < self.confidence_thresholds["auto_process"]:
            return RoutingDecision(
                action=RoutingAction.MANUAL_REVIEW,
                destination="manual_review_queue",
                priority=2,
                reason=f"Confidence {confidence:.2f} below threshold {self.confidence_thresholds['auto_process']}",
                document_type=doc_type,
                confidence=confidence,
                extracted_metadata=metadata,
                flags=flags
            )
        
        # --- Rule 4: High-value invoice escalation ---
        if doc_type == "VENDOR_INVOICE":
            amount_str = metadata.get("total_amount", "0")
            try:
                # Strip currency symbols and commas
                amount = float(re.sub(r'[^\d.]', '', str(amount_str)))
                if amount > 50000:
                    flags.append(f"High-value invoice: {amount_str}")
                    return RoutingDecision(
                        action=RoutingAction.ESCALATE,
                        destination="ap_approval_queue",
                        priority=3,
                        reason=f"Invoice value {amount_str} exceeds $50K approval threshold",
                        document_type=doc_type,
                        confidence=confidence,
                        extracted_metadata=metadata,
                        flags=flags
                    )
            except (ValueError, TypeError):
                flags.append("Could not parse invoice amount for threshold check")
        
        # --- Rule 5: Legal documents always get human review regardless of confidence ---
        if doc_type in ("VENDOR_CONTRACT", "CUSTOMER_CONTRACT"):
            return RoutingDecision(
                action=RoutingAction.MANUAL_REVIEW,
                destination=self.taxonomy[doc_type]["routing_destination"],
                priority=3,
                reason="All contracts require legal team review per policy",
                document_type=doc_type,
                confidence=confidence,
                extracted_metadata=metadata,
                flags=flags
            )
        
        # --- Rule 6: Apply custom rules registered by callers ---
        for rule_fn in self._custom_rules:
            result = rule_fn(doc_type, confidence, metadata, flags)
            if result is not None:
                return result
        
        # --- Default: Auto-process to designated pipeline ---
        destination = self.taxonomy[doc_type]["routing_destination"]
        return RoutingDecision(
            action=RoutingAction.AUTO_PROCESS,
            destination=destination,
            priority=2,
            reason=f"Classified as {doc_type} with confidence {confidence:.2f}",
            document_type=doc_type,
            confidence=confidence,
            extracted_metadata=metadata,
            flags=flags
        )

This design gives you something powerful: business rules are readable, testable functions in Python code. When the CFO says "actually, all invoices over $25K need approval," you change one number in one place. When compliance says "all GDPR-related documents need to go to the DPO queue," you add a rule that checks for GDPR keywords in the metadata. No prompt re-engineering required.


Layer 4: Dispatch and Observability

A routing decision sitting in memory isn't useful. You need to actually dispatch it and log enough information to know whether your system is working.

import logging
import datetime
from typing import Union
import json

logger = logging.getLogger(__name__)

class RoutingDispatcher:
    """
    Sends routing decisions to their destinations and records outcomes.
    In production, replace the stub methods with real queue/webhook calls.
    """
    
    def __init__(self, audit_log_path: str = "routing_audit.jsonl"):
        self.audit_log_path = audit_log_path
    
    def dispatch(self, 
                 routing_decision: RoutingDecision, 
                 extraction_result: dict,
                 classification: dict) -> dict:
        """
        Execute the routing decision and log the outcome.
        Returns a dispatch receipt.
        """
        
        receipt = {
            "timestamp": datetime.datetime.utcnow().isoformat(),
            "file_name": extraction_result["file_name"],
            "document_type": routing_decision.document_type,
            "confidence": routing_decision.confidence,
            "action": routing_decision.action.value,
            "destination": routing_decision.destination,
            "priority": routing_decision.priority,
            "reason": routing_decision.reason,
            "flags": routing_decision.flags,
            "metadata": routing_decision.extracted_metadata,
            "llm_model": classification.get("_meta", {}).get("model"),
            "prompt_tokens": classification.get("_meta", {}).get("prompt_tokens"),
            "dispatch_success": False,
            "dispatch_error": None
        }
        
        try:
            if routing_decision.action == RoutingAction.AUTO_PROCESS:
                self._send_to_pipeline(routing_decision)
            elif routing_decision.action == RoutingAction.MANUAL_REVIEW:
                self._send_to_review_queue(routing_decision)
            elif routing_decision.action == RoutingAction.ESCALATE:
                self._send_to_escalation(routing_decision)
            elif routing_decision.action == RoutingAction.REJECT:
                self._handle_rejection(routing_decision)
            
            receipt["dispatch_success"] = True
            
        except Exception as e:
            receipt["dispatch_error"] = str(e)
            logger.error(f"Dispatch failed for {extraction_result['file_name']}: {e}")
        
        # Always write to audit log regardless of dispatch success
        self._write_audit_log(receipt)
        
        return receipt
    
    def _send_to_pipeline(self, decision: RoutingDecision):
        """Stub: Replace with actual queue message, webhook call, or DB write."""
        logger.info(f"AUTO_PROCESS -> {decision.destination} | {decision.document_type}")
        # e.g., sqs_client.send_message(QueueUrl=QUEUES[decision.destination], ...)
    
    def _send_to_review_queue(self, decision: RoutingDecision):
        """Stub: Replace with actual manual review system integration."""
        logger.info(f"MANUAL_REVIEW -> {decision.destination} | Priority: {decision.priority}")
    
    def _send_to_escalation(self, decision: RoutingDecision):
        """Stub: Replace with escalation workflow trigger."""
        logger.warning(f"ESCALATE -> {decision.destination} | {decision.reason}")
    
    def _handle_rejection(self, decision: RoutingDecision):
        """Stub: Replace with rejection notification logic."""
        logger.warning(f"REJECT: {decision.reason}")
    
    def _write_audit_log(self, receipt: dict):
        """Append receipt to JSONL audit log."""
        with open(self.audit_log_path, "a", encoding="utf-8") as f:
            f.write(json.dumps(receipt) + "\n")

That .jsonl audit log is your most important debugging tool. Every document that passes through your system leaves a trail: what it was classified as, with what confidence, what the LLM extracted, what routing decision was made, and whether dispatch succeeded. When a contract ends up in the AP queue by mistake, you open the audit log and find out exactly what happened.

Key insight

Log the LLM's raw rationale string in your audit records. When you're reviewing a batch of mis-routed documents two weeks later, the LLM's one-sentence explanation of why it thought a contract was an invoice is infinitely more useful than a bare confidence score.


Putting the Pipeline Together

Here's the complete orchestration layer that wires all four components together:

def process_document(file_path: str, 
                     taxonomy: dict,
                     router: DocumentRouter,
                     dispatcher: RoutingDispatcher) -> dict:
    """
    Full pipeline: extract -> classify -> route -> dispatch.
    Returns a summary of what happened.
    """
    
    print(f"\nProcessing: {file_path}")
    
    # Layer 1: Extract
    extraction = extract_text(file_path)
    if extraction["extraction_warnings"]:
        print(f"  Extraction warnings: {extraction['extraction_warnings']}")
    
    # Layer 2: Classify
    classification = classify_document(extraction, taxonomy)
    print(f"  Classification: {classification['document_type']} "
          f"(confidence: {classification['confidence']:.2f})")
    print(f"  Rationale: {classification.get('classification_rationale', 'N/A')}")
    
    # Layer 3: Route
    decision = router.route(classification)
    print(f"  Routing: {decision.action.value} -> {decision.destination}")
    
    # Layer 4: Dispatch
    receipt = dispatcher.dispatch(decision, extraction, classification)
    status = "SUCCESS" if receipt["dispatch_success"] else "FAILED"
    print(f"  Dispatch: {status}")
    
    return receipt


# Example usage
if __name__ == "__main__":
    router = DocumentRouter(taxonomy=DOCUMENT_TAXONOMY)
    dispatcher = RoutingDispatcher(audit_log_path="routing_audit.jsonl")
    
    # Add a custom business rule: flagging any document mentioning NDA
    def nda_escalation_rule(doc_type, confidence, metadata, flags):
        if any("nda" in flag.lower() or "non-disclosure" in flag.lower() 
               for flag in flags):
            return RoutingDecision(
                action=RoutingAction.ESCALATE,
                destination="legal_review_queue",
                priority=4,
                reason="Document contains NDA references requiring legal review",
                document_type=doc_type,
                confidence=confidence,
                extracted_metadata=metadata,
                flags=flags + ["NDA detected"]
            )
        return None  # Rule doesn't apply, continue to next rule
    
    router.add_rule(nda_escalation_rule)
    
    # Process a batch of documents
    test_files = [
        "documents/acme_invoice_march.pdf",
        "documents/vendor_msa_2024.docx",
        "documents/support_ticket_export.txt",
        "documents/budget_forecast_q2.xlsx"
    ]
    
    for file_path in test_files:
        receipt = process_document(file_path, DOCUMENT_TAXONOMY, router, dispatcher)

Improving Accuracy with Few-Shot Examples

Zero-shot classification works reasonably well for unambiguous document types. It breaks down on domain-specific documents where the distinction between types requires business context that isn't obvious from the text alone. The difference between a "Statement of Work" and a "Project Charter" might be crystal clear to your legal team and completely opaque to a general-purpose LLM.

The solution is few-shot examples embedded in your classification prompt. The mechanics of this approach — including when to use them and how many to include — are covered in the lesson on few-shot and zero-shot prompting.

For routing systems specifically, you want contrastive examples: pairs of documents that look similar but should be routed differently. These teach the model the discriminating features.

FEW_SHOT_EXAMPLES = [
    {
        "document_excerpt": """STATEMENT OF WORK
Agreement between Meridian Analytics Inc. ("Client") and DataBridge Solutions LLC ("Vendor").
This SOW is entered into pursuant to the Master Service Agreement dated January 15, 2024.
Scope: Development of custom ETL pipeline for financial data consolidation.
Timeline: 12 weeks from project kickoff.
Total Fixed Fee: $84,000, payable in three equal installments.""",
        "correct_classification": {
            "document_type": "VENDOR_CONTRACT",
            "confidence": 0.92,
            "classification_rationale": "Statement of Work is a contractual document governing vendor deliverables, not a payment request. Despite mentioning a fee, it establishes obligations rather than requesting immediate payment."
        }
    },
    {
        "document_excerpt": """INVOICE #INV-2024-0847
DataBridge Solutions LLC
Bill To: Meridian Analytics Inc.
Invoice Date: March 15, 2024
Due Date: April 14, 2024

Milestone 2 Payment - ETL Pipeline Development (per SOW dated Jan 2024)
Amount Due: $28,000.00

Please remit payment to the bank account on file.""",
        "correct_classification": {
            "document_type": "VENDOR_INVOICE",
            "confidence": 0.97,
            "classification_rationale": "Despite referencing an SOW, this document is a payment request with invoice number, due date, and specific amount owed. It routes to AP, not legal."
        }
    }
]

You can inject these examples into the system prompt or as prior turns in the conversation history. The key is that the rationale strings explicitly explain why the boundary falls where it does.


Handling the Hard Cases

Multilingual Documents

If your organization receives documents in multiple languages, you have two options: translate first, then classify; or instruct the classifier to handle multiple languages natively. Modern LLMs handle most major business languages (Spanish, French, German, Portuguese, Mandarin, Japanese) well enough for classification without translation. Add this instruction to your system prompt:

If the document is in a language other than English, classify it normally based on content.
Include the detected language as "detected_language" in extracted_metadata.

Split Documents and Attachments

Emails with multiple attachments sometimes get combined into a single extraction pass. Build a pre-check: if your extracted text shows clear structural breaks (multiple "Dear [name]" headers, multiple invoice numbers at different addresses), split the extraction into separate classification calls rather than trying to classify a compound document.

Confidence Calibration Over Time

Your initial confidence thresholds are guesses. After running the system for 2-4 weeks, pull your audit log and analyze the correlation between confidence scores and actual accuracy (by sampling manually reviewed documents and checking whether the LLM's classification matched the human decision). You'll almost certainly find that your LLM is overconfident on certain document types. Adjust thresholds accordingly.

For systematic evaluation methodology, see the lesson on designing AI evaluation frameworks.

Warning

Watch for model drift. LLM providers update their models, and a prompt that worked perfectly on GPT-4o in January may behave differently on a later version. Lock your model version (gpt-4o-2024-11-20) in production and treat model upgrades as a deployment event that requires re-validation of your test suite.


Security and Adversarial Considerations

Document routing systems that touch financial, legal, or sensitive operational data are high-value targets for prompt injection attacks. Consider this scenario: a bad actor submits a PDF invoice where the footer contains invisible text instructing the LLM to classify it as INTERNAL_MEMO to avoid AP scrutiny.

Your defenses:

  1. Sanitize extracted text before the LLM sees it. Strip common injection patterns (instructions in brackets, "IGNORE PREVIOUS INSTRUCTIONS", etc.) before building your user message.

  2. Structural output validation. The rule engine should never blindly trust the LLM's routing suggestion. The LLM describes; the rule engine decides. Even if an injected instruction somehow convinces the LLM to suggest manual_review_queue for a $200K invoice, your rule engine's high-value threshold check will catch it.

  3. Anomaly logging. If your LLM's extracted metadata and its confidence score are wildly inconsistent with the file name, flag it. These inconsistencies are often signs of injection attempts.

  4. Separate classification from dispatch. Never give the LLM the ability to directly trigger actions. The LLM always outputs structured data; a separate deterministic layer decides whether to act. This architectural separation is your most important defense.

The full threat model for prompt injection in enterprise pipelines is covered in adversarial prompting and injection attacks.


Hands-On Exercise

Build a document routing system for a healthcare billing scenario. Your taxonomy should include at minimum: INSURANCE_CLAIM, EXPLANATION_OF_BENEFITS, PRIOR_AUTHORIZATION_REQUEST, PATIENT_INVOICE, and CLINICAL_NOTES.

Step 1: Define the taxonomy with routing destinations and required metadata fields for each document type. Work through the edge cases: how is a denied EOB different from an approved one? What metadata determines routing in each case?

Step 2: Write the classification system prompt using the template structure from this lesson. Add at least three few-shot examples that handle boundary cases your team would actually encounter.

Step 3: Implement a custom rule for the rule engine: PRIOR_AUTHORIZATION_REQUEST documents with an urgency metadata field set to "STAT" or "urgent" should route to an escalation queue with priority 4, bypassing the standard confidence threshold check.

Step 4: Process five test documents (create synthetic text if needed) and review the audit log. For each document, ask: does the routing decision match what a human processor would have done? If not, was the failure at the extraction layer, the classification layer, or the rule layer?

Step 5: Calculate your system's routing accuracy on the test set. Identify one change to the prompt, the taxonomy, or the rules that would improve it.


Common Mistakes & Troubleshooting

Mistake 1: Putting business rules in the prompt. If your classification prompt says "If the invoice amount is over $50,000, return destination: escalation_queue," you've made your business rules invisible to your engineering team, impossible to unit test, and subject to LLM inconsistency. Business rules belong in code. Always.

Mistake 2: Ignoring the confidence score in routing. Routing every classification at face value regardless of confidence is the most common cause of costly misrouting. A confidence of 0.6 means "I think this is probably a contract but I'm not sure." Treat it that way.

Mistake 3: Extracting the entire document for classification. Sending 50 pages of a contract to the LLM for classification wastes tokens and doesn't improve accuracy. Classification is determined by the first 1-2 pages in 90% of cases. Use sampling strategies: first page + last page + a sampled middle page works well for most document types.

Mistake 4: Not logging LLM rationale. You will have misrouting incidents. Without the LLM's reasoning string in your audit log, you have no way to know whether the failure was a prompt problem, a document quality problem, or a threshold calibration problem. Log everything.

Mistake 5: Treating the taxonomy as fixed. Your document taxonomy will evolve. New document types emerge. Business rules change. Build your system so that the taxonomy is a data structure (like our DOCUMENT_TAXONOMY dict), not hardcoded into prompts or rule conditions. Changes to the taxonomy should require zero prompt edits.

Mistake 6: Not testing the failure modes. Test what happens when the LLM returns invalid JSON. Test what happens when the PDF is completely blank. Test what happens when two document types appear in one file. Your error handling paths should be as well-exercised as your happy path.

Note

When debugging classification failures, the most efficient approach is to run the failing document through the classification prompt interactively in your LLM's playground UI. You can inspect the raw output, adjust the prompt, and iterate quickly before changing production code.


Summary & Next Steps

You've built a complete LLM-powered document routing system with four distinct layers: deterministic text extraction, LLM-based content understanding, rule-based business logic, and audited dispatch. The architectural principle that holds it together is separation of concerns — the LLM describes documents, your rule engine makes decisions, and nothing gets dispatched without both layers agreeing.

The critical production practices we covered:

  • Sampling document text rather than sending full documents, to control token costs
  • Requiring structured JSON output with confidence scores, not freeform text
  • Keeping business rules in Python, not in prompts
  • Logging LLM rationale alongside routing decisions for auditability
  • Treating low-confidence classifications as triggers for human review, not auto-routing
  • Hardening against prompt injection through architectural separation

Where to go from here:

The natural evolution of this system is toward a more agentic architecture where the routing system can take additional actions: querying a CRM to look up a customer mentioned in a document, calling an external API to validate a vendor's tax ID before processing an invoice, or summarizing a document's key terms before routing it for legal review. That architecture is covered in agentic AI workflows.

For handling documents with complex visual layouts — forms, tables, images embedded in PDFs — you'll want to extend the extraction layer with multimodal inputs. The techniques for that are in multimodal prompting.

As your volume scales, the per-call LLM cost will become material. Semantic caching (caching classification results for documents that are near-duplicates of previously processed files) can cut costs by 30-60% in high-volume scenarios. That's covered in semantic caching and vector search for LLM applications.

Build the audit log from day one, run a weekly accuracy review for the first month, and treat your first three months of production data as a calibration dataset. By the time you've processed a few thousand documents, you'll have enough ground truth to make your confidence thresholds genuinely data-driven rather than intuition-driven — and that's when this system starts to feel genuinely reliable.

Work With Us

From insight to implementation

Reading is the start. When you're ready to build the data, automation, or AI systems behind it, our team turns strategy into shipped results.

Let's Build

Intro to AI & Prompt Engineering

Previous

Tone and Voice Control in AI Prompts: How to Adjust Formality, Style, and Persona for Different Business Audiences

Related Insights

AI & Machine LearningExpert

Implementing Plan-and-Execute Agents: Separating Task Planning from Execution to Build Reliable Long-Horizon LLM Workflows

31 min
AI & Machine LearningPractitioner

Building a LangGraph Agent with Persistent State: Implementing Nodes, Edges, and Conditional Branching for Multi-Step AI Workflows

22 min
AI & Machine LearningPractitioner

Tone and Voice Control in AI Prompts: How to Adjust Formality, Style, and Persona for Different Business Audiences

24 min

On this page

  • Introduction
  • Prerequisites
  • The Architecture Before the Code
  • Layer 1: Document Ingestion and Text Extraction
  • Extracting Text from Common Formats
  • Layer 2: Designing the Classification Prompt
  • Defining Your Document Taxonomy
  • Writing the Classification Prompt
  • Calling the LLM and Parsing the Response
  • Layer 3: The Rule Engine
  • Layer 4: Dispatch and Observability
  • Putting the Pipeline Together
  • Improving Accuracy with Few-Shot Examples
  • Handling the Hard Cases
  • Multilingual Documents
  • Split Documents and Attachments
  • Confidence Calibration Over Time
  • Security and Adversarial Considerations
  • Hands-On Exercise
  • Common Mistakes & Troubleshooting
  • Summary & Next Steps