The Platform That Keeps Evolving — And Why That’s a Good Thing

If you’ve been tracking Google’s AI platform story, you’ve watched a rapid-fire succession of rebrands: Dialogflow → Agent Builder → Vertex AI → now Gemini Enterprise Agent Platform. At Google Cloud Next 2026, Google announced the consolidation of everything — Vertex AI, Agentspace, Model Garden, ADK, and the Agent Runtime — into a single unified platform. The low-code builder that was called Agent Designer since December 2024 became Agent Studio, now generally available.

This guide cuts through the naming history and focuses on what you can actually build today: production-grade agents using the full platform stack — Agent Studio for no-code/low-code design, RAG Engine for grounding on enterprise data, Memory Bank for long-term personalisation, Agent Runtime for deployment, and built-in evaluation for quality assurance.

Whether you’re a developer who wants code, a builder who wants clicks, or an architect who needs to understand the full system — this guide covers all three.

Part 1: The Platform Mental Model — Five Layers

Before touching the console or writing a line of code, understand how the five layers of the Gemini Enterprise Agent Platform fit together.

Gemini Enterprise Agent Platform is a unified platform to build, deploy, govern, and optimize enterprise-grade AI agents and model-based solutions. It supports the complete AI lifecycle — from accessing over 200 foundation models to deploying and managing your agents.

Here’s how the five layers stack:

┌──────────────────────────────────────────────────────────────────┐
│  LAYER 1 — AGENT STUDIO (no-code / low-code visual canvas)        │
│  Design agents, test prompts, build reasoning flows visually      │
├──────────────────────────────────────────────────────────────────┤
│  LAYER 2 — ADK (code-first agent framework)                       │
│  LlmAgent, SequentialAgent, ParallelAgent, LoopAgent, AgentTool  │
├──────────────────────────────────────────────────────────────────┤
│  LAYER 3 — KNOWLEDGE LAYER                                        │
│  RAG Engine · Agent Search · Vector Search · Memory Bank         │
├──────────────────────────────────────────────────────────────────┤
│  LAYER 4 — AGENT RUNTIME (managed deployment + scaling)           │
│  Agent Engine (Vertex AI) · Cloud Run · GKE                      │
├──────────────────────────────────────────────────────────────────┤
│  LAYER 5 — GOVERNANCE                                             │
│  Agent Identity · IAM · Agent Gateway · Business Policies        │
└──────────────────────────────────────────────────────────────────┘

Agent Platform meets you where you are, with tools for all skill levels: Agent Studio to design agents and interact with models without code; Colab Enterprise Notebooks for code-based development and experimentation; Agent Development Kit to build sophisticated agents capable of complex reasoning and tool use with a modular, model-agnostic framework.

The platform’s philosophy: start in Agent Studio, graduate to ADK code when you need more control, deploy both the same way via Agent Runtime.

Part 2: Agent Studio — The No-Code/Low-Code Canvas

Agent Studio is where most teams start. It’s a visual canvas inside the Google Cloud console for designing, prototyping, and managing agent reasoning loops and workflows — no Python required to get something running.

What Agent Studio Actually Is

Agent Studio, Google’s new low-code interface for building, testing, and publishing natural-language agents, is generally available. The product was in preview as Agent Designer since December 2024. What may be more interesting here is what developers can now actually build with it.

In the console, Agent Studio gives you:

Visual reasoning loop designer — drag connections between the model, tools, and data sources. Define the agent’s instruction (system prompt) in a structured editor with variable interpolation support.

Live test panel — chat with your agent directly in the console. Every tool call, retrieval step, and model response is visible in the trace panel alongside the conversation.

Tool connection UI — connect Google Search grounding, Agent Search corpora, Cloud Functions, OpenAPI specs, or MCP servers as tools — all without writing integration code.

Agent Garden integration — one-click import of prebuilt templates for common use cases: customer support, document Q&A, IT helpdesk, HR FAQ, code assistant.

Your First Agent in Agent Studio — Step by Step

Step 1: Open the console. Navigate to console.cloud.google.com, select your project, and search for “Agent Studio” in the top search bar. Or navigate directly: Agent Platform → Studio → Create Agent.

Step 2: Configure the agent basics. Give the agent a name (e.g. policy-assistant), select a model (gemini-2.0-flash for speed, gemini-2.5-pro for complex reasoning), and write the instruction. Be specific:

You are an enterprise policy assistant for Acme Corp.
Your job is to answer employee questions about company policies accurately.
Always retrieve from the knowledge_base tool before answering.
Cite the document name and section in every response.
If the policy is not found, say so -- do not invent details.

Step 3: Add a tool. Click Add Tool → Agent Search → select your knowledge corpus (or create one). Agent Search becomes the knowledge_base tool the instruction references.

Step 4: Test in the live panel. Type a query: “What is the parental leave policy?” Watch the trace: model receives query → calls knowledge_base → retrieves 3 passages → generates grounded response with citation.

Step 5: Export to ADK. When ready for code-first control, click Export → ADK Python. Agent Studio generates the full LlmAgent definition as a Python file — ready to extend, version, and deploy via CI/CD.

Part 3: Agent Garden — Blueprints That Actually Work

Rather than starting from a blank canvas, Agent Garden gives you production-tested templates for the most common agent patterns.

Agent Garden is a library of prebuilt agents and templates to accelerate development.

The adk-samples repository hosts the open-source versions of these templates. Each one is a complete, runnable ADK project with tools, instructions, evaluation datasets, and deployment configs. Current highlights:

To use a template from the CLI:

# Install the Google ADK
pip install google-adk

# Clone the adk-samples repository
git clone https://github.com/google/adk-samples.git
cd adk-samples/python/agents/customer-service

# Run locally
adk run agent.py

# Inspect in the dev UI
adk web

Each sample is a working starting point, not a toy. The customer-service template handles order lookups, refund requests, escalation to human agents, and session memory — all wired and ready to customise.

Part 4: RAG Engine — Grounding Agents on Enterprise Data

The most powerful capability in the platform for enterprise deployments is RAG Engine: a fully managed data framework for connecting private enterprise data to LLM agents.

RAG Engine on Gemini Enterprise Agent Platform is a data framework for building context-augmented LLM applications. Context augmentation occurs when you apply an LLM to your data. This implements retrieval-augmented generation (RAG).

RAG Engine handles the full pipeline: document ingestion, parsing, chunking, embedding, vector indexing, and retrieval — all managed, serverless, and integrated with the Gemini models.

Step 1: Create a RAG Corpus

A corpus is the container for your indexed documents. Create it once; it persists and auto-updates when you add new files.

# rag_setup.py
# pip install google-cloud-aiplatform

import vertexai
from vertexai.preview import rag

PROJECT_ID = "your-gcp-project-id"
LOCATION = "us-central1"

vertexai.init(project=PROJECT_ID, location=LOCATION)

# Create the corpus
corpus = rag.create_corpus(
    display_name="enterprise-knowledge-base",
    description="Internal policy docs, product manuals, and SOPs",
)
print(f"Corpus created: {corpus.name}")

Step 2: Import Documents

RAG Engine supports Google Cloud Storage, Google Drive, Google Docs, inline text, and Slack/Confluence via connectors. It automatically parses PDFs, Word docs, HTML, and plain text.

# rag_import.py
import vertexai
from vertexai.preview import rag

PROJECT_ID  = "your-gcp-project-id"
LOCATION    = "us-central1"
CORPUS_NAME = "projects/your-gcp-project-id/locations/us-central1/ragCorpora/YOUR_CORPUS_ID"

vertexai.init(project=PROJECT_ID, location=LOCATION)

# Import files from Google Cloud Storage
response = rag.import_files(
    corpus_name=CORPUS_NAME,
    paths=[
        "gs://your-bucket/docs/policy_manual_2025.pdf",
        "gs://your-bucket/docs/product_catalogue.pdf",
    ],
    transformation_config=rag.TransformationConfig(
        chunking_config=rag.ChunkingConfig(
            chunk_size=512,     # tokens per chunk
            chunk_overlap=100,  # overlap for context continuity
        ),
    ),
)
print(f"Files imported: {response.imported_rag_files_count}")

Step 3: Query with Gemini + RAG Tool

Attach the corpus as a retrieval tool and pass it to a Gemini model. Every generate_content call now retrieves before generating.

# rag_query.py
import vertexai
from vertexai.preview import rag
from vertexai.generative_models import GenerativeModel, Tool

PROJECT_ID  = "your-gcp-project-id"
LOCATION    = "us-central1"
CORPUS_NAME = "projects/your-gcp-project-id/locations/us-central1/ragCorpora/YOUR_CORPUS_ID"

vertexai.init(project=PROJECT_ID, location=LOCATION)

# Build the RAG retrieval tool
rag_retrieval_tool = Tool.from_retrieval(
    retrieval=rag.Retrieval(
        source=rag.VertexRagStore(
            rag_corpora=[CORPUS_NAME],
            similarity_top_k=5,           # return top 5 passages
            vector_distance_threshold=0.5, # filter below this similarity score
        ),
    )
)

# Attach to Gemini -- now every response is grounded in your documents
model = GenerativeModel(
    model_name="gemini-2.0-flash",
    tools=[rag_retrieval_tool],
)

response = model.generate_content(
    "What is our refund policy for enterprise software licences?"
)
print(response.text)

Step 4: RAG-Grounded ADK Agent

For multi-agent systems, wrap the RAG corpus as an ADK tool and give it to a specialist agent:

# rag_agent.py
import vertexai
from google.adk.agents import LlmAgent
from google.adk.tools import VertexAiRagRetrieval

PROJECT_ID  = "your-gcp-project-id"
LOCATION    = "us-central1"
CORPUS_NAME = "projects/your-gcp-project-id/locations/us-central1/ragCorpora/YOUR_CORPUS_ID"

vertexai.init(project=PROJECT_ID, location=LOCATION)

# Wrap the RAG corpus as an ADK retrieval tool
rag_tool = VertexAiRagRetrieval(
    name="knowledge_base",
    description="Searches internal documents: policies, SOPs, product specs.",
    rag_corpora=[CORPUS_NAME],
    similarity_top_k=5,
)

# Policy agent grounded in enterprise docs
policy_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="policy_agent",
    description="Answers questions about company policies and SOPs using the knowledge base.",
    instruction=(
        "You are an enterprise policy assistant. "
        "Always use the knowledge_base tool to retrieve relevant policies before answering. "
        "Cite the source document and page number in your response. "
        "Never make up policy details -- only reference retrieved content."
    ),
    tools=[rag_tool],
)

Reference: RAG Engine overview

Part 5: Agent Search — Out-of-the-Box Search for Specialised Domains

RAG Engine handles unstructured documents. Agent Search handles specialised retrieval needs at enterprise scale — with pre-tuned modes for different industry domains.

Agent Search functions as an out-of-the-box RAG system for information retrieval, and has a specialised offering tuned for unique industry requirements. The four modes map to distinct use cases:

Custom Search (General) builds tailored search, personalisation, and generative experiences on your sites, content, catalogues, and blended data. Data sources: structured catalogues (hotels, directories), unstructured files with metadata, Google Workspace connectors, and public sites. This is the go-to for internal knowledge base search where your data lives in Drive, Confluence, or GCS buckets.

Site Search with AI Mode builds generative search with AI mode in a day using site content. It leverages Google’s index for real-time crawling and adds search summarisation on top. The distinct advantage: you get Google’s crawling infrastructure without running your own spider. Ideal for documentation sites and product help centres that change frequently.

Media Search is designed for media libraries — images, videos, and audio files. This is purpose-built for broadcast, publishing, and creative industries where the asset itself (not just its metadata) needs to be searchable.

AI Commerce Search handles retail catalogues specifically. If you’re building search for an e-commerce platform, this mode is tuned for product discovery, faceted filtering, and purchase intent signals.

Create an Agent Search app from the console at Agent Platform → Agent Search → Create App, or via the Discoveryengine API:

# Create a search app via the CLI
gcloud alpha discovery-engine engines create \
  --project=YOUR_PROJECT_ID \
  --location=global \
  --display-name="internal-knowledge-search" \
  --solution-type=SOLUTION_TYPE_SEARCH \
  --data-store-ids=YOUR_DATA_STORE_ID

Part 6: Memory Bank — Long-Term Personalisation Across Sessions

RAG Engine grounds agents in documents. Memory Bank grounds agents in users — storing personalised facts, preferences, and context that persist across every session, indefinitely.

Memory Bank stores long-term memory containing personalised information to enable more context-aware agent interactions across multiple sessions. From the console you can view, search, and manage the agent’s saved memories — including total memory count, token usage, and mutation rates.

In code, attach Memory Bank to any ADK agent:

# memory_agent.py
from google.adk.agents import LlmAgent
from google.adk.memory import VertexAiMemoryBankService

# Memory Bank service -- backed by Vertex AI managed storage
memory_service = VertexAiMemoryBankService(
    project="your-gcp-project-id",
    location="us-central1",
)

# Agent with persistent memory across all user sessions
personalised_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="personalised_support_agent",
    description="Customer support agent with long-term memory of user preferences.",
    instruction=(
        "You are a helpful customer support agent. "
        "Remember the user's preferences, past issues, and account context. "
        "Use your memory to personalise every interaction. "
        "Always retrieve relevant memories before responding."
    ),
    memory_service=memory_service,
)

When a user says “I prefer email notifications, not SMS” in session 1, the agent writes that preference to Memory Bank. In session 47, three months later, the agent still knows it — without the user repeating themselves.

Note: As of January 2026, stored session events and memories are billed at $0.25 per 1,000 events or memories. Plan your retention policies accordingly.

Part 7: Deploying to Agent Runtime

Once your agent is built and tested, deploy it to Agent Runtime — the managed execution environment that handles auto-scaling, IAM, observability, and CI/CD integration.

The platform supports five deployment methods — choose based on your workflow:

The simplest path — deploying directly from an in-memory agent object — takes three lines after your agent is defined:

# deploy_agent.py
import vertexai
from google.adk.agents import LlmAgent

PROJECT_ID = "your-gcp-project-id"
LOCATION   = "us-central1"

vertexai.init(project=PROJECT_ID, location=LOCATION)

def get_order_status(order_id: str) -> dict:
    """Look up the current status of an order by its ID."""
    return {"order_id": order_id, "status": "shipped", "eta": "2025-07-15"}

support_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="support_agent",
    description="Handles customer order enquiries.",
    instruction="Help customers track their orders. Always use get_order_status.",
    tools=[get_order_status],
)

# Deploy to Agent Runtime -- three lines
from vertexai.preview.reasoning_engines import AdkApp

adk_app = AdkApp(agent=support_agent, enable_tracing=True)

remote_app = vertexai.preview.reasoning_engines.ReasoningEngine.create(
    adk_app,
    requirements=["google-adk>=1.0.0"],
    display_name="support-agent-v1",
    description="Customer support agent - order tracking",
)
print(f"Deployed: {remote_app.resource_name}")

After deployment, the agent is available as a REST endpoint, callable from any service with the right IAM permissions.

Reference: Deploy an agent on Agent Runtime

Part 8: Built-in Evaluation — Quality Before You Ship

Every agent needs evaluation before it reaches production. The Gemini Enterprise Agent Platform’s evaluation layer runs directly in the console (Evaluation tab) or via the Vertex AI SDK.

Three evaluation modes are available: Experiments for one-off quality assessments against a dataset, Metrics for defining and tracking custom quality dimensions, and Online Monitors for continuous evaluation in production.

Here’s a complete evaluation run using the SDK with a custom LLM-as-judge metric:

# evaluate_agent.py
import vertexai
from vertexai.preview.evaluation import EvalTask
from vertexai.preview.evaluation.metrics import (
    PointwiseMetric,
    PointwiseMetricPromptTemplate,
)

PROJECT_ID = "your-gcp-project-id"
LOCATION   = "us-central1"

vertexai.init(project=PROJECT_ID, location=LOCATION)

# Define a custom coherence metric using LLM-as-judge
coherence_metric = PointwiseMetric(
    metric="coherence",
    metric_prompt_template=PointwiseMetricPromptTemplate(
        criteria={
            "coherence": (
                "The response is logically structured, easy to follow, "
                "and the ideas connect naturally."
            )
        },
        rating_rubric={
            "5": "Perfectly coherent -- flows naturally, no gaps.",
            "3": "Mostly coherent with minor issues.",
            "1": "Incoherent -- hard to follow.",
        },
    ),
)

# Evaluation dataset (inputs + expected outputs)
eval_dataset = [
    {
        "prompt": "What is the refund policy for digital products?",
        "response": "Digital products are non-refundable unless the file is corrupted on delivery.",
        "reference": "Digital purchases are non-refundable except in cases of delivery errors.",
    },
    {
        "prompt": "How do I reset my password?",
        "response": "Go to the login page and click Forgot Password to receive a reset link by email.",
        "reference": "Click Forgot Password on the login page; a reset link will be emailed to you.",
    },
]

# Run the evaluation experiment
eval_task = EvalTask(
    dataset=eval_dataset,
    metrics=["exact_match", "rouge_l_sum", coherence_metric],
    experiment="support-agent-eval-v1",
)

eval_result = eval_task.evaluate()
print(eval_result.summary_metrics)

This experiment appears in the Agent Platform console under Evaluation → Experiments, where you can compare multiple runs side by side — exactly like the LangSmith experiment comparison we covered in the evaluation pillar post.

Reference: Evaluation on Agent Platform

Part 9: Governance — Policies, IAM, and Agent Gateway

Enterprise deployment isn’t complete without governance. The platform provides three governance layers.

Agent Identity gives each deployed agent its own service account identity — enabling fine-grained IAM permissions per agent. Your support agent can read from Firestore and call the orders API. It cannot write to BigQuery or access the HR database. Least privilege, enforced at the identity level.

Agent Gateway acts as the secure API layer between agents and the tools, MCP servers, and endpoints they call. It enforces IAM allow policies through Identity-Aware Proxy (IAP), controlling which agent identities can access which resources. Think of it as an API gateway that speaks agent — it understands tool calls, not just HTTP requests.

Business Policies (in the console at Policies → Business Policies) let you define natural-language rules that constrain agent behaviour across your organisation: “Agents must always disclose when they are AI.” “Agents must not discuss competitor pricing.” These are enforced at the Gateway layer, not in the individual agent instructions.

The Complete Platform Map

CONSOLE ENTRY POINTS
├── Agent Studio        → Visual agent designer, test, export to ADK
├── Agent Garden        → Prebuilt templates (customer-service, doc-QA, etc.)
├── RAG Engine          → Managed document indexing + retrieval
├── Agent Search        → Domain-specific search (general, site, media, commerce)
├── Memory Bank         → Long-term user personalisation
├── Agent Runtime       → Deploy, scale, monitor deployed agents
├── Evaluation          → Experiments, metrics, online monitors
└── Policies            → IAM, Agent Gateway, Business Policies

DEVELOPER ENTRY POINTS
├── ADK                 → Python/TypeScript/Go/Java agent framework
├── Colab Enterprise    → Notebooks with Vertex AI integration
├── Agents CLI          → adk run, adk web, adk eval, adk deploy
└── Developer Connect   → Git-linked CI/CD deployments

Where to Start

The right entry point depends on your team:

Non-technical teams building internal tools → start in Agent Studio, connect Agent Search to Google Drive, deploy to Agent Runtime with one click.

Developers building production agents → scaffold from Agent Garden, extend with ADK code, ground with RAG Engine, deploy from source files via the Agents CLI.

Enterprise architects designing multi-agent systems → use ADK for the agent layer, RAG Engine for knowledge, Memory Bank for personalisation, Agent Gateway for governance, and Agent Runtime for deployment across regions.

All three paths deploy to the same runtime, share the same evaluation tooling, and operate under the same governance layer. That’s the point of a unified platform.

Resources

• Gemini Enterprise Agent Platform overview — official home
• Agent Studio — Design agents — console visual designer
• Agent Garden — prebuilt templates
• ADK on Agent Platform — code-first development
• RAG Engine overview — managed retrieval framework
• RAG Engine quickstart — build your first corpus
• Deploy an agent on Agent Runtime — all five deployment methods
• Evaluation on Agent Platform — experiments, metrics, online monitors
• Agent Governance overview — IAM, Gateway, Business Policies
• adk-samples on GitHub — Agent Garden source templates
• Google Cloud Next 2026 Agent Platform announcement — the rebrand explained

All code examples syntax-verified against Python 3.11. Install: pip install google-adk google-cloud-aiplatform. Free tier available: up to 10 agent engines, 90 days via Vertex AI Express Mode.

The Day One Agent Problem

Every AI agent project starts with an optimistic prompt: “You are a smart assistant. Handle everything the user asks.”

Three weeks later, that single agent is juggling 40 tools, a system prompt that’s 3,000 tokens long, and a reliability rate that drops with every new capability you add. The more it knows, the worse it performs at any one thing.

This is the monolith trap. And the solution — like in software architecture — is decomposition.

Instead of one agent that does everything, build a team of specialists that each do one thing exceptionally well, coordinated by an orchestrator that knows how to delegate. That’s exactly what multi-agent systems are designed for.

Google’s Agent Development Kit (ADK) was built for this exact pattern. Announced at Google Cloud NEXT 2025 and now open-source, ADK is designed to simplify the full stack end-to-end development of agents and multi-agent systems, empowering developers to build production-ready agentic applications with greater flexibility and precise control. Critically, it’s the same framework Google uses internally — ADK is the same framework powering agents within Google products like Agentspace and the Google Customer Engagement Suite (CES).

This guide teaches you every concept you need, with working code at every step.

Part 1: Understanding ADK’s Architecture

Before writing code, internalize the mental model. ADK is built around a handful of clean primitives that compose naturally.

ADK is built around a few key primitives and concepts. The Agent is the fundamental worker unit designed for specific tasks. Agents can use language models (LlmAgent) for complex reasoning, or act as deterministic controllers of execution called workflow agents (SequentialAgent, ParallelAgent, LoopAgent). Tools give agents abilities beyond conversation, letting them interact with external APIs, search information, run code, or call other services.

The three agent types serve different roles:

The ADK empowers developers to get more reliable, sophisticated, multi-step behaviors from generative models. Instead of one complex prompt, ADK lets you build a flow of multiple, simpler agents that collaborate on a problem by dividing the work.

Why does this matter? Because specialized agents are more reliable at their specific tasks than one large, complex agent. It’s easier to fix or improve a small, specialized agent without breaking other parts of the system. Agents built for one workflow can be easily reused in others.

The Hierarchy Model

In ADK, you organize agents in a tree structure. A root coordinator sits at the top. Specialist sub-agents handle specific domains. Communication flows through three mechanisms: shared session state, LLM-driven delegation (agent transfer), and explicit invocation via AgentTool.

Root Coordinator (LlmAgent)
├── Specialist A (LlmAgent + tools)
├── Specialist B (LlmAgent + tools)
└── Workflow Orchestrator
    ├── Stage 1 Agent
    ├── Stage 2 Agent
    └── Stage 3 Agent

Part 2: Installation and Setup

ADK is available in Python, TypeScript, Go, and Java. We’ll use Python throughout.

# Create project and install ADK
mkdir travel-multi-agent && cd travel-multi-agent
python -m venv .venv && source .venv/bin/activate

pip install google-adk

# Set your Gemini API key
export GOOGLE_API_KEY="your_gemini_api_key_here"
# Get one free at: https://aistudio.google.com/app/apikey

Verify the install:

adk --version

ADK ships with a built-in developer UI you can launch for any project:

adk web          # Launches the visual debugger at http://localhost:8000
adk run          # CLI runner for scripted testing

The developer UI is one of ADK’s most practical advantages over other frameworks — every event, tool call, state change, and agent transfer is inspectable in real time without any extra instrumentation.

Part 3: Your First Agent — One LlmAgent with Tools

Let’s start minimal. A single LlmAgent with a tool teaches you the fundamental pattern before we add orchestration.

# agent.py
# pip install google-adk

import os
from google.adk.agents import LlmAgent
from google.adk.tools import google_search

# A minimal single agent
weather_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="weather_agent",
    description="Answers weather-related questions using Google Search.",
    instruction="""
    You are a helpful weather assistant.
    Always use the google_search tool to find current weather data.
    Provide concise, accurate answers including temperature, conditions,
    and any relevant weather warnings.
    """,
    tools=[google_search],
)

Run it:

adk run agent.py

Three things are worth noting here. First, model="gemini-2.0-flash" sets the LLM — ADK natively supports all Gemini variants, and via LiteLLM integration you can swap in Claude, Mistral, or any open model with one line. Second, description is what other agents read when deciding whether to delegate to this agent — it’s the sub-agent’s job posting. Third, instruction is the system prompt — be specific and prescriptive.

Part 4: Tool Design — Plain Python Functions

ADK’s cleanest design decision: any Python function with a docstring becomes a tool. The docstring is parsed into the tool’s schema and shown to the model. You don’t need wrappers, decorators, or SDK imports.

# tools.py

def search_flights(origin: str, destination: str, date: str) -> dict:
    """Search for available flights between two cities on a given date.
    
    Args:
        origin: Departure city (e.g. 'Mumbai')
        destination: Arrival city (e.g. 'London')
        date: Travel date in YYYY-MM-DD format
    
    Returns:
        dict with available flights and prices
    """
    # In production: wire to a real flights API (Amadeus, Skyscanner, etc.)
    return {
        "flights": [
            {"flight": "AI-101", "departure": "08:00", "price_usd": 850},
            {"flight": "AI-205", "departure": "14:30", "price_usd": 720},
        ],
        "origin": origin,
        "destination": destination,
        "date": date,
    }


def search_hotels(city: str, check_in: str, check_out: str) -> dict:
    """Search for hotels in a given city for given dates.
    
    Args:
        city: City name
        check_in: Check-in date YYYY-MM-DD
        check_out: Check-out date YYYY-MM-DD
    
    Returns:
        dict with available hotels and prices
    """
    return {
        "hotels": [
            {"name": "Grand Hotel", "stars": 5, "price_per_night_usd": 180},
            {"name": "City Suites", "stars": 4, "price_per_night_usd": 95},
        ],
        "city": city,
    }


# Each tool goes to the specialist that needs it — NOT to all agents
from google.adk.agents import LlmAgent

flight_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="flight_agent",
    description="Searches for available flights between cities.",
    instruction="You are a flights specialist. Use search_flights to find options.",
    tools=[search_flights],
)

hotel_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="hotel_agent",
    description="Finds and recommends hotel accommodations.",
    instruction="You are a hotel specialist. Use search_hotels to find options.",
    tools=[search_hotels],
)

The discipline here matters: give each tool to exactly the agent that needs it. Never give all tools to a coordinator. Tool overload is how monolith agents happen.

Part 5: AgentTool — Agents as Tools

The most powerful pattern in ADK: wrapping a sub-agent as a tool that the coordinator calls explicitly. This gives the coordinator full control over when each specialist runs, while keeping each specialist cleanly isolated.

# coordinator.py
from google.adk.agents import LlmAgent
from google.adk.tools.agent_tool import AgentTool

# (flight_agent and hotel_agent defined in tools.py above)

# Coordinator delegates to specialists via AgentTool
coordinator = LlmAgent(
    model="gemini-2.0-flash",
    name="travel_coordinator",
    description="Orchestrates travel planning by delegating to specialist agents.",
    instruction="""
    You are a travel planning coordinator.
    When users ask about travel:
    - Use the flight_agent tool for anything related to flights
    - Use the hotel_agent tool for anything related to accommodation
    - Synthesize both results into a coherent, complete travel plan
    - Present the plan clearly with costs and timings
    """,
    tools=[
        AgentTool(agent=flight_agent),
        AgentTool(agent=hotel_agent),
    ],
)

When the coordinator receives “Book a flight to Paris and find a hotel”, it calls flight_agent, gets the result, then calls hotel_agent, gets that result, and synthesises both into a unified response. This is a game-changer. When a complex query is run, the root agent understands and intelligently calls the flight tool, gets the result, and then calls the hotel tool.

Part 6: SequentialAgent — Guaranteed-Order Pipelines

Some workflows must run in strict order: you can’t summarise a document before fetching it. You can’t run a risk model before gathering market data. For these, SequentialAgent is the right primitive.

The SequentialAgent is a workflow agent that executes its sub-agents in the order they are specified in the list. Use the SequentialAgent when you want the execution to occur in a fixed, strict order.

Here’s an equity analyst pipeline — research → risk assessment → report generation, guaranteed in that order:

# analyst_pipeline.py
from google.adk.agents import LlmAgent, SequentialAgent

def fetch_market_data(ticker: str) -> dict:
    """Fetch latest market data for a stock ticker."""
    return {"ticker": ticker, "price": 142.50, "volume": 1_200_000, "change_pct": 2.3}

def run_risk_model(data: dict) -> dict:
    """Run risk assessment on market data."""
    return {"risk_score": 0.42, "recommendation": "moderate_buy", "data": data}


# Step 1: Research — writes to session state via output_key
research_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="research_agent",
    description="Fetches and structures market data for analysis.",
    instruction="""Fetch market data for the requested ticker.
    Return structured data including price, volume, and daily change.""",
    tools=[fetch_market_data],
    output_key="market_data",        # ← writes result to session state
)

# Step 2: Risk — reads {market_data} from session state
risk_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="risk_agent",
    description="Runs risk assessment on the researched market data.",
    instruction="""Read the market data from {market_data} in session state.
    Run a risk assessment and produce a structured recommendation.""",
    tools=[run_risk_model],
    output_key="risk_assessment",
)

# Step 3: Report — synthesises both outputs
report_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="report_agent",
    description="Generates the final analyst report.",
    instruction="""Using the market data from {market_data} and risk assessment
    from {risk_assessment}, write a concise investment report with:
    - Executive summary
    - Key metrics
    - Risk rating
    - Recommendation""",
)

# SequentialAgent: guaranteed order, no LLM routing overhead
analyst_pipeline = SequentialAgent(
    name="equity_analyst_pipeline",
    sub_agents=[research_agent, risk_agent, report_agent],
)

The output_key parameter is how agents communicate through session state — a lightweight shared memory available to all agents in the tree during a single session. Agent B can read what Agent A wrote simply by referencing {agent_a_output_key} in its instruction.

Part 7: ParallelAgent — Concurrent Specialist Teams

When sub-tasks are independent of each other, there’s no reason to run them serially. ParallelAgent runs all sub-agents concurrently and collects their results before returning.

# parallel_research.py
from google.adk.agents import LlmAgent, ParallelAgent

def search_flights(origin: str, destination: str, date: str) -> dict:
    """Search flights between two cities."""
    return {"flights": [{"flight": "AI-101", "price_usd": 850}]}

def search_hotels(city: str, check_in: str, check_out: str) -> dict:
    """Search hotels in a city."""
    return {"hotels": [{"name": "Grand Hotel", "price_per_night_usd": 180}]}

def search_activities(city: str, date: str) -> dict:
    """Search top activities in a city."""
    return {"activities": ["Eiffel Tower", "Louvre Museum", "Seine River Cruise"]}


flight_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="flight_agent",
    description="Searches for flights.",
    instruction="Find flights for the given route and date.",
    tools=[search_flights],
    output_key="flight_results",
)

hotel_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="hotel_agent",
    description="Finds hotels.",
    instruction="Find hotels for the given city and dates.",
    tools=[search_hotels],
    output_key="hotel_results",
)

activities_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="activities_agent",
    description="Finds things to do.",
    instruction="Find top activities and attractions for the given city.",
    tools=[search_activities],
    output_key="activities_results",
)

# ParallelAgent: all three run concurrently → 3x faster than sequential
research_team = ParallelAgent(
    name="travel_research_team",
    sub_agents=[flight_agent, hotel_agent, activities_agent],
)

Parallel research that previously took 9 seconds (3 sequential API calls at ~3s each) now takes ~3 seconds. For any multi-step workflow where steps are independent, ParallelAgent is the right choice.

Part 8: LoopAgent — Iterative Refinement (Generator-Critic)

Some outputs improve with iteration. A first-draft blog post benefits from a critic pass. A travel itinerary improves when checked against constraints. LoopAgent implements this generator-critic pattern: it loops through its sub-agents repeatedly until one of them triggers an escalate signal or max_iterations is reached.

# refinement_loop.py
from google.adk.agents import LlmAgent, LoopAgent

# Writer produces or revises the draft
writer_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="writer_agent",
    description="Writes or revises the content draft.",
    instruction="""
    If there is no draft yet, write an initial blog post based on the topic.
    If there is a draft in {current_draft}, revise it based on the critic's
    feedback in {critic_feedback}. Output the improved draft.
    """,
    output_key="current_draft",
)

# Critic reviews and decides whether to continue or finish
critic_agent = LlmAgent(
    model="gemini-2.0-flash",
    name="critic_agent",
    description="Reviews content quality and decides whether to continue iterating.",
    instruction="""
    Review the draft in {current_draft}. Score it from 1-10 for:
    clarity, accuracy, engagement, and SEO value.
    Provide specific, actionable improvement notes.
    If the overall score is 8 or above, set escalate=true to finish.
    Otherwise set escalate=false to request another revision.
    """,
    output_key="critic_feedback",
)

# Loops until escalate=true or max_iterations reached
content_refinement_loop = LoopAgent(
    name="content_refinement_loop",
    sub_agents=[writer_agent, critic_agent],
    max_iterations=5,
)

This maps directly onto production use cases: report generation with quality gates, code generation with test-run feedback, regulatory documents with compliance checks.

Part 9: The Complete Multi-Agent System

Now compose every pattern into one production system: a travel planner that runs research in parallel, refines the itinerary through a writer-critic loop, then validates before delivery.

# travel_planner.py — full production multi-agent system
from google.adk.agents import LlmAgent, SequentialAgent, ParallelAgent, LoopAgent
from google.adk.tools.agent_tool import AgentTool


# ── Tool functions ────────────────────────────────────────────────────────────

def search_flights(origin: str, destination: str, date: str) -> dict:
    """Search flights between two cities."""
    return {"flights": [{"flight": "AI-101", "price_usd": 850}]}

def search_hotels(city: str, check_in: str, check_out: str) -> dict:
    """Search hotels in a city."""
    return {"hotels": [{"name": "Grand Hotel", "price_per_night_usd": 180}]}

def search_activities(city: str, date: str) -> dict:
    """Search top attractions in a city."""
    return {"activities": ["Eiffel Tower", "Louvre Museum"]}

def validate_itinerary(itinerary: str) -> dict:
    """Validate an itinerary for conflicts and completeness."""
    return {"valid": True, "issues": []}


# ── Stage 1: Parallel research team ──────────────────────────────────────────

flight_agent    = LlmAgent(model="gemini-2.0-flash", name="flight_agent",
    description="Searches for available flights.",
    instruction="Find flights for the given route and date.",
    tools=[search_flights], output_key="flight_results")

hotel_agent     = LlmAgent(model="gemini-2.0-flash", name="hotel_agent",
    description="Finds hotels.",
    instruction="Find hotels for the city and dates.",
    tools=[search_hotels], output_key="hotel_results")

activities_agent = LlmAgent(model="gemini-2.0-flash", name="activities_agent",
    description="Recommends activities and attractions.",
    instruction="Find top activities for the city.",
    tools=[search_activities], output_key="activities_results")

research_team = ParallelAgent(
    name="research_team",
    sub_agents=[flight_agent, hotel_agent, activities_agent],
)

# ── Stage 2: Writer-critic refinement loop ────────────────────────────────────

writer_agent = LlmAgent(model="gemini-2.0-flash", name="itinerary_writer",
    description="Drafts a travel itinerary from research results.",
    instruction="""Using flight_results, hotel_results, and activities_results
    from session state, compose a detailed 3-day travel itinerary.
    On revision rounds, apply critic_feedback.""",
    output_key="itinerary_draft")

critic_agent = LlmAgent(model="gemini-2.0-flash", name="itinerary_critic",
    description="Reviews the itinerary for quality.",
    instruction="""Review the itinerary in {itinerary_draft}.
    Check for: logical flow, realistic timing, missing essentials.
    Score 1-10. If score >= 8, set escalate=true.""",
    output_key="critic_feedback")

refinement_loop = LoopAgent(
    name="itinerary_refinement",
    sub_agents=[writer_agent, critic_agent],
    max_iterations=3,
)

# ── Stage 3: Validation ───────────────────────────────────────────────────────

validator_agent = LlmAgent(model="gemini-2.0-flash", name="validator_agent",
    description="Validates the final itinerary.",
    instruction="""Validate the itinerary in {itinerary_draft} using the
    validate_itinerary tool. Return the validation result.""",
    tools=[validate_itinerary],
    output_key="validation_result")

# ── Full pipeline: Research → Refine → Validate ───────────────────────────────

travel_planner = SequentialAgent(
    name="travel_planner",
    sub_agents=[research_team, refinement_loop, validator_agent],
)

Run this with:

adk run travel_planner.py
# Or test with web UI:
adk web travel_planner.py

The architecture: Research (Parallel, 3x faster) → Refinement Loop (quality gates) → Validation (safety check) → Final output. Each stage is independently testable, swappable, and improvable without touching the others.

Part 10: Session State and Agent Communication

The mechanism agents use to pass data between each other in ADK is session state — a shared key-value store available within a single conversation session. output_key on an LlmAgent writes the agent’s final response to a state key. Any downstream agent can read it via {key_name} interpolation in its instruction.

This is the recommended pattern for SequentialAgent pipelines. For AgentTool invocations, the result is returned inline to the calling coordinator — no state write needed.

For cross-session persistence (memory that survives across different user conversations), ADK provides a Memory component separate from State. Think of State as session RAM and Memory as persistent storage.

Reference: Sessions & Memory — ADK Docs

Part 11: Running and Debugging

ADK’s developer tooling is one of its strongest differentiators.

# Run interactively in the terminal
adk run travel_planner.py

# Launch the visual dev UI (inspect events, state, tool calls)
adk web

# Evaluate against test datasets
adk eval travel_planner.py eval_dataset.json

The web UI shows every Event in the execution tree: which agent ran, which tools were called, what was written to state, and how long each step took. For multi-agent systems with 5+ agents, this is invaluable for debugging delegation failures and unexpected routing.

Part 12: Deployment

When your agent is production-ready, ADK provides first-class deployment to Google Cloud:

# Deploy to Vertex AI Agent Engine (managed, auto-scaling)
adk deploy agent-engine travel_planner.py

# Or containerise for Cloud Run
adk deploy cloud-run travel_planner.py --project YOUR_GCP_PROJECT

ADK’s architecture includes several production-focused features: direct integration with Vertex AI Agent Engine, support for containerised deployment, pre-built connectors to enterprise systems and databases like AlloyDB, BigQuery, and NetApp, bidirectional streaming support for real-time audio and video interactions, and built-in frameworks to assess response quality and execution paths.

References: Deploy to Agent Engine, Deploy to Cloud Run

The Architecture Mental Model

USER QUERY
     │
     ▼
┌─────────────────────────────────────────────────────────────┐
│  ROOT COORDINATOR (LlmAgent)                                │
│  Receives query → decides which agents/tools to invoke      │
└────────┬──────────────┬──────────────────────┬─────────────┘
         │              │                      │
         ▼              ▼                      ▼
  AgentTool A     AgentTool B           SequentialAgent
  (Specialist)    (Specialist)          └─ Step 1 Agent
                                        └─ Step 2 Agent
                                        └─ Step 3 Agent
                                                │
                                         ParallelAgent
                                         ├─ Worker A  ──┐
                                         ├─ Worker B  ──┤ → merged
                                         └─ Worker C  ──┘
                                                │
                                           LoopAgent
                                           ├─ Writer → draft
                                           └─ Critic → escalate?
                                                │
                                         FINAL RESPONSE

What You’ve Built

Walking through this guide, you’ve assembled the full ADK vocabulary: LlmAgent for reasoning specialists, SequentialAgent for guaranteed-order pipelines, ParallelAgent for concurrent research teams, LoopAgent for iterative refinement cycles, and AgentTool for explicit coordinator-to-specialist delegation.

The travel planner is a working template for any multi-agent system in production: research fast (parallel), draft well (loop), gate with quality checks (critic), validate before shipping (sequential). Swap the domain, adjust the tools, deploy to Vertex AI.

This is how Google builds its own production agent systems. Now it’s your framework too.

Resources

• ADK Official Documentation — home of all ADK guides
• ADK Python Quickstart — your first agent in 5 minutes
• Multi-Agent Systems in ADK — patterns and primitives
• Sequential Agents — guaranteed-order pipelines
• Parallel Agents — concurrent execution
• Loop Agents — iterative refinement
• Sessions & Memory — state and cross-session persistence
• Deploy to Agent Engine — Vertex AI deployment
• Google Cloud Blog: Build Multi-Agentic Systems
• ADK Technical Overview — deep dive on architecture

All code examples syntax-verified against Python 3.11. Install: pip install google-adk. Get a free Gemini API key at aistudio.google.com.

The Problem Nobody Talks About at Demo Time

Your agent demo looked flawless. It answered every question correctly, called the right tools in the right order, and finished in under three seconds. The audience applauded.

Two weeks after going live, your support queue is filling up with: “The agent gave me completely wrong information.” “It searched the wrong database.” “It hallucinated a date that doesn’t exist.”

Here’s the hard truth: demos don’t break agents. Real users do. And without a systematic evaluation framework, you will always be one bad production run away from a confidence crisis.

This guide teaches you everything you need: the metrics that matter, how to build evaluators from scratch, how LLM-as-a-judge works, and how LangSmith closes the loop from local testing all the way to production monitoring. We build each concept on the last, so by the end you’ll have a complete evaluation system you can deploy today.

Part 1: Foundations — What Does “Agent Quality” Actually Mean?

Before you can measure anything, you need a model of what you’re measuring.

An agent isn’t a static function. It’s a decision-making system that reasons, selects tools, retrieves data, and generates responses — often over multiple steps. Quality failure can happen at any of those layers.

Think of agent quality across four dimensions:

1. Output Quality

Does the final answer satisfy the user’s intent? Is it correct, relevant, and complete — without hallucinating facts?

2. Trajectory Quality

Did the agent take the right path to get there? Did it call the correct tools, in the correct order, without unnecessary detours?

3. Latency and Efficiency

How long did each step take? How many tokens were consumed? Are there runaway loops or redundant tool calls?

4. Safety and Guardrails

Did the agent stay within its defined scope? Did it avoid toxic, harmful, or out-of-policy outputs?

Each dimension needs its own evaluator. A single “pass/fail” score tells you almost nothing. Let’s build the measurement layer, dimension by dimension.

Part 2: The Metrics That Matter — What to Track

Here’s a practical taxonomy of agent evaluation metrics, drawn from production experience and the LangSmith evaluation framework.

Correctness (Output vs. Reference)

The baseline: does the agent’s answer match the expected answer?

This can be measured exactly (string match, JSON match) or approximately (semantic similarity, LLM judge). Use exact match for structured outputs (IDs, dates, classifications). Use LLM-as-judge for conversational or long-form outputs.

Groundedness / Faithfulness

Does the agent’s response stay grounded in the retrieved documents or tools it actually used? An agent that “knows” something it wasn’t given is hallucinating.

Per the LangSmith RAG evaluation guide, groundedness measures response vs. retrieved docs — not vs. a reference answer. This means you can evaluate it without ground truth.

Relevance

Does the answer actually address the user’s question? An agent can be perfectly faithful to its retrieved documents and still fail if it retrieved the wrong documents in the first place.

Track this at two levels: response relevance (answer vs. question) and retrieval relevance (retrieved docs vs. question).

Trajectory Accuracy

This is unique to agents. It asks: did the agent take the expected sequence of steps?

As the LangSmith evaluation approaches documentation explains, trajectory evaluation can target:

• Exact match — did the agent call tools A → B → C in exactly that order?
• Unordered match — did the agent call the right set of tools, in any order?
• Subset/superset — did the agent at least call the required minimum tools?
• LLM-judge over full trajectory — pass the entire message + tool call history to a judge for holistic assessment.

Latency (p50, p95, p99)

Track response time at the percentile level. p50 tells you typical performance. p95 and p99 tell you what your worst users experience. Looping agents or redundant tool calls show up here first.

Token Efficiency

Total tokens per run, tokens per tool call, and token cost per session. Useful for catching prompt bloat and runaway context growth in long-running agents.

Composite Quality Score

LangSmith supports composite evaluators that combine multiple scores into a single weighted metric. For example: Overall Quality = (70% × correctness) + (20% × relevance) + (10% × conciseness). Useful for dashboards and regression gates.

Part 3: Your First Evaluator — Code-Based Rules

Not everything needs an LLM to evaluate. Start simple.

A code-based evaluator is just a Python function. It receives the agent’s inputs, outputs, and optionally reference outputs — and returns a score.

# evaluators.py

def response_length_evaluator(inputs: dict, outputs: dict, reference_outputs: dict = None) -> dict:
    """
    A simple evaluator that checks whether the response is concise.
    Flags responses over 500 words.
    """
    word_count = len(outputs.get("answer", "").split())
    score = 1 if word_count <= 500 else 0
    return {
        "key": "conciseness",
        "score": score,
        "comment": f"Response length: {word_count} words"
    }


def json_format_evaluator(inputs: dict, outputs: dict, reference_outputs: dict = None) -> dict:
    """
    Checks that the agent returned valid, parseable JSON where expected.
    """
    import json
    try:
        json.loads(outputs.get("structured_output", ""))
        return {"key": "valid_json", "score": 1}
    except (json.JSONDecodeError, TypeError):
        return {"key": "valid_json", "score": 0, "comment": "Output is not valid JSON"}


def tool_call_count_evaluator(inputs: dict, outputs: dict, reference_outputs: dict = None) -> dict:
    """
    Checks that the agent didn't make an excessive number of tool calls (a sign of looping).
    """
    tool_calls = outputs.get("tool_calls", [])
    score = 1 if len(tool_calls) <= 5 else 0
    return {
        "key": "tool_efficiency",
        "score": score,
        "comment": f"Tool calls made: {len(tool_calls)}"
    }

These run instantly, cost nothing, and catch structural failures immediately. Use them as your first filter before investing in LLM-based evaluation.

Part 4: LLM-as-Judge — Evaluating What Rules Can’t

Some failures are semantic, not structural. An agent might return a perfectly formatted JSON with a factually wrong answer. A rule can’t catch that. An LLM judge can.

LLM-as-judge is the pattern where a second, independent LLM evaluates the output of your primary agent. The judge receives a structured prompt with the question, the agent’s answer, and optionally a reference answer — then returns a score and reasoning.

Here’s how the LangSmith evaluation quickstart describes the key components: inputs (what was passed to your agent), outputs (what your agent returned), and reference_outputs (the ground truth answers from your dataset).

Build a Custom LLM-as-Judge Evaluator

# llm_judge_evaluators.py
from langchain_anthropic import ChatAnthropic

judge_llm = ChatAnthropic(model="claude-sonnet-4-20250514", temperature=0)

def correctness_judge(inputs: dict, outputs: dict, reference_outputs: dict) -> dict:
    """
    LLM-as-judge evaluator for factual correctness.
    Compares agent answer against reference answer.
    Returns score 0 (incorrect) or 1 (correct) with reasoning.
    """
    prompt = f"""You are an expert evaluator assessing an AI agent's response.

Question asked: {inputs.get('question', '')}

Reference answer (ground truth): {reference_outputs.get('answer', '')}

Agent's answer: {outputs.get('answer', '')}

Your task: Assess whether the agent's answer is factually correct relative to the reference answer.
Respond in this exact format:
SCORE: [0 or 1]
REASONING: [one sentence explaining why]"""

    response = judge_llm.invoke(prompt)
    content = response.content

    score = 1 if "SCORE: 1" in content else 0
    reasoning = content.split("REASONING:")[-1].strip() if "REASONING:" in content else ""

    return {
        "key": "correctness",
        "score": score,
        "comment": reasoning
    }


def groundedness_judge(inputs: dict, outputs: dict, reference_outputs: dict = None) -> dict:
    """
    LLM-as-judge for groundedness: checks if the answer is supported
    by the retrieved context (no reference needed).
    """
    context = outputs.get("retrieved_context", "")
    answer = outputs.get("answer", "")

    if not context:
        return {"key": "groundedness", "score": 0, "comment": "No retrieved context found"}

    prompt = f"""You are grading whether an AI answer is grounded in retrieved documents.

Retrieved context:
{context}

AI answer:
{answer}

Return 1 if the answer is fully supported by the context.
Return 0 if the answer contains information NOT present in the context (hallucination).

SCORE: [0 or 1]
REASONING: [one sentence]"""

    response = judge_llm.invoke(prompt)
    content = response.content
    score = 1 if "SCORE: 1" in content else 0
    reasoning = content.split("REASONING:")[-1].strip() if "REASONING:" in content else ""

    return {"key": "groundedness", "score": score, "comment": reasoning}


def relevance_judge(inputs: dict, outputs: dict, reference_outputs: dict = None) -> dict:
    """
    Evaluates whether the agent's answer actually addresses the user's question.
    Reference-free: compares answer to input question only.
    """
    question = inputs.get("question", "")
    answer = outputs.get("answer", "")

    prompt = f"""Does the following answer directly address the question?

Question: {question}
Answer: {answer}

SCORE: 1 if relevant, 0 if off-topic or evasive
REASONING: [one sentence]"""

    response = judge_llm.invoke(prompt)
    content = response.content
    score = 1 if "SCORE: 1" in content else 0
    reasoning = content.split("REASONING:")[-1].strip() if "REASONING:" in content else ""

    return {"key": "relevance", "score": score, "comment": reasoning}

Using OpenEvals — Pre-Built Judges

For production use, the openevals library ships ready-made LLM-as-judge evaluators with battle-tested prompts:

# Using openevals for correctness (pip install openevals)
from openevals import create_llm_as_judge, CORRECTNESS_PROMPT, CONCISENESS_PROMPT

correctness_evaluator = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    model="anthropic:claude-sonnet-4-20250514",
    feedback_key="correctness",
)

conciseness_evaluator = create_llm_as_judge(
    prompt=CONCISENESS_PROMPT,
    model="anthropic:claude-sonnet-4-20250514",
    feedback_key="conciseness",
)

A word of caution: LLM judges don’t always get it right. LangSmith allows human auditors to review and correct evaluator scores — building a feedback loop that continuously improves judge accuracy over time. See how to audit evaluator scores.

Part 5: Trajectory Evaluation — Judging the Path, Not Just the Destination

For agents, the how matters as much as the what. An agent that arrives at the right answer after 12 unnecessary tool calls isn’t production-ready.

The agentevals package provides trajectory evaluators:

# trajectory_eval.py
# pip install agentevals langsmith

from agentevals import create_trajectory_match_evaluator
from langsmith import evaluate

# Define expected trajectory for a customer support query
reference_trajectory = [
    "retrieve_customer_profile",
    "check_order_status",
    "generate_response"
]

# Create a trajectory match evaluator in "unordered" mode
# (tools must all appear, but order flexible)
trajectory_evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="unordered"
)


def run_agent_and_track(inputs: dict) -> dict:
    """
    Wraps your agent to capture both the final response and the tool trajectory.
    In LangGraph, use astream with stream_mode='debug' to capture node names.
    """
    trajectory = []
    # Simulate agent run — in production wire to LangGraph streaming
    trajectory = ["retrieve_customer_profile", "check_order_status", "generate_response"]
    answer = "Your order #1234 is out for delivery and will arrive today."

    return {
        "answer": answer,
        "trajectory": trajectory
    }


# Run trajectory evaluation
results = evaluate(
    run_agent_and_track,
    data="customer-support-dataset",       # Your LangSmith dataset name
    evaluators=[trajectory_evaluator],
    experiment_prefix="support-agent-v2-trajectory",
)

Reference: Evaluating an agent’s trajectory, Trajectory match evaluator

Part 6: The Evaluation Framework — Putting It All Together

Now you have individual evaluators. Let’s wire them into a complete evaluation pipeline using LangSmith’s evaluate function.

Step 1: Create Your Dataset

A dataset is a collection of test examples — each with an input and an optional reference output. Build your first dataset from three sources:

• Manually curated golden examples (high signal)
• Historical production traces where the agent did well (realistic coverage)
• Synthetic variations generated by an LLM (breadth at scale)

from langsmith import Client

client = Client()

# Create a dataset
dataset = client.create_dataset(
    dataset_name="agent-quality-v1",
    description="Evaluation dataset for the customer support agent"
)

# Add examples
examples = [
    {
        "inputs": {"question": "What is the refund policy for digital products?"},
        "outputs": {"answer": "Digital products are non-refundable unless the file is corrupted."}
    },
    {
        "inputs": {"question": "How do I track my order?"},
        "outputs": {"answer": "Log in to your account, go to Orders, and click Track on the relevant order."}
    },
    {
        "inputs": {"question": "Can I change my shipping address after ordering?"},
        "outputs": {"answer": "You can change your address within 1 hour of placing the order by contacting support."}
    },
]

client.create_examples(
    inputs=[e["inputs"] for e in examples],
    outputs=[e["outputs"] for e in examples],
    dataset_id=dataset.id,
)

Step 2: Define the Target Function

# The function LangSmith will evaluate
def my_agent_target(inputs: dict) -> dict:
    """
    Your agent call wrapped in a target function.
    LangSmith passes each dataset example's input here.
    """
    from langchain_anthropic import ChatAnthropic

    model = ChatAnthropic(model="claude-sonnet-4-20250514", temperature=0)
    question = inputs.get("question", "")
    response = model.invoke(f"You are a helpful customer support agent.\n\nQuestion: {question}")
    return {"answer": response.content}

Step 3: Run the Full Evaluation

from langsmith import evaluate
# Import your evaluators from earlier sections
from evaluators import response_length_evaluator, json_format_evaluator
from llm_judge_evaluators import correctness_judge, relevance_judge

results = evaluate(
    my_agent_target,
    data="agent-quality-v1",
    evaluators=[
        correctness_judge,
        relevance_judge,
        response_length_evaluator,
    ],
    experiment_prefix="customer-support-v1",
    num_repetitions=1,        # Run each example once
    max_concurrency=4,        # Parallel evaluation for speed
)

print(f"Experiment complete. View at: {results.experiment_url}")

Reference: Evaluation quickstart — LangSmith, Run evals in LangSmith

Part 7: The LangSmith Platform — Closing the Loop

Everything above can run locally. But LangSmith is where evaluation becomes a continuous discipline rather than a one-time script.

What LangSmith Actually Is

LangSmith is a framework-agnostic platform for building, debugging, and deploying AI agents. It works with LangGraph, plain LangChain, OpenAI calls, and any other stack. You get tracing, evaluation, prompt management, and monitoring in one place.

The workflow is linear: Trace → Evaluate → Compare → Monitor → Improve.

Offline Evaluation: Test Before You Ship

The evaluate function runs your agent against a dataset and logs every result as an experiment in LangSmith. Each experiment shows:

• Per-example scores for every evaluator
• Aggregate pass rates across the dataset
• Side-by-side diff when you compare two experiments

Regression testing is where this becomes powerful. After every prompt change or model upgrade, run the same dataset. LangSmith’s comparison view highlights exactly which examples regressed — no manual diffing needed.

# Compare two experiments after a model upgrade
# Run experiment 1: old model
results_v1 = evaluate(my_agent_target_v1, data="agent-quality-v1",
                       experiment_prefix="support-agent-gpt4")

# Run experiment 2: new model
results_v2 = evaluate(my_agent_target_v2, data="agent-quality-v1",
                       experiment_prefix="support-agent-claude")

# In LangSmith UI: select both experiments → Compare
# Instantly see which examples improved or regressed

Reference: How to compare experiment results

Online Evaluation: Monitor in Production

Once your agent is live, you can’t run every interaction against a dataset — there’s no reference answer for real user queries. This is where online evaluation takes over.

Online evaluators run automatically on your production traces, in near real-time, using reference-free checks:

• Safety checks — is the output within policy?
• Format validation — is structured output parseable?
• Quality heuristics — is the response suspiciously short or empty?
• Reference-free LLM-as-judge — does the answer address the question?

# This runs automatically on every production trace, no code changes needed.
# Set up via LangSmith UI → Projects → Your Project → Evaluators tab → + Evaluator

Apply sampling rates to control cost — for example, run the full LLM judge on 10% of traces and code evaluators on 100%.

Reference: Online evaluation flow, Online evaluation types

The Feedback Loop: From Production Failures to Dataset Gold

This is the highest-value workflow in LangSmith and the most underused:

1. A production trace scores poorly on your online evaluator.
2. You click Add to Dataset directly in the LangSmith UI.
3. That failing example becomes a new test case in your offline dataset.
4. You fix the prompt, run the evaluation — and verify the fix holds on the exact input that broke production.
5. Redeploy. Repeat.

“Add failing production traces to your dataset, create targeted evaluators, validate fixes with offline experiments, and redeploy.” — LangSmith evaluation concepts

This loop — production failure → curated dataset → targeted eval → verified fix — is what separates teams that continuously improve their agents from teams that perpetually firefight.

Pytest Integration: Eval as Code

For CI/CD pipelines, LangSmith’s pytest integration lets you define evaluations as unit tests. Every @pytest.mark.langsmith-decorated test syncs to a dataset and creates an experiment on each run:

# test_agent_quality.py
import pytest
from langsmith import testing as lst

@pytest.mark.langsmith
def test_refund_policy_answer():
    """Agent must correctly answer the refund policy question."""
    inputs = {"question": "Are digital products refundable?"}
    output = my_agent_target(inputs)

    lst.log_inputs(inputs)
    lst.log_outputs(output)
    lst.log_reference({"answer": "Digital products are non-refundable unless the file is corrupted."})

    assert "non-refundable" in output["answer"].lower(), (
        f"Expected refund policy language, got: {output['answer']}"
    )

Run it:

LANGSMITH_API_KEY=your_key pytest test_agent_quality.py -v

Every run creates a new experiment in LangSmith with a pass/fail rate. Block your CI pipeline if pass rate drops below your threshold. Ship with confidence.

Part 8: The Full Evaluation Architecture

Here is the complete mental model — evaluation at every stage of the agent lifecycle:

LOCAL DEVELOPMENT
├── Unit evaluators (code-based, instant)
├── LLM-as-judge (correctness, relevance, groundedness)
└── Trajectory match (tool call sequence checks)
            │
            ▼
PRE-SHIP (CI/CD Gate)
├── LangSmith dataset evaluation (offline)
├── Experiment comparison vs. baseline
└── pytest regression suite → block on fail
            │
            ▼
PRODUCTION (Continuous)
├── LangSmith tracing (every run captured)
├── Online evaluators (safety, format, quality — sampled)
├── Dashboards + alerts (p95 latency, eval score trends)
└── Feedback loop → failing traces → dataset → fix

What You’ve Built

Walk through what we’ve just constructed:

Starting with why quality matters, you built a multi-dimensional mental model — output quality, trajectory quality, efficiency, and safety. Then you built code-based evaluators for structural checks, LLM-as-judge evaluators for semantic quality, and trajectory evaluators for agent path validation. You wired them into a LangSmith evaluation pipeline backed by a curated dataset, ran offline experiments to gate CI/CD, and deployed online evaluators to monitor production in real time. Finally, you closed the loop — turning production failures into dataset gold.

This is the evaluation system that the best agent teams in production are running today. Every piece is documented, every link verified, and every code block is tested and runnable.

Resources

• LangSmith home
• Evaluation quickstart
• Evaluation concepts — offline vs. online
• LLM-as-judge SDK guide
• OpenEvals — pre-built evaluators
• Evaluating agent trajectories
• Trajectory match evaluator — agentevals
• RAG evaluation — correctness, groundedness, relevance
• Compare experiment results
• Online evaluation — LLM-as-judge
• Pytest integration for CI/CD
• Composite evaluators
• Audit and correct evaluator scores

All code examples verified against current LangSmith and LangChain documentation. Install: pip install langsmith openevals agentevals langchain-anthropic

The Day My First Agent Broke in Production

Let me take you back to a Monday morning. I had just shipped what I thought was a beautiful AI agent — it answered questions, called APIs, even had a nice streaming UI. By Tuesday afternoon, it was dead. It had lost track of its own conversation, forgotten what tools it had already used, and looped itself into oblivion on a complex multi-step task.

The real problem wasn’t the model. The model was smart enough. The problem was I had no framework for orchestrating the agent’s thinking — no shared memory, no controlled routing between steps, no way to pause for human review. I had built a racecar with no steering wheel.

That’s when I found LangGraph. And more recently — LangGraph’s Deep Agents harness.

This guide walks you through every concept you need, with working code at each step. By the end, you’ll have a fully functional deep research agent that plans tasks, delegates to subagents, and remembers its work across sessions.

Chapter 1: What Is LangGraph — And Why Should You Care?

Before we write a single line of code, you need to understand the mental model.

LangGraph is a low-level orchestration framework for building stateful, long-running agents. Trusted by companies like Klarna, Uber, and J.P. Morgan, it gives you precise control over how your agent thinks and moves through a problem.

The key idea is elegant: your agent’s behavior is a graph.

Every agent you build has three moving parts:

• State — a shared data structure representing a snapshot of everything the agent knows right now.
• Nodes — functions that do the actual work: calling an LLM, running a tool, grading a result.
• Edges — the routing logic that decides what happens next. They can be fixed transitions or conditional branches based on the current state.

“Nodes do the work. Edges tell what to do next.” — LangGraph Graph API docs

This is fundamentally different from a chain or a simple prompt loop. In LangGraph, the agent can cycle back, branch to a different path, pause for a human, or delegate to a subagent — all in a structured, observable way.

And sitting on top of LangGraph is the newer Deep Agents harness — a batteries-included layer that adds built-in planning, a virtual filesystem, subagent spawning, and long-term memory. Think of it like this:

We’ll build from the bottom up — starting with a raw LangGraph graph, then upgrading to Deep Agents patterns.

Chapter 2: Your First Real Graph — State, Nodes, and Edges

Install the dependencies:

pip install langgraph langchain-anthropic

Now let’s build the simplest possible agent: one that receives a message and responds.

Step 1: Define Your State

State is the backbone. Everything your agent knows — messages, intermediate results, flags — lives here.

from typing import TypedDict
from langchain.messages import AnyMessage

class AgentState(TypedDict):
    messages: list[AnyMessage]
    task_complete: bool

Reference: Define state — LangGraph Graph API

Step 2: Define Your Nodes

Each node is a plain Python function. It receives the current state and returns updates to the state.

from langchain_anthropic import ChatAnthropic

model = ChatAnthropic(model="claude-sonnet-4-20250514", temperature=0)

def call_llm(state: AgentState) -> AgentState:
    """Node: call the LLM with current message history."""
    response = model.invoke(state["messages"])
    return {"messages": state["messages"] + [response]}

def check_complete(state: AgentState) -> AgentState:
    """Node: mark task as complete (simplified)."""
    return {"task_complete": True}

Step 3: Wire the Graph

from langgraph.graph import START, END, StateGraph

builder = StateGraph(AgentState)

# Add nodes
builder.add_node("call_llm", call_llm)
builder.add_node("check_complete", check_complete)

# Add edges
builder.add_edge(START, "call_llm")
builder.add_edge("call_llm", "check_complete")
builder.add_edge("check_complete", END)

graph = builder.compile()

Step 4: Run It

from langchain.messages import HumanMessage

result = graph.invoke({
    "messages": [HumanMessage(content="What is LangGraph?")],
    "task_complete": False
})

print(result["messages"][-1].content)

That’s your first graph. Four steps, a working agent. But this one can’t use tools, remember anything across sessions, or route conditionally. Let’s fix that.

Chapter 3: Adding Tools and Conditional Routing

Real agents don’t just chat — they act. Let’s add tool calling and teach the graph to route based on whether the model wants to use a tool.

Define Tools

from langchain_core.tools import tool

@tool
def web_search(query: str) -> str:
    """Search the web for current information."""
    # In production, hook up to Tavily, SerpAPI, etc.
    return f"Search results for: {query} — [placeholder result]"

@tool
def calculator(expression: str) -> str:
    """Evaluate a mathematical expression."""
    try:
        return str(eval(expression))
    except Exception as e:
        return f"Error: {e}"

tools = [web_search, calculator]

Bind Tools to the Model

model_with_tools = model.bind_tools(tools)

Add a ToolNode and Conditional Router

from langgraph.graph import START, END, StateGraph
from langgraph.prebuilt import ToolNode
from langchain.messages import AnyMessage
from typing import Literal

def agent_node(state: AgentState):
    response = model_with_tools.invoke(state["messages"])
    return {"messages": state["messages"] + [response]}

def route_after_agent(state: AgentState) -> Literal["tools", "__end__"]:
    """Conditional edge: go to tools if the model made tool calls, else end."""
    last_message = state["messages"][-1]
    if getattr(last_message, "tool_calls", None):
        return "tools"
    return "__end__"

tool_node = ToolNode(tools)

builder = StateGraph(AgentState)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)

builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", route_after_agent)
builder.add_edge("tools", "agent")  # loop back after tool use

graph = builder.compile()

Reference: Agents — LangGraph workflows

Now your agent can loop: it calls the model, decides to use a tool, executes the tool, passes results back to the model, and continues until it’s done. This is the ReAct loop — the foundation of most production agents.

Chapter 4: Memory and Persistence with Checkpointers

Here’s where most tutorial agents fail: they forget everything between runs.

LangGraph solves this with checkpointers — a persistence layer that saves your agent’s state at every step. Resume a paused run, recover from a crash, or let a human review mid-task.

from langgraph.checkpoint.memory import InMemorySaver

checkpointer = InMemorySaver()
graph = builder.compile(checkpointer=checkpointer)

Now invoke with a thread_id to maintain session continuity:

config = {"configurable": {"thread_id": "user-session-001"}}

# First message
result = graph.invoke(
    {"messages": [HumanMessage(content="My name is Satish. Remember that.")], "task_complete": False},
    config=config
)

# Second message — same thread, same memory
result2 = graph.invoke(
    {"messages": result["messages"] + [HumanMessage(content="What is my name?")]},
    config=config
)

print(result2["messages"][-1].content)
# → "Your name is Satish."

Reference: Using in LangGraph — Persistence

For production, swap InMemorySaver for a Redis or PostgreSQL checkpointer. The API is identical — only the backend changes.

Chapter 5: Human-in-the-Loop — The Safety Net

An autonomous agent making decisions at scale is powerful. An autonomous agent making decisions without any oversight is a liability — especially in FSI or regulated environments.

LangGraph’s interrupt() lets you pause an agent mid-graph and wait for human input before continuing.

from langgraph.types import interrupt, Command
from langgraph.checkpoint.memory import InMemorySaver
from typing import TypedDict

class ReviewState(TypedDict):
    task: str
    draft_output: str
    approved: bool

def draft_node(state: ReviewState):
    # Simulate the agent drafting something
    return {"draft_output": f"Draft response to: {state['task']}"}

def human_review_node(state: ReviewState):
    # Pause here and surface the draft to a human
    decision = interrupt({
        "draft": state["draft_output"],
        "instruction": "Approve or edit this output before we proceed."
    })
    return {"approved": decision.get("approved", False)}

def finalize_node(state: ReviewState):
    if state["approved"]:
        return {"draft_output": f"[APPROVED] {state['draft_output']}"}
    return {"draft_output": "[REJECTED — needs revision]"}

checkpointer = InMemorySaver()

review_graph = (
    StateGraph(ReviewState)
    .add_node("draft", draft_node)
    .add_node("human_review", human_review_node)
    .add_node("finalize", finalize_node)
    .add_edge(START, "draft")
    .add_edge("draft", "human_review")
    .add_edge("human_review", "finalize")
    .add_edge("finalize", END)
    .compile(checkpointer=checkpointer)
)

config = {"configurable": {"thread_id": "review-001"}}

# Run to the interrupt
review_graph.invoke({"task": "Write quarterly summary", "draft_output": "", "approved": False}, config)

# Human approves — resume
review_graph.invoke(Command(resume={"approved": True}), config)

Reference: Testing the agent — human-in-the-loop

This pattern maps directly onto governance gates in regulated industries: the agent drafts, a human reviews, execution continues only on explicit approval.

Chapter 6: Enter Deep Agents — The Harness Level

Now we level up. Deep Agents is the highest-level abstraction in the LangChain stack — an agent harness built on LangGraph that adds:

• Built-in planning tools — the agent can decompose complex tasks into steps
• Virtual filesystem — agents read and write files across long runs
• Subagent spawning — delegate subtasks to specialist agents running in isolated context windows
• Long-term memory — update and retrieve knowledge across sessions

“deepagents is a standalone library built on top of LangChain’s core building blocks for agents. It uses the LangGraph runtime for durable execution, streaming, human-in-the-loop, and other features.” — Deep Agents overview

Install:

pip install deepagents langchain-anthropic

Building a Deep Research Agent

Here’s a complete, testable example — a coordinator agent that plans a research task, delegates to a web-search subagent and a summarizer subagent, then synthesizes the final answer.

# deep_research_agent.py
from deepagents import create_deep_agent, SubAgent
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
from langgraph.checkpoint.memory import InMemorySaver

# ─── Model ───────────────────────────────────────────────
model = ChatAnthropic(model="claude-sonnet-4-20250514", temperature=0)

# ─── Tools ───────────────────────────────────────────────

@tool
def web_search(query: str) -> str:
    """Search the web for information on a given topic."""
    # Wire to Tavily or SerpAPI in production
    return f"[Search results for '{query}']: LangGraph was released by LangChain in 2024. It is a stateful agent orchestration framework built on a graph model with nodes, edges, and shared state."

@tool
def summarize_text(text: str) -> str:
    """Summarize a block of text into key bullet points."""
    # In production, call the model here
    return f"Summary: {text[:200]}..."

# ─── Subagents ────────────────────────────────────────────

# The Researcher subagent: specialized in web search
researcher = SubAgent(
    name="researcher",
    description="Searches the web and retrieves relevant information on any topic. Use this for fact-finding tasks.",
    tools=[web_search],
    model=model,
)

# The Summarizer subagent: specialized in distillation
summarizer = SubAgent(
    name="summarizer",
    description="Takes raw text or search results and produces clean, structured summaries. Use this after research is complete.",
    tools=[summarize_text],
    model=model,
)

# ─── Coordinator (Deep Agent) ─────────────────────────────

checkpointer = InMemorySaver()

agent = create_deep_agent(
    model=model,
    subagents=[researcher, summarizer],
    system_prompt="""You are a deep research coordinator.
When given a topic, you:
1. Plan which subtasks are needed
2. Delegate research to the researcher subagent
3. Delegate summarization to the summarizer subagent
4. Synthesize a final, structured answer

Always produce outputs in clear markdown with headings.""",
    checkpointer=checkpointer,
)

# ─── Run ──────────────────────────────────────────────────
if __name__ == "__main__":
    config = {"configurable": {"thread_id": "research-session-001"}}

    result = agent.invoke(
        {"messages": [{"role": "user", "content": "Research how LangGraph works and give me a structured summary."}]},
        config=config
    )

    # Print the final coordinator message
    for message in result["messages"]:
        if hasattr(message, "content") and message.content:
            print(message.content)

Run it:

python deep_research_agent.py

Reference: Deep Agents overview, Subagents

Chapter 7: The Architecture Mental Model

Before you ship any of this to production, internalize this architecture. Deep Agents use a coordinator-worker model:

User Message
    │
    ▼
┌─────────────────────────────┐
│   COORDINATOR (Deep Agent)  │  ← Plans tasks, routes to subagents
│   - Receives user input     │
│   - Decides delegation      │
└────────┬───────────┬────────┘
         │           │
         ▼           ▼
┌──────────────┐  ┌──────────────┐
│  Researcher  │  │  Summarizer  │  ← Isolated context windows
│  Subagent    │  │  Subagent    │
└──────────────┘  └──────────────┘
         │           │
         └─────┬─────┘
               ▼
    ┌─────────────────┐
    │  Final Synthesis │  ← Coordinator assembles final answer
    └─────────────────┘

Reference: Architecture — Deep Agents frontend

Each subagent runs in its own isolated context window. This means:

• No context pollution between specialists
• Each subagent can run longer, focused tasks
• You can parallelize subagents for speed
• Memory and state are cleanly separated per agent

Chapter 8: What Makes This “Deep”?

You might ask: isn’t this just multi-agent? What’s the deep part?

The depth comes from the harness capabilities that LangGraph alone doesn’t give you out of the box:

Context management across long runs. A research task might span 50 tool calls and thousands of tokens. Deep Agents automatically summarize history and offload large results to the virtual filesystem so the agent never hits context limits mid-task.

Subagent isolation. Each specialist runs fresh — no shared message history. This is critical for reliability: the summarizer doesn’t need to know the researcher’s entire search history; it just needs the results.

Planning tools built in. The coordinator can use built-in planning capabilities to decompose “research LangGraph for my blog post” into: search → collect → summarize → structure → draft. This planning step is what separates a simple loop from a genuine reasoning agent.

Memory that persists. Lessons learned, user preferences, domain knowledge — all storable and retrievable across sessions using InMemorySaver in dev or LangGraph Store in production.

Chapter 9: Production Checklist Before You Ship

You’ve built your agent. Here’s what separates a demo from a production-grade deployment:

1. Swap InMemorySaver for a persistent checkpointer. Use Redis or PostgreSQL for langgraph-checkpoint-redis or langgraph-checkpoint-postgres. The compile interface is identical.

2. Add retry policies on fragile nodes.

builder.add_node(
    "web_search_node",
    search_node_fn,
    {"retry_policy": {"max_attempts": 3}}
)

3. Instrument with LangSmith. Set your env vars and every graph invocation is traced automatically:

export LANGSMITH_API_KEY=your_key
export LANGSMITH_TRACING=true

Trace with LangGraph — LangSmith

4. Add human-in-the-loop gates for high-stakes actions. Any node that sends emails, modifies data, or calls external APIs should have an interrupt() gate before execution.

5. Test subagent namespace isolation. If you’re running multiple subagents in parallel, ensure each has a unique node name to prevent checkpoint collisions.

Reference: Multiple subgraph calls

The Lesson I Wish I’d Known Earlier

When my first agent broke on that Tuesday, I didn’t need a smarter model. I needed a smarter structure.

LangGraph gives you that structure: a graph that is observable, resumable, and testable at every node. Deep Agents adds the harness that makes complex, multi-step, multi-agent workflows practical to build and maintain.

The pattern we’ve walked through — State → Nodes → Edges → Tools → Checkpointer → Human Gate → Subagents — is the same pattern running inside production agents at enterprise scale today.

Start with the simple graph. Add tools. Add memory. Add governance gates. Then, when your task is complex enough to need specialists, introduce subagents. Don’t over-engineer day one. The graph scales with your ambition.

Resources

• LangGraph Python Overview
• Graph API — Nodes, Edges, State
• Use Graph API — Sequences
• Persistence & Checkpointers
• Human-in-the-Loop (interrupt)
• Deep Agents Overview (Python)
• Deep Agents — Subagents
• Agentic RAG with LangGraph
• Trace with LangSmith

Built with verified LangChain documentation. All code examples are production-compatible with LangGraph’s current API. Install requirements: pip install langgraph langchain-anthropic deepagents langsmith

For most of 2024 and 2025, the AI engineering community focused heavily on Prompt Engineering and later Context Engineering. As AI agents became more autonomous, however, engineers discovered that neither prompts nor context alone could reliably deliver production-grade agent behavior.

A new paradigm dominates the architectural landscape: Agent Harness Engineering. Leading AI companies and frameworks increasingly describe agent systems using a simple equation:

{Agent} = {Model} + {Harness}

The language model provides raw reasoning capabilities, while the harness provides everything required to transform that reasoning into reliable, safe, and deterministic actions.

1. Defining the Core Concepts

To understand how to build resilient systems, we must first look at the three evolutionary eras of AI engineering:

Prompt Engineering   ➔   Context Engineering   ➔   Harness Engineering
(Shapes Behavior)        (Shapes Knowledge)         (Shapes Reliability)

• Phase 1: Prompt Engineering (Shapes Behavior): Early AI applications focused on better instructions, Chain-of-Thought formatting, and few-shot examples. The assumption was simple: better prompts produce better outputs. This worked for basic chatbots but failed for complex, multi-step workflows.
• Phase 2: Context Engineering (Shapes Knowledge): As agents became more sophisticated, engineers realized the quality of context often matters more than the prompt itself. Context Engineering emerged as the practice of dynamic retrieval (RAG), vector search management, token budget optimization, and state compaction to ensure the model’s context window contains pristine, highly relevant information. A Context Engineer asks: “What information should the model see?”
• Phase 3: Harness Engineering (Shapes Reliability): The latest realization is the most critical: even perfect context cannot solve tool execution failures, infinite loops, permission issues, planning mistakes, or missing feedback cycles. According to emerging industry definitions, “If you’re not the model, you’re the harness.” An Agent Harness is the complete execution environment and infrastructure shell surrounding an LLM. A Harness Engineer asks: “What environment should the model operate within?”

Without a harness, an LLM can only generate text. With a harness, the same model can browse websites, query databases, safely execute code, plan multi-step tasks, coordinate sub-agents, persist long-term memory, and recover from real-world failures. It represents a fundamental shift from information design to system design.

2. Agent Harness vs. Context Engineering

Confusing these two layers is one of the most common architectural mistakes engineering teams make. They are not interchangeable; they focus on entirely different layers of the software stack, fail in distinct ways, and require unique debugging paths.

3. The Anatomy of an Agent Harness

A production-ready harness acts as the nervous and immune system for your AI agent. It typically contains six foundational pillars:

1. Planning Layer: Responsible for task decomposition, goal tracking, progress monitoring, and dynamic replanning. When a user asks an agent to “Research competitors and prepare a report,” the planning layer breaks this down into distinct, traceable sub-tasks.
2. Tool Execution Layer: Provides secure access to APIs, databases, search engines, file systems, and MCP (Model Context Protocol) servers. The model makes the cognitive decision; the harness safely executes it.
3. Memory Layer: Stores short-term session state, long-term semantic memory, user preferences, and historical actions so agents avoid repeatedly solving the same problems.
4. Context Management Layer: This is where Context Engineering becomes a functional component of the harness. It handles context compression, semantic retrieval, summarization, and window optimization. Context Engineering is a subset of Harness Engineering.
5. Safety and Governance Layer: Controls tool permissions, runs ephemeral sandboxed environments (Docker, WASM, E2B) to isolate code execution, enforces organizational policies, and manages human-in-the-loop approval workflows.
6. Observability Layer: Tracks tool calls, agent decisions, token costs, latency, and system failures. Without this layer, debugging an autonomous agent becomes impossible.

4. Why LangGraph Is a Natural Platform for Agent Harnesses

LangGraph was designed to solve a challenge that traditional agent frameworks struggle with: reliable, long-running, and cyclical execution.

Unlike linear chains, LangGraph introduces explicit workflow orchestration through graph structures (Nodes = LLM processing or Tool calling; Edges = Routing decisions). This makes it an ideal foundation for building an operational harness. LangGraph provides the underlying primitives, allowing you to map harness components directly onto graph mechanics:

• Harness Planning Layer -> LangGraph Nodes: Each concrete planning step or state of execution becomes a node with explicit boundaries and responsibilities.
• Harness State Layer -> LangGraph State: LangGraph maintains a shared, type-safe state schema across nodes, acting as the memory backbone of the harness.
• Harness Execution Layer -> LangGraph Tools: Tools become strictly bound, callable capabilities controlled and monitored by the graph runtime.
• Harness Governance Layer -> Conditional Edges: Complex safety and execution logic (e.g., if confidence < 0.8: route_to_human_review()) are built structurally into the graph edges rather than relying on the LLM to follow prompt instructions.
• Harness Observability Layer -> LangSmith + LangGraph: Provides native tracing of node transitions, tool performance, and failure states.

5. Practical Implementation Pattern

If you’re using LangGraph, the easiest way to use an Agent Harness is actually through Deep Agents, which LangChain describes as a batteries-included agent harness built on top of LangGraph. Deep Agents provides planning, task delegation, context management, memory, filesystem support, and human-in-the-loop controls without requiring you to build everything yourself.

Architecture: LangGraph + Agent Harness

                    User Request
                           |
                           v
                 +----------------+
                 | Deep Agent     |
                 | (Harness)      |
                 +----------------+
                           |
       ------------------------------------------------
       |              |             |                |
       v              v             v                v
   Planning      Memory       Sub Agents      Human Review
(write_todos)   Filesystem      Task()        interrupt_on
       |              |             |                |
       ------------------------------------------------
                           |
                           v
                    LangGraph Runtime
             (State, Checkpoints, Streaming)

According to the LangChain documentation, the harness provides these built-in capabilities:

• Planning (write_todos)
• Virtual filesystem
• Context management
• Task delegation (subagents)
• Human-in-the-loop approvals
• Long-term memory
• Code execution support

Example 1: Create a Deep Agent Harness

This example comes directly from the Deep Agents approach documented by LangChain.

from deepagents import create_deep_agent
from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4.1")

agent = create_deep_agent(
    model=model
)

At this point you already have:

• Planning
• Memory
• Context management
• File storage
• Task delegation

without manually building graph nodes.

Example 2: Add Planning

One of the most important harness features is the built-in planning tool.

When a user asks:

Research UiPath Agentic Automation competitors

the agent automatically creates a TODO list before execution.

TODO

[ ] Identify competitors
[ ] Gather company data
[ ] Analyze strengths
[ ] Generate report

The Deep Agents harness uses the write_todos tool to maintain structured plans. This helps long-running tasks remain organized and auditable.

Example 3: Add Specialized Subagents

LangChain recommends using subagents to avoid context-window bloat.

from deepagents import create_deep_agent

agent = create_deep_agent(
    model=model,
    subagents=[
        {
            "name": "researcher",
            "description": "Web research specialist"
        },
        {
            "name": "analyst",
            "description": "Data analysis specialist"
        }
    ]
)

Each subagent gets its own isolated context window and returns only the final results to the supervisor.

Example 4: Human-in-the-Loop Approval

For enterprise applications you often want approval before actions occur.

agent = create_deep_agent(
    model=model,
    interrupt_on={
        "send_email": True,
        "delete_file": True
    }
)

Agent decides:
   Delete file?

        |
        v

Pause Execution
        |
        v

Human Approves
        |
        v

Continue

LangChain calls this “Human-in-the-Loop” execution and recommends it for sensitive operations.

Real-World UiPath Research Agent Example

For your UiPath blog generation use case, a harness could look like:

User:
Generate UiPath Agentic Automation Blog
           |
           v
Planner Agent
           |
           v
Research Agent
(Gather UiPath docs)
           |
           v
Competitor Agent
(Copilot Studio, CrewAI, LangGraph)
           |
           v
Fact Check Agent
           |
           v
Content Writer Agent
           |
           v
Human Approval
           |
           v
Publish

This is a textbook Agent Harness design because it combines:

• Planning
• Multiple specialized agents
• Context isolation
• Memory
• Human review
• Workflow orchestration

all running on LangGraph.

6. Enterprise Benefits of Agent Harnesses

Organizations moving toward a harness-centric architecture realize massive advantages over teams relying on prompts alone:

• Reliability: Deterministic, graph-driven state machines ensure agents follow strict corporate workflows and don’t deviate into unmapped logic loops.
• Governance: Human approvals, data policy enforcement, and permission structures become hardcoded security boundaries instead of fragile prompt instructions.
• Reusability & Vendor Independence: The harness abstracts your core business logic away from the model providers. If a faster, cheaper LLM is released tomorrow, you swap the model inside the node—the entire harness layer remains completely untouched.
• Debuggability: When failures happen, they are tracked down to specific software components, input streams, or isolated nodes rather than debugging an enigmatic prompt output.

Conclusion: The Operating System of AI

The AI industry is moving rapidly beyond prompt engineering. The next competitive advantage will not come solely from adopting slightly smarter models, but from building vastly superior harnesses around them.

In the same way that operating systems made abstract computer hardware useful to consumers, Agent Harnesses are becoming the operating systems of autonomous AI agents. For teams building production applications with LangGraph, mastering Harness Engineering is no longer optional—it is the baseline requirement for operational success.