Building Production AI Agents: Tool Calling, Memory and Human-in-the-Loop

The Demo-to-Production Gap for AI Agents

A demo agent processes 10 carefully crafted queries. Each query is clear, well-formed, and within the agent's designed scope. The tools respond instantly with clean data. The agent reasons correctly through every step. Leadership approves production deployment. Production traffic arrives: users send ambiguous queries ("fix my thing"), misspelled inputs, queries in unexpected languages, requests outside the agent's scope, and adversarial prompts testing boundaries. Tools return errors, timeouts, and unexpected data formats. The agent encounters edge cases the demo never tested — and its behavior on edge cases determines whether users trust or abandon it.

Production agent engineering addresses 7 areas that demos ignore: tool reliability (what happens when tools fail?), memory management (what happens on the 50th message?), human-in-the-loop (when does the agent ask for help?), error handling (what does "I'm confused" look like?), testing (how do you test an agent with 10,000 possible reasoning paths?), monitoring (how do you know the agent is behaving correctly?), and scaling (how do you serve 1,000 concurrent agent sessions?).

Production Tool Calling: Reliability at Scale

In demos, tools always work. In production, tools fail 5-15% of the time — API timeouts, authentication expiration, rate limiting, unexpected response formats, and downstream system outages. Production tool calling must handle every failure mode gracefully.

Retry with Exponential Backoff

Transient failures (timeout, rate limit) resolve on retry. Pattern: attempt → wait 1 second → retry → wait 2 seconds → retry → wait 4 seconds → final attempt → if still failing, report error to agent. The agent's system prompt includes instructions for handling tool failures: "If a tool call fails, explain the issue to the user and suggest an alternative or offer to try again."

Fallback Tools

Critical tools should have fallbacks. If the primary order lookup API is unavailable, the agent falls back to a cached data source (less current but available). If the payment processing tool fails, the agent creates a manual processing ticket instead of leaving the customer's request unresolved. Fallback tools provide degraded but functional service during outages.

Tool Response Validation

Don't trust tool responses blindly. Validate: does the response match the expected schema? Are the values within reasonable ranges? (An order total of $-500 or $99,999,999 probably indicates an error.) Is the response relevant to the query? (Did the tool return data for the right customer?) Validation catches tool bugs before the agent incorporates incorrect data into its reasoning.

Memory Engineering: Managing State Across Sessions

Agent memory management is an engineering challenge, not a feature checkbox. Production agents face: context window overflow (long conversations exceed the LLM's limit), irrelevant context accumulation (earlier messages no longer relevant but consuming tokens), cross-session continuity (user returns tomorrow expecting the agent to remember today's conversation), and memory consistency (multiple concurrent sessions for the same user must not corrupt each other).

Sliding Window with Summarization

Keep the last N messages in full detail. Summarize older messages into a compressed context block. The summary preserves key facts (user identity, task progress, decisions made) while discarding conversational filler. Pattern: messages 1-10 → summarize to 200 tokens → keep messages 11-20 in full → when message 21 arrives, summarize messages 11-15, keep 16-21 in full. The agent always has recent context in full detail and historical context in summary.

Task State Tracking

Separate conversational memory (what was said) from task memory (what was accomplished). Task state tracks: current task, completed steps, remaining steps, data collected, actions taken, and pending confirmations. Task state is structured (JSON, not prose) and persisted to a database — surviving session boundaries, browser refreshes, and connection interruptions. When the user returns, the agent reconstructs context from task state: "Welcome back. Last time, we were processing your refund for order #12345. I've verified the purchase — ready to process the $85 refund?"

Human-in-the-Loop: The Safety Net That Makes Agents Deployable

Human-in-the-loop isn't a fallback for agent failure — it's a designed capability that makes agents deployable in enterprise environments where autonomous mistakes have consequences.

Three HITL Patterns

Confirmation gate: The agent prepares the action, presents it to the user for approval, and executes only on confirmation. Used for: irreversible actions (refunds, deletions, external communications), high-value actions (transactions above threshold), and first-time actions (the first time the agent performs a new action type, require confirmation; learn from the pattern for future automation).

Escalation to human agent: The agent recognizes it can't resolve the issue — low confidence, out of scope, emotionally charged conversation, or complex multi-party situation. The agent summarizes the conversation and hands off to a human agent with full context. The human agent sees: conversation history, tools called, results obtained, and the agent's assessment of the issue.

Supervisory review: The agent operates autonomously but every action is logged and a sample (10-20%) is reviewed by a human supervisor post-hoc. Supervisory review catches systematic errors (the agent consistently misinterprets a specific policy) that individual-transaction confirmation misses. Review findings feed back into prompt improvements, tool descriptions, and guardrail updates.

Error Handling: What to Do When the Agent Gets Confused

Agents get confused. The user's request is ambiguous. Two tools return conflicting information. The reasoning chain leads to a dead end. The agent's response doesn't make sense even to itself. Production agents need explicit confusion handling — not the default behavior of generating a confident-sounding wrong answer.

Confusion detection: The agent should detect its own confusion: the same tool called 3 times with the same parameters (loop detection), conflicting information from different sources, user query that doesn't match any available tool, and reasoning that requires information the agent doesn't have. When confusion is detected, the agent should say so: "I'm not sure how to handle this. Let me explain what I found and connect you with someone who can help."

Graceful degradation: When the agent can't complete the full task, it should complete as much as possible and clearly communicate what's unfinished: "I've verified your order and confirmed it's eligible for a refund. However, I wasn't able to process the refund because the payment system is currently unavailable. I've created a ticket (#54321) so our team can process it when the system is back — typically within 2 hours."

Testing AI Agents: Beyond Happy-Path Demos

Agent testing requires testing the reasoning process, not just the final output — because agents take different paths to different outcomes depending on tool responses, user inputs, and conversation history.

Trajectory testing: Define expected tool call sequences for representative queries. "For order status query: the agent should call get_order(order_id) → get_shipping(order_id) → format response." Verify the agent calls the expected tools in the expected order. Trajectory testing catches: unnecessary tool calls (agent queries 5 systems when 2 suffice), missing tool calls (agent answers from knowledge instead of checking the live system), and incorrect tool parameter usage.

Adversarial testing: Test with inputs designed to confuse the agent: ambiguous queries, contradictory information, out-of-scope requests, prompt injection attempts, and queries that test guardrail boundaries. The agent's behavior on adversarial inputs determines production safety.

Load testing: Test with production-level concurrent sessions. 100 simultaneous agent conversations, each making tool calls, maintaining memory, and generating responses. Load testing reveals: memory leaks (state accumulation over long sessions), resource contention (multiple agents competing for the same API quotas), and latency degradation (response time at 100 concurrent users vs. 1).

Production Monitoring for Autonomous Systems

Agent monitoring extends traditional application monitoring with agent-specific metrics:

Task completion rate: What percentage of user tasks does the agent complete successfully without human escalation? Target: 60-80% for task agents, increasing over time as edge cases are addressed. Below 50%, the agent creates more work than it saves.

Tool call success rate: Per tool, what percentage of calls succeed? A tool with 85% success rate drags down the entire agent's reliability. Identify and fix unreliable tools first.

Average reasoning steps: How many think-act cycles does the agent take per task? Increasing step counts over time suggest: tool descriptions are degrading (agent tries multiple tools before finding the right one), user queries are getting more complex, or the agent's reasoning is becoming less efficient.

Escalation analysis: What types of queries get escalated to humans? Clustering escalated queries reveals: capability gaps (the agent needs a new tool), knowledge gaps (the knowledge base needs content), and policy gaps (the agent doesn't know how to handle a common scenario). Each escalation cluster is a product improvement opportunity.

Deployment Architecture and Scaling

Agent deployment requires stateful infrastructure — each agent session maintains conversation history, task state, and tool context. Stateless architectures (typical for APIs) don't work because the agent needs continuity across multiple request-response cycles within a session.

Session-affinity architecture: Route each user session to the same compute instance (using session cookies or load balancer affinity). The instance maintains the agent's state in memory during the session. Limitation: instance failure loses session state.

Externalized state architecture (recommended): Store session state in a fast external store (Redis, Cosmos DB). Any compute instance can serve any session by loading state from the store. This enables: horizontal scaling (add instances without session migration), instance failure recovery (state survives instance restart), and session resumption (user can return hours later and continue).

Go Deeper

Continue building your understanding with these related resources from our consulting practice.