The jump from chatbot to agent feels smaller than it is. Both use language models. Both take user input and produce output. The underlying technology looks the same from the outside.
But the design challenge is completely different.
Chatbots Answer One Question
A chatbot turn looks like this: user sends a message, model reads the message plus conversation history, model produces a response. Done. Even in a multi-turn conversation, each turn is structurally a single inference call. The model does not persist between messages. It does not take actions in the world. It does not need to track its own state across steps.
When a chatbot makes a mistake, the user sees it immediately. They ask again, rephrase, or give up. The blast radius of a single bad response is one response.
Agents Run for Many Steps
An agent has to:
- Understand the goal
- Break it into steps
- Retrieve relevant knowledge
- Call tools
- Read tool feedback
- Update its working state
- Decide the next action
- Avoid drifting from the original objective
- Know when it is done
Every tool call creates new context. Every result adds more tokens. Every step increases the chance that something goes sideways.
The Failure Modes That Don't Exist in Chatbots
Task drift
An agent starts with a clear objective. Three tool calls in, the context has grown and some detail from an early tool output has started pulling the reasoning in a different direction. The agent is now working on a subtly different problem than the one it was given. Nobody told it to change direction — the accumulated context did.
Chatbots don't drift. They respond to whatever is in the current message. Agents drift because their reasoning evolves over a long sequence of inputs they are accumulating themselves.
Compounding errors
In a chatbot, a wrong answer is visible and isolated. In an agent, a wrong intermediate output gets passed to the next step as if it were correct. The next step reasons on top of it. The error is now embedded in the chain. By the time the final output is produced, the original error may be invisible — buried under several layers of downstream reasoning that all look sound.
Tool boundary confusion
Agents have to decide when to call a tool and when to reason inline. This is harder than it sounds. Models often try to simulate a tool result rather than calling the tool — especially when the tool call feels expensive or the model thinks it already knows the answer. The result is a hallucinated API response, a made-up database query result, or a plausible-looking computation that was never actually run.
Knowing when to stop
A chatbot always stops after one response. An agent has to determine when its task is complete — and that determination can be wrong in both directions. Stopping too early produces incomplete work. Stopping too late produces an agent that has taken actions beyond its mandate or spent ten times the intended token budget.
What This Means for Design
Agent design is not just about model quality or prompt quality. It is about:
- Context control — managing what accumulates in the window so the agent doesn't drift or get confused by irrelevant earlier outputs
- State management — deciding what needs to be tracked explicitly versus inferred from context
- Tool boundaries — defining clearly when the agent should call a tool and what constitutes a valid tool output
- Evaluation — testing agent behavior at the workflow level, not just the single-turn level
- Recovery — designing for what happens when an intermediate step fails, not just the happy path
Most teams start building agents by treating them like better chatbots. They discover the gap between the two the hard way — usually somewhere between the third demo and the first production incident.