Chatbot vs AI agent

Safety here is a property of scope rather than of intelligence. A more capable model makes mistakes less often, and that is worth having, but it does not change what a mistake is able to do. The size of the worst outcome is set by what the system can reach: which data it can read, which operations it can perform, on whose behalf, and with what limits. Those are decisions made by the people who build it, and they are the first thing to design, before any prompt is written.

A chatbot’s safe envelope is wide because its reach is narrow. It can answer from approved content, explain a process, compare options, collect the details a person will need, triage a request and hand it over, and with the user properly signed in it can look up that user’s own records through read-only calls. Its risks are about what it says. It can state something false with complete confidence, describe a policy you do not have, promise a discount nobody authorised, or repeat something from its context that the user should not see. None of that is trivial: a tribunal has already ordered at least one company to compensate a customer who relied on its website chatbot’s incorrect account of a refund policy. Treat what your chatbot says as a statement made by your organisation. The controls follow from that. Ground every answer in content you have approved, instruct the bot to say when the answer is not there, keep it away from offers and commitments it cannot check, and test that it declines when it should.

An agent’s safe envelope is exactly as wide as its tools, so the design work is in the tools. Mechanically, a tool is a function described to the model with a name, a purpose and a schema for its inputs. OpenAI and Claude both support this natively: the model returns a structured request to call a tool, and your code decides whether to run it, runs it, and hands back the result. That decision point is where safety lives, because the model never touches your systems directly. Your code does, and your code can refuse. Classify every tool as read-only, reversible write, consequential write or irreversible, give it the narrowest scope that does the job, and attach hard limits: an amount cap on a refund tool, an allowlist of recipients on an email tool, a rate limit on everything. When the agent acts for a particular user, it should act with that user’s permissions, never with a service account that can see and change everything. Open standards such as the Model Context Protocol make it easier to expose the same tools to different models, which is convenient, and they do nothing to make any individual tool safer.

The risk peculiar to agents is that they read things, and what they read can contain instructions. An email, a web page, a support ticket or a shared document can include text written to redirect the model, and a model with tools may follow it. This is prompt injection, and no current model is immune to it. The dangerous combination is an agent that can read private data, is exposed to content an outsider controls, and has some way to send information out. Remove any one of the three and the worst case shrinks considerably. In practice that means separating the step that reads untrusted content from the step that holds powerful tools, restricting where data can be sent, treating every tool result as untrusted input, and putting a person in front of any action that communicates outside the organisation.

For any action that matters, the pattern that works is propose and confirm. The agent does the gathering, the reasoning and the preparation, then presents the action it intends to take, and a person approves, edits or rejects it. This keeps the slow, tedious part automated and the accountable decision human. It is also the cheapest way to begin, because you can launch with every write action gated and remove gates one at a time as evidence accumulates, which is far easier than adding control to an agent that has been acting freely.

Decide in advance which actions always need approval, and write the list down. Moving money, sending anything to a customer or a third party, deleting or overwriting data, changing permissions or configuration, and anything with legal or contractual effect belong on it. So does anything irreversible, however small it looks. Reversible internal actions, such as tagging a ticket, filling a field, creating a draft or routing work to a queue, can usually run without a gate, provided they are logged and a sample is reviewed. The test for each action is simple: if the agent did this wrongly a hundred times before anyone noticed, how bad would that be?

Approval only works if reviewers engage with it, and badly designed approval queues train people to click yes. If every step needs sign-off, the volume becomes a chore and attention drops. If the request shows only a summary, the reviewer has nothing to judge. Good approval design shows exactly what will happen, such as the amount, the recipient, or the record before and after the change, together with the evidence the agent relied on, and it puts approve, edit and reject in one place. Group similar requests so they can be reviewed together, and route each kind to someone who can actually judge it.

Treat the approval queue as a measuring instrument as well. For each action type, track how often proposals are approved unchanged, edited or rejected. An action that is approved unchanged consistently, across a meaningful number of cases, is a candidate to run automatically under a limit, with sampled review afterwards. An action whose edit rate rises after a model or prompt change should move back behind the gate. Autonomy then becomes something granted per action type, on evidence, and revocable, which is a position you can explain to a risk committee or an auditor without hand-waving.

A chatbot is comparatively easy to evaluate, because its output is text and each reply stands on its own. Build a test set from real questions taken from support tickets, search logs and transcripts, including the awkward ones, and for each record the correct answer or the source it should come from. Score correctness, whether every claim is supported by the retrieved content, whether the bot declines when the answer is not available, whether it hands over when it should, and whether the tone is acceptable. A language model can do much of the scoring, provided you check its judgements against human ones on a sample. Run the set on every change to the prompt, the model, the retrieval or the content, and a regression shows up before customers see it.

An agent’s output is a changed world, so the test has to happen in one. That means a sandbox: test instances or faithful fakes of the systems the agent acts on, seeded with known data. Each scenario defines a starting state, a goal and the correct end state, and the evaluation checks whether the task was completed, whether anything changed that should not have, whether the path was sensible (the right tools, no loops, no wasted calls), and how many steps and how much spend it took. Because the same agent can take a different route through the same task, every scenario should run several times. An agent that succeeds on average and fails badly every so often is not ready, and a single run will not show you that.

Agents also need adversarial tests that chatbots barely require. Include documents and emails with instructions planted in them, requests ambiguous enough that the right move is to ask, tasks that should be refused, and tool failures such as timeouts, errors and partial results. The question in each case is whether the agent stays within its scope, stops when it should, and reports honestly what it did and did not do. These scenarios belong in the same suite as the functional ones and should block a release in exactly the same way.

Evaluation does not end at launch for either. For a chatbot, review a sample of transcripts every week, and watch the share of questions it could not answer, the handover rate, and what people ask that the content does not cover. For an agent, watch approval and edit rates, rollbacks, steps and cost per task, tool error rates and any action outside its normal pattern, and alert on all of them. Tracing every model call and tool call is essential for agents, because when something goes wrong the first question is always what the agent saw and why it chose what it did, and only a full trace answers it.

The pricing models are simple even though the rates move. Hosted language models are billed by usage, per token, with input and output usually priced separately and larger models costing more per token than smaller ones. Packaged chatbot and agent platforms add their own pricing on top: per seat, per conversation, per resolution or per task. Rates and tiers change often, so check the current rate card for any provider you are considering, and model your own traffic against it rather than trusting a headline figure.

A chatbot’s cost per interaction is predictable. Each turn is one model call, sometimes two, and the main driver is how much text goes into the prompt: the instructions, the conversation so far and the retrieved passages. An agent’s cost per task is larger and less predictable. Each step is a model call, and each call usually carries the accumulated history of the task, including every tool result so far, so the tokens per call grow as the task proceeds. Retries and dead ends cost as much as successful steps. The tools themselves may carry charges, and the human review time behind approvals is frequently larger than the model bill. Two runs of the same task can differ widely, so set hard caps on steps and spend per task and alert when they are hit.

Reliability compounds in a way that is easy to underestimate. A chatbot’s errors are independent: one bad reply affects one reply. An agent’s task succeeds only if every step does, so per-step error rates multiply. A step that goes right nine times in ten, chained ten times, completes cleanly only about a third of the time. That is arithmetic rather than a benchmark, and it is why the dependable agents are the narrow ones. Keep tasks short, let ordinary code handle the steps whose logic you already know, make tools validate their inputs and refuse nonsense, make them idempotent so a retry cannot apply a change twice, and checkpoint long tasks so a failure resumes rather than starting again.

The fair comparison is cost per successfully completed task, including review and correction, against what that task costs today. For a chatbot, the unit is a question genuinely resolved, which is different from a conversation that ended because the user gave up. Be wary of containment figures that count everyone who never reached a person as a success. For an agent, the unit is a task done correctly from end to end, with the reviewer’s time included. On those terms, agents often justify themselves on high-volume back-office work and rarely on low-volume tasks, where the engineering and the oversight never pay back.

Before building either, look at what people actually ask. Pull a few months of support tickets, chat logs and site searches, group them, and read the largest groups. If the top requests are where an order is, how to reset a password and when you are open, the best answer is a tracking page, a self-service reset and a clear page with the hours on it. Each of those is faster for the user than any conversation, costs almost nothing to run and cannot hallucinate.

A good FAQ and a good search box remain underrated. Maintained answers written for the questions people really ask, a search that copes with synonyms and misspellings, and navigation that gets someone to the right page in a couple of clicks absorb a great deal of demand. They also expose the problem many chatbot projects discover late: the content is wrong, out of date or contradictory. A chatbot over poor documentation delivers poor answers fluently. Fixing the content helps every channel, including the chatbot you may still build, because retrieval-augmented generation is only ever as good as what it retrieves.

On the agent side, the overlooked alternative is a workflow. If the steps of a process are known, write them as ordinary code and call a model only at the points that need judgement: classifying an incoming email, extracting fields from a document, drafting a reply for review. Because the code fixes the sequence, the system is cheaper, faster, easier to test and easier to audit than an agent, and it fails in predictable places. Orchestration libraries such as LangChain, with its graph-based companion LangGraph, let you mix fixed steps and model decisions in one design, so most of a process stays deterministic and the model gets a choice only where a choice is genuinely needed.

A useful test: could an experienced member of staff write the procedure down as a flowchart? If they can, build the flowchart and use a model inside the boxes that need reading or writing. If the flowchart has a box that says, in effect, "work out what to look at next depending on what you find", and that box is where the time goes, that is the part an agent is for. Many processes turn out to be mostly flowchart with one or two genuinely open-ended steps, and the best design hands only those steps to an agent.

The progression that works is a ladder, and each rung produces the evidence needed to climb the next. First, content and search good enough that people can help themselves. Second, a grounded chatbot that answers from that content, shows its sources and hands over cleanly. Third, read-only tools, so the chatbot can look up the user’s own order, booking or case once they have signed in. Fourth, drafted actions: the system prepares the change and a person approves it. Fifth, low-risk actions running automatically within limits, with consequential ones still gated. Beyond that, autonomy widens one action type at a time, on the strength of the approval data.

Most of what you build on the lower rungs carries upward. The knowledge base and retrieval layer stay, because an agent needs to know your policies as much as a chatbot does. The evaluation set grows from questions into scenarios. The conversation logs become the requirements: they show which requests end in the same routine action and how often. Escalation paths, monitoring and the habit of reviewing transcripts all transfer. What is new at the agent stage is the tool layer, the identity and permission model, the approval interface, the sandbox, the audit store and an incident process for actions that went wrong, and that is where most of the effort in AI agent development goes.

The common mistake is skipping rungs because a demo worked. A demo shows the model choosing sensible actions on a handful of friendly cases. It says nothing about the long tail of odd requests, the planted instruction in a forwarded email, or what happens when a tool times out halfway through a task. The related mistake is building one general agent with every tool attached. One narrow agent per job, with a small tool set, its own evaluation suite and its own approval rules, is easier to test, easier to trust and easier to switch off. Start with the single action type that the logs show is the most frequent and the most routine.

Moving back down the ladder is normal and should be easy. If an automatic action starts being corrected, return it to approval while you investigate. If a model upgrade changes behaviour, gate everything again until the evaluations pass. If a chatbot is not being used, check the content and the entry points before blaming the model. A system built with a gate available on every write action from the start can tighten in minutes, and that ability is what makes it reasonable to loosen anything at all.