Teams usually ask whether an AI agent can do a task. The better question is who should own the decision at the end of it. An agent can read, summarise, extract and draft very well. It’s a poor place to put the final say on something that costs money or breaks a promise to a customer.
This is the rule we work to: an agent proposes, and code or a person decides. The cases below show where that line goes and why.
Three kinds of decision
Every step in a process is one of three things.
A lookup or a calculation. The answer exists in your data or follows from a rule. What is the price for two adults on 14 March? Is the room available? A database query or a function answers this exactly, every time. Use code. A language model adds a way to get the answer slightly wrong.
A reading task. Someone wrote a messy message and you need to know what they want. They mention dates, a party size, a budget and an irritated tone. Models are good at this, because the input is unstructured and a human skim would do the same job.
A judgement call with consequences. Do we refund this? Do we hold the room without a deposit for this customer? Do we agree to a cancellation outside the terms? These carry cost, precedent and sometimes legal weight.
An agent belongs in the second kind, helps with the third by preparing the case, and should stay out of the first except to call the right tool.
Why money and bookings stay with a person
The reason is accountability, plus a failure mode that’s hard to spot. A model that’s wrong often sounds as confident as one that’s right. When the output is a draft that a person reads, an error costs a few seconds. When the output is an action, an error can be a refund sent, a seat sold twice or a deposit waived, and nobody knows until the customer or the accountant notices.
So we put a human step in front of these actions:
- Confirming or cancelling a booking.
- Taking or refunding a payment.
- Changing a price, applying a discount or waiving a fee.
- Promising anything that isn’t in your published terms.
- Replying to a complaint, a dispute or someone who is upset.
The agent still does most of the work. It gathers the facts, checks availability, drafts the reply and states what it would do and why. The person reads a prepared case and clicks approve, which takes far less time than assembling it.
OWASP’s guidance on excessive agency lists the same idea among its mitigations: require human approval for high-impact actions, and enforce authorisation in the downstream system instead of trusting the model to decide what is allowed. That second point matters. If your booking API will refund anything the agent asks for, the safeguard exists only in the prompt, and prompts aren’t safeguards.
Put the limits in code, not in the prompt
"Never offer a discount above 10 percent" written in a system prompt is a request. A function that rejects any discount above 10 percent is a limit. Build the second.
A workable structure looks like this:
- The agent has a small set of tools, each doing one narrow thing: check availability, fetch a price, draft a quote, create a draft booking.
- Tools that change state create drafts or proposals. They don’t commit.
- A separate step, run by code after a person approves, performs the commit.
- Every tool call and every approval is logged with the input and the result.
When something goes wrong, the log tells you what the agent saw, what it proposed and who approved it. That’s the record you’ll want in a dispute.
Where agents should decide on their own
Not everything needs approval, and insisting on it removes the value. Some decisions are safe to automate:
- Sorting an inbound message into a category, where a wrong category only means a human re-files it.
- Answering a factual question from your own published information, with a citation, such as opening hours or what an excursion includes.
- Sending an acknowledgement that a request was received.
- Asking a clarifying question when the dates or party size are missing.
The test is the one from our article on what agents are good at: if this is wrong, who finds out, how quickly, and what does it cost? If the answer is "nobody, eventually, nothing", automate it.
The cases agents handle badly
Some situations need a person even when no money is involved.
- Someone who is upset. A customer whose trip has fallen apart wants to hear from a person who can act. A fluent apology from a machine can make it worse.
- Ambiguity with consequences. If "next Friday" could mean two dates, ask. A guess that books the wrong date creates work for everyone.
- Anything medical, legal or safety related. Route it to a qualified person.
- Unusual requests. If a request doesn’t look like the ones in your examples, treat low confidence as a reason to escalate.
Make escalation a normal outcome. If the only options are "answer" and "fail", the agent will answer. Give it an explicit "hand to a person" path and measure how often it uses it.
Test the split before you rely on it
Take a few hundred real messages and the replies your staff sent. Sort them by what happened: could have been closed by an agent, could have been drafted, shouldn’t have been touched. If most of the volume falls in the third group, an agent is the wrong investment for that process, and it’s better to say so early.
Then build an evaluation set from those cases and run it before every release and after any model change. Agents drift when the underlying model is updated, and an evaluation is how you notice before a customer does.
Security is part of this
An agent that reads customer text can be steered by that text. If it also holds tools that move money or change bookings, a crafted message becomes an attack. Narrow tools, human approval for state changes and server-side authorisation limit the damage. We cover this in prompt injection and business automations.
If you want help drawing this line for a specific process, an AI Readiness Review includes a list of what we’d advise you not to automate, and our AI agents service covers the build.