Building an AI Service Agent with Safe Human Escalation

#The problem, as it actually presented

The brief in this situation is almost never a technical one. It is closer to "the board has asked what we are doing with AI, and service is the obvious place to start".

That framing is not wrong, but it carries a hidden assumption: that the difficulty lives in the model. It does not. The retailer in this scenario handles a large, repetitive stream of order and returns enquiries and a much smaller stream of account cases that are genuinely complicated. The repetitive stream looks made for automation, and it is. What sinks these projects is everything around the answer.

Trace a failed agent programme backwards and the same four things appear. The scope was written as capability rather than as questions, so nobody could say where it stopped. The knowledge it cited contradicted itself, because three teams had each written a returns policy article and none of them had been retired. Refusal was tested by asking forbidden questions directly, which any agent handles. And escalation was built as a route to a queue, with nothing designed about when it fires or what travels with it.

So the design work below treats the escalation path as the product. The answering is the easy half.

#Scope: the list you can find transcripts for

The first decision is that an intent only enters scope if somebody can point at closed conversations where real customers asked it.

This sounds like a documentation preference and it is a control. An intent with no transcripts behind it has three problems at once: you have no evidence the demand exists, you have no material to build a test set from, and you have no idea what the question looks like when a real person asks it badly. All three surface in the first week of live traffic, together.

So the intent list is built from a quarter of real contacts and written in customer language. Not "handle returns enquiries" but "can I still send this back, I bought it six weeks ago". The first phrasing is a category. The second is testable, and the difference between them decides whether the refusal boundary means anything.

Why "handles everything" is a support-ticket generator. A broad agent is not a more capable agent, it is an agent with an undefined edge. Customers find the edge quickly, because they do not know it exists. Each time they cross it the agent produces something plausible, the customer acts on it, and the correction arrives as a second contact through a channel that costs more. The organisation then reads the pattern as poor AI quality, when what it actually built was an unbounded promise.

The scope that survived was four intents: order status, delivery timing, returns eligibility and refund progress. Everything else was explicitly out, including several things the business wanted and one thing the demo had handled well.

#Grounding: remediation comes first

For each intent in scope, exactly one source is the authoritative answer, that source has a named owner, and the owner has confirmed a date on which they last checked it is correct.

Stated that plainly it sounds trivial. It is the work that consumes the most calendar time in this shape of project, because the state you start from is almost always worse than anyone believes.

Three failure patterns show up, in this order. Duplicates that disagree. Several articles describe the same returns window differently because they were written for different campaigns and never retired. The agent does not pick one, it produces a confident synthesis of all of them, and that behaviour survives every prompt rewrite you try, which is why teams misdiagnose it as a model problem. Orphaned content. The owner recorded against an article is a team that reorganised two years ago, so nobody will confirm whether it is current and nobody will take it out. Facts that live somewhere else. The intent requires a ship date the agent cannot reach, because the field sits in a system that was never brought across.

The remediation that precedes the agent is therefore editorial before it is technical. Walk the four intents, and for each one have the content owner point at the single article that is right. Where they hesitate or name two, you have found the work. Retire or redirect the rest, and record the decision so the next person does not restore them.

The customer data side is separate and is where Data 360 earns its place. The agent needs a single view of who it is talking to, assembled from the systems that hold order history, contact history and service records, rather than four partial views it has to reconcile mid-conversation. That unification is a data programme with its own timeline, and treating it as a configuration step inside the agent build is how these projects slip. Our fuller position on that is in the grounding checklist we run before an agent talks to a customer.

#Refusal that survives paraphrase

Every agent refuses the forbidden question when it is asked directly. That test proves nothing.

The refusal list itself is the straightforward part: pricing commitments, goodwill decisions, anything about an account other than the one in context, anything touching a live complaint or a legal dispute, anything involving a customer flagged as vulnerable. Failure here is rarely an absent list. It is a list the project team wrote that nobody with a risk mandate ever read, so the fix is a signature from legal, risk and the operations lead who owns complaints, obtained before launch rather than after a screenshot circulates.

The harder check is behavioural. Refusal has to hold when the forbidden thing is asked sideways, wrapped inside a reasonable question, framed as a hypothetical, or several turns into a conversation that started legitimately. Those three routes are where refusal degrades, and they degrade quietly whenever a new topic or grounding source is added.

So the negative test set covers each refusal category three ways: direct, indirect through a legitimate-sounding route, and mid-conversation after several on-topic turns. It runs before launch and before every change to instructions, topics or sources. A negative suite made entirely of direct requests is a suite that would be passed by a cautious new starter reading it cold, which means it is not testing the agent at all.

#Escalation triggers that are not confidence thresholds

Escalation gets built as one thing, a route. It is three: triggers, routing, and carried context. Two of the three are usually skipped, and they are the two customers notice.

Triggers come in four categories and the design needs all four.

Explicit. The customer asks for a person. Honoured immediately, without negotiation, every time. An agent that responds to a request for a human by asking the customer to describe the problem again is generating a complaint.

Behavioural. Repeated rephrasing, the same question asked twice in different words, rising message length. These say a conversation is going wrong before the customer says so.

Confidence. Low retrieval confidence, or no authoritative source found. Necessary, and insufficient on its own.

Business rule. The category that gets left out and matters most. Any mention of a regulator or ombudsman, any complaint language, financial hardship, bereavement, or a vulnerable-customer signal escalates regardless of how confident the agent is.

That last category exists because of a specific failure the other three cannot catch. A confidence threshold assumes the risk is uncertainty. The dangerous case is the opposite: the agent is entirely certain, retrieval was clean, and the answer is technically correct while being the wrong thing to say to this person. Confidence measures whether the agent found a source. It has no view on what kind of conversation it is in.

#What travels with the handover

Routing is the part most implementations do build, and it still gets one thing wrong. Escalated conversations are not average conversations. The straightforward questions have already been handled, so what reaches a person is skewed toward the difficult and the sensitive. Routing them into the general queue with the general skill requirements concentrates the hardest contacts into a channel staffed for a normal mix.

There is a second routing check that has nothing to do with skills: the receiving queue has to be staffed for the hours the agent runs. An agent available on a Sunday evening that escalates into a queue opening Monday morning has not escalated, it has abandoned.

Carried context is what decides how the handover feels. The consultant should see, without opening anything, five things: the transcript, the identified account and contact, the intent the agent believed it was handling, the actions it has already taken, and the trigger that fired. Anything less and the customer starts again, having spent five minutes with an agent for nothing. That experience is measurably worse than no agent at all.

The internal version of that failure is the one that lasts. If consultants decide in week two that escalations arrive without useful context, they stop reading the context permanently, and repairing the payload three months later does not undo it. Front-line trust is spent once. The measurement framework around this is set out in deflection that does not cost you the customer.

#Testing against things that really happened

The evaluation set is built from closed cases with known outcomes, using the resolution a human actually gave as the known-good answer.

Invented test questions are the default and they are the wrong tool. They are suspiciously clean: correct spelling, one intent each, all the context in the first sentence, no missing details. Real contacts arrive with two questions at once, a reference number belonging to a different order, and the load-bearing detail buried three messages in. An agent tuned against invented questions is tuned for a customer who does not exist, and its first week is spent discovering that in front of real ones.

So the replay set is pulled from resolved cases across the four intents, deliberately including the ones that went badly, and it includes questions the agent is supposed to be unable to answer. Most test suites contain none of those, which means the most common production behaviour, declining, is the one nobody rehearsed.

Record the baseline before launch, re-run before every change to instructions, topics, actions or content, and agree the bar that blocks a release before anyone has seen the first score. A bar chosen afterwards is a bar chosen to pass.

#The signals to instrument beyond containment

Containment reported on its own is the easiest number to improve badly, because an agent reluctant to hand over will show a healthy one while generating callbacks. So four more signals are instrumented while the agent is built, not retrofitted afterwards.

Escalation rate with the reason attached, so a designed handover can be told apart from a failure. Repeat contact within seven days of an agent-only resolution, joined across channels rather than within one, because the customer who was not helped simply used a different door and a single-channel view is structurally unable to see them. A weekly sample of transcripts read by a person, because a regression set only tests what somebody thought to ask.

And the quietest one: conversations that end in neither a resolution nor a handover. The customer left. Nothing recorded that they were unhappy, no escalation fired, and containment counted it as a success. That signal has to be defined deliberately because no standard report produces it, and it is usually the first place a scope problem becomes visible.

#Who can turn it off

A named person, reachable on the first morning, can disable the agent without seeking approval, and has done it once already before launch.

Two failures live here. The common one is that the person who first notices an agent behaving badly is a front-line supervisor, while the person authorised to stop it is three levels up and in another time zone. The subtler one is that the authority exists, the mechanism has never been used, and the first attempt happens under pressure with an audience. Decide in advance what happens to conversations already in flight when the agent stops, and what customers meet instead.

#The decisions that were contested

Cutting the intent the demo handled well. An account-change intent tested beautifully and had no transcript evidence behind it, because the volume was low and the cases were unlike each other. It came out. That was unpopular, and it was the right call: low volume plus high variance is the worst possible first intent, since it delivers little and fails visibly.

Refusing to automate goodwill. There was appetite for the agent to make small goodwill decisions on delayed orders. We argued against it, on the grounds that if three experienced consultants would decide differently, the agent is being asked to judge rather than retrieve, and no amount of grounding fixes that. The narrow version, where the agent gathers the information and prepares the case for a person to decide, captures most of the handling time with none of the exposure.

Not adding "try to resolve before escalating". It reads as a sensible instruction. It is the single most common cause of an agent talking a frustrated customer in circles, and it is how containment improves while service gets worse. Persistence should come from better grounding, never from reluctance to hand over.

Building a separate escalation skill. The tidier design sends escalations into the existing queue. The business reason not to is that the composition of what arrives has changed, and staffing an unchanged queue for a changed mix is how the hardest contacts end up with the least experienced consultants.

#What changes

The outcomes worth claiming from this design are operational, and they follow from the decisions rather than from effort.

The second-order effect is the one leaders notice: the conversation in service review changes shape. It stops being about how much the agent handled and becomes about what happened next to the customers it handled, which is the question the business was asking in the first place.

#What we would tell you before starting

Start with the escalation path, not the answering. If you design the handover first, the scope question answers itself, because you can only escalate cleanly out of a boundary you have drawn. Teams that build the answering first discover the boundary by finding out where it was missing.

Budget for the content work honestly. In this shape of project the editorial remediation is usually the longest task, and it is the one most likely to be estimated as a fortnight and take longer. Making it visible in the plan is what stops it being absorbed silently into the build.

And an agent is the wrong answer entirely in two situations. If the volume is not repetitive, an agent has nothing to learn from and nothing to be tested against; you are building a bespoke system for a handful of contacts a week, and a well-written knowledge article would serve customers better for a fraction of the cost. And if the decisions in scope are judgement calls your organisation has never written down, an agent will surface that disagreement rather than resolve it, in front of customers. Write the policy first. Then decide whether it needs an agent at all.

If you are scoping a service agent and want a second read on the scope, grounding and escalation design before you build, talk to an architect.