We’ve built digital agents and customer service automation for close to a decade. The tooling has changed completely. The hardest problem hasn’t.
The old failure mode: low confidence, visible gaps
Pre-LLM conversational systems ran on intent classification. A user query got matched against a fixed set of trained intents using something like an SVM or a fine-tuned BERT classifier, and the system returned a confidence score. Below a threshold, usually somewhere around 0.6 to 0.7 depending on how the model was tuned, the bot would trigger a fallback: a clarifying question, a menu of options, or an escalation to a human agent.
This made the system’s limits legible. You could pull a confusion matrix, see exactly which intents got misclassified against which, and retrain on the gaps. Handling time dropped because the bot resolved the high-volume, low-complexity tickets (password resets, order status, hours of operation) and routed everything else to a human. NPS moved because customers got instant resolution on the easy stuff instead of waiting in a queue for it. The economics were straightforward: measure containment rate, measure CSAT on contained sessions versus escalated ones, tune the threshold.
The new failure mode: high confidence, no gap
LLM-based agents don’t have an equivalent confidence signal in the same sense. A transformer generates the next token based on probability distributions learned from training data, and there’s no built-in mechanism that says “I don’t actually know this.” The model produces fluent, grammatically correct, well-structured text whether the underlying claim is grounded in a real source or not.
That’s the operational risk. A pre-LLM bot’s failure looked like a failure: a fallback menu, a “sorry, I didn’t get that.” An LLM’s failure looks like a normal answer. It cites a policy that doesn’t exist, gives a refund amount that’s wrong, or states a shipping timeline with the same tone of voice it uses for the correct ones. Detecting it requires a different toolkit than confidence thresholds: retrieval-augmented generation to ground answers in an actual knowledge base, citation or attribution checks against source documents, and human review sampling on outputs rather than just on low-confidence inputs.
Why the expectation gap matters
Users calibrate trust based on how something sounds, not on how it was produced. A monotone, obviously scripted bot got low expectations by default. A fluent LLM agent sounds like it’s reasoning, so people extend it the same trust they’d give a competent human agent. When it’s wrong, the cost of that miscalibration lands on the business, in a wrong policy quoted to a customer, a hallucinated feature, an incorrect legal or financial detail.
Closing that gap isn’t a prompting problem. It’s an evaluation and grounding problem: building retrieval pipelines that constrain the model to verified content, running adversarial test sets specifically designed to surface confident-but-wrong answers, and setting up production monitoring that flags ungrounded claims after deployment, since pre-launch testing alone won’t catch everything.
Ten years in, the metrics we track (handling time, containment rate, CSAT, NPS) haven’t changed. What changed is that a good number on those metrics no longer tells you the system is safe. It just tells you it’s convincing.