Does AI improve
customer service?

“AI customer service” can mean two different things.

These designs distribute responsibility differently. A copilot can help a worker write or find an answer. An autonomous agent can act on behalf of the service. Comparing them requires different measures, and the human handoff matters in both.

A large workplace study found an average productivity gain—with uneven benefits.

A study published in the Quarterly Journal of Economics followed the staggered introduction of a generative AI conversational assistant among 5,172 customer-support agents at one Fortune 500 business-software company. The tool offered real-time reply suggestions based on customer chats. Agents remained in charge and could use, edit, or ignore each suggestion.

Access to the assistant was associated with 15% more customer issues resolved per hour on average. The gains were largest for less experienced and lower-skilled workers, who improved both speed and quality. The most experienced and highest-skilled workers saw smaller productivity gains; the authors also found small declines in some quality measures for this group.

The average customer-satisfaction result did not significantly change. That matters: an increase in cases handled per hour is not the same measure as customers feeling their problems were resolved well.

The study is useful real-world evidence, not a universal forecast. It examined one company’s support operation, one copilot, and a staggered rollout—not a randomized trial across many industries.

Faster autonomous handling did not mean better ratings for every chat.

A 2026 working-paper preprint reports a randomized field experiment on Alibaba’s Taobao platform. It involved 647 customer-service workers and 680,676 chats. The AI agent handled standardized cases designated as AI-eligible; workers supervised the system and continued handling cases outside that category.

Across all chats, average duration fell by 3.2%, while repeat-contact rates and customer ratings showed no statistically significant overall change. Within AI-eligible chats, average duration fell by 16.8%, but customer ratings were lower by 0.412 points on a five-point scale. For the AI-ineligible cases handled by people, ratings were slightly higher and chats slightly faster.

This is a preprint about one platform and a short field experiment. It is not a universal verdict on AI agents. It does show why system-wide averages can hide a poor experience in the cases the model is actually handling.

Escalation works only when people can still help.

The Alibaba study separates technical failures from emotional escalations. Human intervention was more effective when the system reached a technical limit than when a customer was already frustrated. Earlier intervention helped preserve service quality; a late handoff left the worker less room to recover the interaction.

A useful handoff passes along the conversation, the action already taken, and the reason for escalation. The person taking over should be able to correct the system and continue the task without asking the customer to start again.

In practice, this means a service should recognize both “I cannot complete this task” and “the customer is losing confidence.” A model can be fast at the first reply while the overall support path still fails.

Handle time is one signal, not the scorecard.

Track these measures together. A system that closes more chats quickly may still create repeat contacts, lower ratings, or heavier recovery work for the next person.

Test the workflow before expanding the automation.

1. Define the eligible request. Name which issues the system may help with or handle, and how it recognizes exceptions.
2. Keep the existing service as a real comparator. Measure the new path against the process customers and workers would otherwise use.
3. Give people a clear way to take over. Preserve conversation context and make override and escalation available at the right moment.
4. Measure quality and speed together. Include resolution, repeat contacts, customer ratings, handling time, and human recovery work.
5. Inspect difficult cases separately. Review unusual requests, emotional escalations, language differences, and cases where the AI repeatedly fails.
6. Review the effect on workers. Look at training, confidence, workload, and whether suggestions help new agents build skill or weaken experienced agents’ judgment.

For broader agent evaluation, see how to test an AI agent ↗ and when an agent is useful ↗. For evidence about AI at work, continue with AI literacy at work ↗.