Courier CS Quality Console

A courier runs an AI chatbot beside a human support team across website chat, WhatsApp and the app. Leadership can see the volume of contacts, but not the quality, and not where the service is quietly losing customers.

Pasting a few transcripts into a chat tool grades one conversation. It cannot measure a thousand, score them the same way twice, or prove the numbers. The Courier CS Quality Console does the measuring, then turns it into a monthly review a manager can act on.

It reads the whole conversation log, classifies each of 1,200 contacts into one of 20 reasons, and scores the agent's decision (resolve, escalate, or decline) against an agreed answer key. The picture: the chatbot is fast at 2.5 minutes and handles 59% of contacts on its own, but it is right 76% of the time against the human team's 91%.

The damage concentrates in one place. 107 high-stakes cases (claims, payment disputes, rude-rider complaints) were closed instead of being passed to a person. Those are the cases that pull customer ratings from 4.08 down to 1.82.

Before any figure is shown, the measurement is graded against a held-out truth:

• It recovers the correct contact reason on 100% of conversations, and beats a naive "trust the agents" read that would report no problems at all.
• A governed "ask the data" box answers plain questions with exact figures and refuses anything off the menu, so customer records never leak.
• Every number on the console is computed from the locked results, not generated, so the same question returns the same figure every time.

The pattern that keeps showing up: a chatbot and a busy human both fail silently. Neither grades its own decisions across every conversation, and the expensive mistakes hide in the cases nobody reviewed.

Measuring an AI agent against a known answer is what turns "we have a chatbot" into "we can prove how well it works, and fix what it does not."