AITechnicalData

Answering support before it becomes a ticket

Organisation: iLyF · Easy, Instant Insurances
Role: Led the build, the knowledge base behind it, and the escalation boundary
Period: Malaysia

An omnichannel AI assistant across WhatsApp, Telegram and web, where most of the effort went into deciding what it was allowed to say. Underneath it, Segment, Mixpanel and Sentry read as one signal, so a customer in trouble gets reached before they think to complain.

of routine tickets handled without a human: 70–80%of routine tickets handled without a human
first response on every channel: 24/7first response on every channel
support headcount as the user base grew: Flatsupport headcount as the user base grew
product events on one Segment schema, one identity: 50+product events on one Segment schema, one identity
contact on friction and error signals, before a ticket existed: Outboundcontact on friction and error signals, before a ticket existed

How the system actually worked

Scroll the figure sideways to read it, or turn your phone.

Retrieval over curated content, not open generation. The assistant answers from validated material or it does not answer, which is what stops it saying something confident and wrong about a policy somebody paid for.
The boundary is drawn explicitly: which classes of question it may attempt at all, decided in advance rather than discovered in production.
An escalation carries the full conversation, so the customer never has to restart their story with a human.
Every escalation is a knowledge-base gap, not a model failure. Closing those gaps moved deflection far more reliably than prompt tuning ever did.
The signal is assembled, not observed in one place: Segment carries one identity into every tool, Mixpanel says where the customer stalled, Sentry says what broke underneath them. Neither analytics nor error tracking can name a person, a step and a cause on its own.
Every signal we could detect was a signal we could message on, which is exactly the temptation to resist. A cool-down, a suppression list and a hard rule against interrupting a live human conversation keep the list of situations short.

Two ways into the same assistant. One path the customer starts, and retrieval decides whether we may answer at all. The other the signal loop starts, before anyone has typed a word. Everything the assistant could not answer became a knowledge-base gap, and that backlog, not prompt tuning, is what drove the next improvement.

Scroll the figure sideways to read it, or turn your phone.

Segment is the unglamorous piece that makes the rest possible: one event schema defined once, so “quote started” and “payment failed” mean the same thing in every tool and carry the same identity. Wire Mixpanel and Sentry into the app separately and there is nothing to join a year later.
A funnel drop tells you people left, not whether the product broke. An exception tells you something broke, not whether anyone abandoned because of it. Joined on one identity, they name the person, the step and the cause in the same record.
The list of situations worth interrupting someone for stayed deliberately short. Anything the rules could not name went to a human with the session attached rather than being guessed at.
Same stream, a second use: the engineering queue ranked by customers affected rather than by error count, which is a different order from the one an error tracker sorts by.

The detection half. Segment is the unglamorous middle: without one schema and one identity, a Mixpanel drop-off and a Sentry exception are two dashboards describing strangers. The short list of named situations is deliberate — everything we could see was something we could message on, and that is the temptation to resist.

Context

Support volume grows with the user base, and most of it is the same twenty questions.
At the rate iLyF was growing, the honest options were to hire linearly or to answer automatically.
The tickets were the loud minority. A payment that failed at 11pm mostly did not become a ticket, it became a closed app, and nothing recorded that they had tried.

What was actually needed

Instant answers on the channels people already used.
Without the failure mode that makes support bots hated: confidently saying something wrong about a policy somebody paid for.
And a way to know someone was in trouble without waiting to be told. In a once-a-year purchase, an abandoned checkout is a lost year, not a lost session.

How I used AI here

Retrieval over a curated knowledge base, not open generation. The assistant answers from validated content or it does not answer.
An explicit escalation boundary: which classes of question it may attempt at all, and which go straight to a human.
Escalations carry the full conversation, so the customer never restarts their story.
It drafts the outreach for a situation the rules already chose. It never chooses the situation, the audience or the moment.
The larger share of the project was writing and structuring that knowledge base, not building the assistant.

Seeing the trouble before the customer reports it

Three tools we already paid for, each covering what the others cannot see.
Segment: one event schema, one identity, fanned out to every tool. Without it there is nothing to join.
Mixpanel: where they stalled. The funnel step that leaks, the retry loop, the screen they sat on before leaving.
Sentry: why. The exception, the release, the insurer call that timed out under the spinner.
Neither is enough alone. A drop-off does not say the product broke; an error does not say anyone abandoned. Joined on one identity, they name the person, the step and the cause.
A short list of situations earns an interruption: payment failure, a quote retried three times, a crash mid-purchase, a renewal opened and dropped.
Each with a cool-down and a suppression list, so nobody mid-conversation with a human gets messaged by a robot.
The outreach says what happened, what we fixed, and the one step left, on the channel they were already on, carrying the retry link or the reissued cover note.
Anything the rules could not name went to a human with the session attached. The same stream ranked the engineering queue by customers affected, not error count.

Decisions that mattered

Treating unanswered questions as a content backlog rather than a model failure.
Every escalation was a signal that the knowledge base had a gap.
Closing those gaps improved deflection far more reliably than prompt tuning ever did.
Instrumenting through Segment rather than wiring each tool in separately. An extra layer at the time, and the only reason the sources could be joined a year later.
Setting a high bar for what earns an interruption. Proving how closely you watch someone is the fastest way to be resented for it.

What changed

Between seventy and eighty percent of routine tickets resolved without a human, around the clock.
Support cost stayed roughly flat while the user base grew.
Human agents spent their time on the complex cases that actually needed judgement.
Failed payments, stuck quotes and crashes stopped depending on the customer to report them.
Support stopped being only a queue and became a queue plus a watchlist.
Detected friction fed the same content backlog as escalations, so the assistant improved on questions about to be asked, not only ones already logged.

The insight I kept

A support assistant’s quality ceiling is set by your documentation, not your model.
Most of our gains came from writing better answers.
A support ticket is a lagging indicator of a product failure, and the rarest outcome of one. Most people do not complain, they leave, and you find out at renewal.
Three tools already on the bill. Joining them on one identity is what turned dashboards nobody watched into something that could act.

Stack & practices

OpenAI
Retrieval over curated KB
Segment
Mixpanel
Sentry
WhatsApp
Telegram
Web