The gap between testing and production

A common issue with AI agents in customer support is that they can look great in testing right up until they go live. The conversations look good, answers are accurate and customers are genuinely impressed. You’d be forgiven for feeling confident in the agent going live… until it actually went live.

Working at Lorikeet, you quickly learn just how complex customer problems can be. We go beyond solving simple FAQs and handle nuanced, high-stakes support queries for some of the most regulated industries in the world, where the range of customer situations are endless. No amount of manual conversation lab testing can ever cover it all.

We initially built Simulations to answer this question: how do you efficiently catch and resolve problems before they reach end customers?

The loop that improves our agents

The idea behind Simulations was straightforward: instead of waiting for real customers to discover workflow issues, you can create, test and improve synthetic ones. The platform automatically generates realistic customer scenarios covering all kinds of situations, frustration levels and edge cases based on your business context. These tests can then be run over and over again to give you full confidence that you are ready to go live with a new channel, use case or workflow.

The next evolution of Simulations is built with automation and self-service at the core, ensuring users can easily test and improve the durability of workflows before going live, without the need for manual configuration. To enable this, our in-app agent, Coach, handles the creation, testing and improvement loop for you; running Simulations, finding issues, fixing them, running Simulations again, and so on – getting sharper and faster with each ticket.

AI is no longer a black box

For the teams we work with now, this significantly changes what going live actually means. Instead of manually simulating conversations one by one, they run Simulations to get a clear picture of how the agent performs across the full range of what their customers ask, then Coach improves what isn’t working until it does. It's faster, more thorough, and it means their Concierge agent is no longer a black box - and they can make quick workflow improvements with confidence.

It also changes how they manage things post-launch. Rather than only noticing an error once CSAT drops, they can simply use Simulations to catch any possible regressions. Problems are caught in the Simulations loop, not in the wild.

Reliable AI needs more than a better model

What we learnt building this is that reliable AI agents you can fully trust is not just about having a better model. It's about having a feedback loop so AI can learn from its mistakes and become more consistent, accurate and reliable.

Since launching Simulations, our customers have run over 58,000 tests and resolve tickets at twice the rate.