Why the real barrier to AI in insurance support isn't the technology. It's that most vendors can't prove what their AI said, or why.

In 2024, an Australian insurer received an enforceable undertaking from ACMA for customer communications that violated regulatory requirements. The content was written by humans, reviewed by compliance, and part of an approved script - but it was still wrong.

That was with humans following a process designed to prevent exactly this outcome. Now the insurance industry is being asked to trust AI that generates content on the fly.

We've had dozens of conversations with insurance CX leaders over the past year, and nearly all arrive at the same place: "We haven't stress-tested our position on using AI that can generate content on the fly, even with predetermined outcomes." This isn't resistance to AI, it's institutional memory. These are people who know what happens when customer-facing content goes wrong - enforceable undertakings, DOI complaints, E&O exposure - because they've lived through the remediation. They're right to be cautious, but the caution needs to be pointed at the right thing.

The fintech demo won't save you

Every AI support vendor has a fintech demo. It handles a billing dispute, cancels a subscription, maybe processes a refund. It looks clean, contained and impressive. Then they walk into an insurance meeting and present the same demo with different branding.

Thankfully there is still hope in the world, because Insurance CX leaders see through this immediately.

The regulatory landscape for insurance isn't a stricter version of fintech. APRA imposes prudential requirements on how insurers manage operational risk, including third-party technology. CPS 230, which took effect in July 2025, explicitly extends operational resilience obligations to material service providers. If your AI support vendor is generating customer-facing content about policy exclusions, excess calculations, or claims status, they're a material service provider - whether they've thought about it that way or not.

ASIC regulates how insurers communicate with policyholders, including the content of claims correspondence and product disclosure. ACMA enforces rules around direct communications and has a history of pursuing enforceable undertakings when insurers get the language wrong.

The gap between "cancel my subscription" and explaining why a policyholder's storm excess is $750 but their flood excess is $1,500 on the same property isn't incremental, it's structural. One requires a refund button, the other requires understanding of peril-specific excess structures, coverage endorsements, and the PDS - plus an audit trail that holds up when a regulator asks how that answer was generated.

Where wrong answers create liability

AI hallucination in insurance is a legal problem, not a user experience problem.

Consider the conversations that actually matter in insurance support: a policyholder asks whether their business interruption policy covers supply chain delays. A claimant disputes a settlement amount and wants the reserve calculation explained. A broker needs to know whether a client's endorsement for additional insured status was processed before a loss event. A customer in financial hardship needs guidance on their options under the General Insurance Code.

A wrong answer about a policy exclusion, a misquoted benefit amount, a promise of coverage that doesn't exist. In most industries, these create a bad customer experience, but in insurance they create binding obligations or regulatory exposure. The cost of a confident wrong answer is categorically higher than the cost of not answering at all.

Most AI support platforms were built around an assumption that makes sense in low-stakes environments: attempt to answer everything, escalate what fails. That architecture is fundamentally misaligned with insurance. When your compliance team has zero tolerance for content errors on coverage questions, claims guidance, and hardship processes, a system that optimises for coverage breadth over precision is a liability generator.

The deeper problem is auditability. If a regulator asks "why did your AI tell a policyholder their claim was covered under Section 4.3(b)?" most platforms can show you the output. They can show you the knowledge base article the output was derived from. What they cannot show you is the full decision path: why the system chose that article over three others, what alternative responses it considered, whether it had sufficient confidence to answer at all, and what guardrails it evaluated before responding.

Without that trail, your compliance team is signing off on a black box. And when the DOI examiner arrives, "we trusted the vendor" isn't a defence.

The build trap

Insurance companies that hear all of this and decide to build internally are making a rational calculation. If no vendor meets the standard, at least an internal build gives you control.

The problem is the assumption that building is a project with a beginning and an end. The underlying models change every six to twelve months. Regulatory requirements evolve - CPS 230 just took effect, and ASIC's guidance on AI in financial services is still developing. Your internal team ships a solution on Claude 3.5, and six months later the model is two generations behind and the compliance landscape has shifted.

Every engineer maintaining your internal AI support system is an engineer not working on your claims platform, your underwriting models, your policy admin system. A specialised vendor amortises the rebuild cost across hundreds of customers while testing against millions of real interactions - FNOL lodgements, endorsement processing, coverage disputes, settlement queries. Your internal build tests against your interactions only.

The honest tradeoff: building gives you control. It also gives you sole responsibility for every failure, with no vendor to share the regulatory exposure. And it gives your compliance team yet another internal system to audit, maintain, and stress-test through every model migration and regulatory change.

What your AI vendor should prove

The technology to handle insurance support safely exists. The question is whether your vendor was built for it.

Auditability by default. Every customer interaction should produce a full decision trail - not just input and output, but every intermediate decision. When a policyholder asks about a coverage exclusion, you should be able to trace the AI's response back to the specific PDS section it referenced, see what alternative responses were considered, and confirm which guardrails were evaluated. Your DOI examiner and ombudsman need this. So does your E&O insurer.

Configurable scope. Your AI should handle what you've approved it to handle, and nothing more. If you haven't approved content for a specific policy type, claims scenario, or hardship process, the system should escalate, not improvise. That means monthly compliance testing before go-live, mandatory human escalation for financial hardship and distress, and topic-level control over what the AI can and cannot say about FNOL, endorsements, coverage, and claims.

Pricing aligned with outcomes. Insurance already thinks in loss ratios and combined ratios - cost per outcome, not cost per seat. If your vendor charges per interaction attempted, they're incentivised to have your AI attempt everything, including the coverage questions it shouldn't touch. Per-resolution pricing creates a different incentive: the vendor only earns when the policyholder's FNOL is lodged, their endorsement is processed, their claims query is resolved - fully and correctly.

No black box. You should be able to explain why your AI said what it said. Not because you trust the vendor's assurance, but because you can see the reasoning yourself.

At Lorikeet, this is the product we built. Architecture designed for regulated industries from day one: auditability, configurable guardrails, per-resolution pricing, and a focus on the insurance workflows that actually carry risk: FNOL-to-settlement, endorsement processing, coverage questions traced to policy wording, and claims status with full decision trails. The question for insurance CX leaders isn't whether AI works for your industry - it does. The question is whether the vendor you're evaluating can prove it.