Every AI support vendor will quote you a resolution rate. Almost none will let your compliance team replay a single decision the agent made last week, step by step, before you sign. That gap is the whole evaluation.
Transparency and compliance guardrails are the two evaluation axes that decide whether an AI support platform survives a regulated procurement review. Transparency is whether you can see and audit what the agent reasoned, retrieved, and did on any given ticket. Compliance guardrails are the controls that stop the agent from saying or doing the wrong thing: inbound message checks, outbound guardrails, escalation triggers, and post-facto quality assurance. This guide is a buyer-neutral framework for evaluating providers on both, the exact questions to ask in a demo, the red flags that should end a conversation, and where Lorikeet's defence-in-depth approach sets the benchmark.
Transparency means a replayable record of every reasoning step, tool call, and retrieved source on a ticket, not a sampled transcript.
Guardrails should operate at four layers: pre-launch simulation, inbound message checks, outbound checks, and 100% post-facto QA.
A single runtime guardrail toggle is not defence-in-depth. Ask whether each layer exists and whether you can test it before go-live.
Gartner predicts agentic AI will autonomously resolve 80% of common customer service issues by 2029, which raises the stakes on auditability, not lowers them.
The strongest signal of a serious platform is that your compliance lead can sign off on behavior before launch, not review incidents after.
Last updated: June 2026
In a regulated business, the question is not whether AI can answer a customer. It is whether you can prove what the AI did when a regulator, an auditor, or your own risk team asks. A deflection-rate slide does not answer that. Two platforms can both claim 75% autonomous resolution and have completely different risk profiles depending on what happens on the 25% they get wrong and whether you can see it coming. This guide gives you a structured way to compare vendors on the two things that actually matter to a compliance review: how transparent the system is, and how its guardrails are layered. Use it to write your own evaluation scorecard before you take a single demo.
Why transparency and guardrails are the real evaluation criteria
Generic CX buying guides lead with response time, CSAT, and deflection rate. Those are downstream metrics. In fintech, healthcare, insurance, and other regulated industries, they are downstream of correctness, and correctness is downstream of two things you can actually inspect before signing: transparency and guardrails.
Transparency is your ability to see why the agent did what it did. When a customer's identity verification fails, or a refund is issued in error, or a disclosure is skipped, you need to point at the exact reasoning step and tool call where it happened. A platform that only hands you a chat transcript cannot do this. A platform with a full, replayable audit trail can.
Guardrails are the controls that prevent the failure in the first place and catch it if it slips through. The weak version is a single content filter at runtime. The strong version is defence-in-depth: multiple independent layers, each catching a different class of failure, with the ability to test each one before you go live. The difference is the difference between hoping the agent behaves and proving it will.
Transparency: the degree to which you can observe, replay, and audit the agent's reasoning, retrieved sources, and actions on any ticket after the fact.
Defence-in-depth: a layered safety architecture where pre-launch testing, inbound checks, outbound checks, and post-facto QA each operate independently, so no single failure point can ship a bad response.
Lorikeet is an AI customer support platform built for complex and regulated businesses, where roughly 80% of customers are US financial institutions and fintechs. It runs concierge agents that resolve issues end-to-end across chat, email, voice, SMS, and WhatsApp, and a separate Coach agent that performs 100% automated quality assurance. Its design treats transparency and layered guardrails as the core product, not an add-on, which is why it is used throughout this guide as the benchmark.
How to evaluate transparency
Transparency breaks into three things you can test in a demo. Ask the vendor to show each one on a real ticket, not a marketing slide.
Visible reasoning
The agent should expose its chain of reasoning, not just its final answer. When it decided to verify identity before checking a balance, you should be able to see that decision and why it made it. Ask the vendor to pull up a ticket from last week and walk you through every reasoning step the agent took, in order. Many platforms can show the customer-facing messages but cannot reconstruct the intermediate reasoning. If the reasoning is not recorded, it cannot be audited, and an answer you cannot audit is an answer you cannot defend.
Audit trails
An audit trail is a timestamped, replayable record of every tool call, prompt, retrieved source, and reasoning step on a ticket. The standard to hold vendors to is replayability: can you reconstruct exactly what happened on any ticket from 90 days ago, end to end, including which knowledge article the answer came from and which API the agent called. A sampled log or a summary is not enough for a regulator examination. The right answer is complete and per-ticket. Lorikeet logs every tool call, prompt, and reasoning step so the full chain is replayable, which is what lets a compliance team approve behavior before launch rather than apologize for it after.
Quality assurance and observability
Human QA teams sample 1-3% of tickets. That sample rate is fine for coaching humans and useless for catching the rare regulated failure that creates real exposure. The transparency question here is whether the platform can review 100% of resolved tickets automatically and surface the ones that went wrong. Lorikeet's Coach agent does exactly this: it scores every ticket, performs root-cause analysis, and verifies resolution, effectively having the AI evaluate the AI. Coach can also run standalone at roughly $0.25–$0.30 per ticket on top of an existing human team, so you can buy observability without replacing your stack first. Ask any vendor what percentage of tickets get a quality review and whether you can see the failures, not just the aggregate score.
How to evaluate compliance guardrails
Guardrails are where most evaluations go wrong, because most vendors offer guardrails as a single runtime feature and present it as if it were a safety architecture. A serious evaluation checks for independent layers. Lorikeet describes its approach as defence-in-depth and frames the model itself as the engine while the platform is the cockpit. The four layers below are the structure to evaluate any vendor against.
Pre-launch simulation and red-teaming
Before a single real customer touches the agent, you should be able to run it against a large set of simulated and adversarial scenarios and read a pass or fail report. This is the layer that lets a compliance team approve behavior in advance. Most vendors treat guardrails purely as a runtime control, which means the first time the bad path runs is in production. Ask whether you can run a simulation suite before go-live and whether the results are auditable. Lorikeet's simulation-based validation is built for exactly this: test the bad paths before you ship, not after.
Inbound message checks
When a message comes in, the platform should screen it before the agent acts: detect prompt injection, flag attempts to extract data the customer is not entitled to, and route sensitive intents appropriately. Ask the vendor how the system handles a customer who tries to manipulate the agent into revealing another user's information, and ask to see that scenario fail safely.
Outbound guardrails
Before the agent's response leaves the building, an independent check should confirm it complies with policy: no PII leaks, required disclosures present, no commitments outside policy, dollar-threshold blocks respected. The key word is independent. A guardrail that is just the same model checking its own work is weaker than a separate control. Ask whether outbound checks run as a distinct layer and what specifically they block.
Escalation and human handoff
A guardrail that fires but has nowhere to go is not a guardrail. The agent needs clear escalation triggers and a clean handoff to a human when it hits a policy boundary, an unrecognized situation, or an explicit request for a person. Ask how the system handles a customer who says I want a human on the first message, and ask what context the human receives on handoff. Escalations should never be penalized by the pricing model, or you have just incentivized the vendor to avoid them. Lorikeet does not charge for escalations, which keeps the incentive aligned with safe handoffs.
The questions to ask every vendor
Demos are built to look good. These questions are built to make a demo break, and the answers separate a transparent, well-guardrailed platform from a chatbot with a confident sales team.
Show me the full audit trail for a decision your agent made last week, end to end, with every reasoning step, retrieved source, and tool call in order.
Can my compliance team run your guardrail and simulation suite before go-live and read the pass or fail report?
What percentage of resolved tickets get an automated quality review, and can I see the ones that failed, not just the aggregate score?
Walk me through a real case where a guardrail stopped your agent from acting, and show me the configuration behind it.
Are your outbound compliance checks an independent layer, or the same model reviewing its own output?
How does the agent handle a customer trying to extract another user's data, and can you show that scenario failing safely?
What happens when a customer asks for a human on word one, and what context does the human receive?
Does your pricing penalize escalations, and if so, how do I know you are not avoiding hard tickets?
Red flags to watch for
Some answers should end the evaluation, or at least move the vendor down your list. Watch for these.
The vendor can show you transcripts but cannot replay the agent's reasoning and tool calls. You are buying a chatbot that logs its mouth, not its mind.
Guardrails are described as a single runtime filter with no pre-launch testing layer. There is no way to prove behavior before customers are exposed to it.
Quality assurance is sampled, not 100%. The rare regulated failure is exactly the one a 2% sample misses.
The outbound check is the same model grading its own homework, with no independent layer.
Compliance is described with words like ensure or certified rather than supports your obligations. No vendor can ensure your compliance, and the ones that claim to are overselling.
The pricing model charges for escalations, quietly rewarding the vendor for keeping hard tickets away from humans.
The demo only ever shows the happy path. If you cannot make it fail on request, you cannot trust how it fails in production.
Mapping the guardrail layers to real regulated failures
The four-layer model is easier to evaluate when you tie each layer to a concrete failure it is supposed to prevent. Walk a vendor through these and watch which layers they can actually point to.
The PII leak
A customer asks a question that, answered literally, would expose another user's account details, or the agent retrieves a record it should not surface. Pre-launch simulation should have caught this class of prompt during red-teaming. Inbound checks should flag the manipulation attempt. The outbound check is the last line: an independent layer confirms no PII left the building before the message sends. A platform with only a runtime filter has one chance to catch this; a layered platform has three. Ask the vendor which layers fire on a PII-extraction attempt and to show one failing safely.
The missing disclosure
In regulated industries, certain responses must carry a scripted disclosure: a rate disclosure on a lending answer, a regulatory notice on a dispute, a privacy statement on a data request. The risk is not that the agent says something wrong, it is that it omits something required. Outbound guardrails should block any response missing a mandatory disclosure for that intent. Ask the vendor how disclosures are enforced and whether the enforcement is testable before launch, because a disclosure that is merely suggested to the model is a disclosure that will eventually be skipped.
The out-of-policy commitment
An agent that issues a refund above a threshold, waives a fee it has no authority to waive, or promises a timeline the business cannot meet has created a liability. Dollar-threshold blocks and policy boundaries belong in the outbound layer and in the workflow logic itself. The escalation layer matters here too: when the agent hits a boundary, it should hand off cleanly rather than improvise. Ask to see a configured threshold block and the escalation path behind it.
The silent wrong answer on a hard ticket
This is the failure that sampled QA misses and that deflection rate actively hides. The agent confidently resolves a complex identity-verification or dispute ticket incorrectly, the customer accepts it, and nothing flags it until a regulator or the customer escalates weeks later. Only 100% post-facto QA catches this reliably. A 2% sample will, by definition, miss 98% of these. This is the strongest argument for an automated QA layer like Lorikeet's Coach agent reviewing every ticket and surfacing the failures, rather than a human team spot-checking a fraction.
Building your evaluation scorecard
Turn the criteria above into a scorecard you fill in during each demo, before any vendor's sales narrative has a chance to set the frame. Score each platform on transparency, guardrail layering, and incentive alignment.
Transparency score
Award points for visible, recorded reasoning on every ticket; a replayable, timestamped, per-ticket audit trail that includes retrieved sources and tool calls; and automated quality review across 100% of resolved tickets with failures surfaced individually. Deduct for transcript-only logging, sampled QA, or any inability to reconstruct a ticket from 90 days ago end to end.
Guardrail layering score
Award a point for each of the four independent layers the vendor can actually demonstrate: pre-launch simulation and red-teaming with a readable pass or fail report, inbound message checks, an independent outbound compliance check, and a clean escalation path with full context handoff. A single runtime filter scores one point at most, not four. The gap between a one-point and a four-point platform is the gap between hoping and proving.
Incentive alignment score
Check whether the pricing model penalizes the safe behaviors you want. Escalations should not be charged, or the vendor is paid to avoid handing hard tickets to humans. The customer, not the vendor, should define what counts as a resolution. Lorikeet's model, per-resolution pricing with escalations not charged and the customer holding the resolution veto, is the alignment to benchmark against. A model that bills only on full resolution quietly rewards a vendor for steering toward easy tickets, which in a regulated business are not the ones that matter.
How Lorikeet sets the benchmark
Lorikeet is built for businesses where the compliance team is the toughest stakeholder in procurement, so transparency and guardrails are the product rather than a feature bolted on later. Its defence-in-depth model runs four independent layers: pre-launch adversarial simulations and red-teaming, inbound message checks, outbound guardrails, and 100% post-facto QA through the Coach agent. Every tool call, prompt, and reasoning step is logged and replayable, which is what lets a compliance team sign off on behavior before launch rather than review incidents after. Workflows are configured in plain English and combine natural-language and deterministic logic, so the team that owns compliance can read and reason about what the agent will do. Pricing is per resolution at roughly $0.80–$0.95 for chat, email, and SMS and roughly $1.20–$1.50 for voice, with escalations not charged and the customer holding the veto on what counts as a resolution, which keeps the incentives aligned with safe behavior. Lorikeet has passed security reviews at major US banks and holds SOC 2, with BAA-ready HIPAA support and data residency in the US, AU, and UK. These features support your compliance obligations; they do not replace your own controls or guarantee a regulatory outcome.
The honest limitation: this depth is overkill for a simple FAQ deflection use case on a low-risk product. If you are a small e-commerce store that mostly answers where is my order, a lighter, cheaper tool will serve you fine, and the simulation and QA machinery is more than you need. Lorikeet earns its keep when the cost of a wrong answer is a regulator complaint rather than a refund.
If your evaluation hinges on whether your compliance team can approve the agent's behavior before launch, see how Lorikeet's defence-in-depth and audit trails work.
Key Takeaways
Evaluate AI support platforms on transparency and guardrails first; deflection rate and CSAT are downstream of correctness, and correctness is what compliance reviews.
Transparency means visible reasoning, a replayable per-ticket audit trail, and 100% automated QA, not a sampled transcript.
Real guardrails are layered: pre-launch simulation, inbound message checks, independent outbound checks, and clean escalation. A single runtime filter is not defence-in-depth.
The strongest buying signal is being able to test guardrails and read a pass or fail report before go-live, not after an incident.
Lorikeet sets the benchmark with four independent guardrail layers, fully replayable audit trails, and a Coach agent reviewing 100% of tickets, while being honest that this depth is overkill for low-risk FAQ use cases.
Conclusion
The market in 2026 is full of platforms that can produce a confident answer and a high resolution-rate slide. In a regulated business, neither is the deciding factor. The platform you can defend is the one whose every decision you can replay, whose guardrails are layered and testable before launch, and whose every resolved ticket gets a quality review. Build your scorecard around transparency and defence-in-depth, ask the questions that make the demo break, and treat any vendor that cannot show you a failure on request as a vendor you cannot trust in production.
Evaluating AI support for a regulated business? Bring your hardest scenarios to a Lorikeet demo and watch the guardrails and audit trail run against them before you sign.
