03. AI Guardrails
Giving supervisors control over generative AI
An observability tool where supervisors see how generative AI models behave, tune them, and control the outcome.
- Client
- AI Guardrails
- Role
- Lead Product Designer
- Sector
- AI observability
- Year
- 2025
Impact at a glance
01. Context
In a customer support contact centre, generative AI writes a summary of every call an agent takes. Supervisors are accountable for those summaries, but tuning a model used to mean editing prompts, a job only engineers could do.
02. The problem
Without a way for non technical people to shape and test the output, every change waited on engineering, and nobody could prove the model behaved as the business needed. Rollouts stalled on risk and compliance questions.
Business goal
Scale generative AI responsibly, with clear accountability and less dependency on engineering.
Customer goal
“Define what a good call summary looks like, test it, and trust the result, without writing a prompt.”
03. Insights
- 01
Supervisors think in outcomes, such as tone, length and what to include, never in prompts.
- 02
Tuning had to be reversible and testable, or nobody would touch it.
- 03
Trust grew when people could compare the model's output against their own rules.
04. Strategy
Start from a use case
Each job the AI does, from Summarization to Sentiment Detection, is a card with its own version history.
Tune with categories
Prompt tuning becomes a set of plain rules grouped by category, so anyone can say what the summary must contain.
Test, then publish
Run the rules against real calls, compare the output, and only publish a version once it behaves as intended.
Chapter 01 · Management
Every AI job, in one place
The Generative AI hub turns each task the model does into a use case card: Summarization, Automatic Disposition, Q&A, Sentiment Detection and more. Supervisors see at a glance what exists, which version is live and when it last changed.
- 01One card per job, so non technical people understand what the AI does
- 02Visible versions on every card, so changes are always traceable
- 03Search and simple navigation instead of configuration files
Chapter 02 · Observability
See what is blocked, and why
A supervisor does not need to read a model to know how it behaves. The guardrails overview opens with three counts: messages blocked, events coming in, events going out. Two charts split what was triggered from what was approved, and break incidents into toxicity, prompt injection, hallucination, personal data and blocked keywords. Below, the incidents are listed one by one with the message itself, the use case it came from and the type that caught it, so the reason is always the words on the screen, never a score.
- 01Blocked messages listed with the incident that stopped them, so the reason is readable at a glance
- 02Inbound and outbound kept apart, so you know whether the risk arrived from the caller or left with the model
- 03Every incident tied to its own interaction, so it can be opened and read inside the call it came from
Chapter 03 · Tuning
Shape the output without writing a prompt
Prompt tuning becomes a set of plain rules grouped by category, such as what to include, tone, length and what to never mention. Each rule is a sentence, not a line of code: the summary should stay under a limit, add the timestamp, add the customer ID, close with the escalation motive. On the right, a simulator replays a real call, so the same screen holds the rule and its consequence. The same language carries the policies: jailbreak, hallucinations, toxicity, compliance and blocked keywords, each one a line a supervisor can read, switch on and fill in.
- 01Categories replace the prompt box, so anyone can define the rules
- 02Policies are written the same way, with the words that are never allowed listed next to the switch
- 03Test against calls where the model scored low, before anything reaches agents
05. Key decisions
Categories instead of a prompt box
Trade-offLess freedom for experts, far more adoption from supervisors.
Preview before publish
Trade-offOne extra step, but safe experimentation and higher trust.
Versions on every use case
Trade-offMore to manage, but every change is accountable and easy to roll back.
Show the blocked words, not just the count
Trade-offMore to read per incident, but the reason is visible, and that is what makes people trust it.
06. Outcome
Supervisors moved from asking engineering for changes to shaping, testing and owning how the AI summarises every call.
What I learned
“With generative AI, control is the feature that makes everything else adoptable.”
Next case
A contact center in 5 minutes