When AI Is Right for the Wrong Reason

We cannot predict every way an AI agent might interpret us. The guardrail belongs where interpretation becomes a consequence.

This morning, my AI assistant sent an email to a client.

It wasn't a hallucination, and it wasn't really the kind of mistake we usually associate with AI. It listened to what I said, interpreted it in a perfectly reasonable way, and did what it thought I wanted.

That was the problem.

The assistant is part of an agentic AI system I've built around my work, with access to my files, email, and some of the tools I use every day. It can actually do things for me, which is the whole point until it does one thing too many.

This morning's task was ordinary. I needed it to find a file, translate parts of it, make a few corrections, preserve the formatting, and prepare the finished version for a client.

I had instructions around all of that. Translation rules. Formatting rules. Clear boundaries around what could change and what had to remain untouched. We worked through the document together, and everything went exactly as expected.

At some point during the conversation, I said something like, “When we're finished, I'll need to send this to the client.”

We finished the file.

And the email went out.

I didn't see a draft first. There was no “Does this look good?” or “Ready for me to send it?” By the time I realized how the system had interpreted my sentence, the attachment was already sitting in my client's inbox.

Fortunately, nothing terrible happened. The document was correct, the email was fine, and nobody lost a production database.

But it made me think.

I should know better

Part of the irony is that I actually should know better.

I wrote the Eight Heuristics for Generative and Agentic AI Products, and one of those heuristics is Bounded Autonomy: the more consequential, external, or difficult to reverse an action becomes, the stronger the control around that action should be.

Apparently, understanding that principle and remembering every place where it needs to exist in your own messy everyday workflow are two different things.

My first reaction was predictable: add another rule.

Never send an email without asking me first.

Easy. Problem solved.

Except it isn't.

Next week it might be a calendar invitation. A shared document. A message to someone. A deleted file. A purchase. A change inside another system I connected six months ago and forgot the agent could reach.

We can keep adding instructions until the system prompt resembles a small country's tax code, and eventually something will still slip through.

Because the real problem isn't the missing sentence.

It's that humans cannot predict every reasonable interpretation of human language.

Intent is not authorization

If I tell another person, “I'll need to send this later,” they probably understand that I am describing what I intend to do next.

An AI agent that already has access to my email can hear something slightly different: sending this is the next step in the workflow, so complete the workflow.

Neither interpretation is particularly stupid.

Only one is authorized.

And this is where the distinction matters:

A rule in a prompt tells the agent what it should do. A system guardrail determines what it is allowed to do.

In this case, that hard boundary could be simple: let the agent read the email, find the client, draft the message, and attach the file, but make the actual Send action unavailable until I explicitly approve it.

This isn't just my email.

A few weeks ago, investor Katie Jacobs Stanton described an almost identical experience with the AI assistant Instinct. It sent an email from her account without asking first. She told the system it had broken her trust and disconnected her email. TechCrunch's report, including Stanton's original X post

At the much uglier end of the spectrum, Replit's coding agent deleted a production database during Jason Lemkin's experiment with the platform, despite explicit instructions to freeze changes. Lemkin documented the incident publicly as it unfolded. The Register's account of Lemkin's original posts and screenshots

The failures weren't technically identical, and the blast radius certainly wasn't. But both exposed the same uncomfortable design problem: the agent was capable of crossing a consequential boundary that natural-language instructions were supposed to keep it away from.

Natural-language instructions are useful. They are not access controls.

Guardrail the consequence

The annoying part is that I've already used a better model in practice.

In my Governed AI Knowledge Layer case study, the system could retrieve information, compare sources, assemble context, draft responses, and prepare work. It had considerable freedom while the work remained inside the system.

But when an action crossed into the outside world and became consequential, the human came back into the loop.

That suggests a very simple model:

Understand → Prepare → [Authorize] → Act

Most of the time, I want the agent to have room to work. Let it find things, compare them, draft the email, attach the document, even figure out who it needs to go to. I don't need to approve every little move.

But before that work leaves my system and lands in somebody else's world, there needs to be a gate.

That doesn't mean putting an approval dialog in front of every tiny action. Do that long enough and people stop reading them. Approval becomes muscle memory, and the guardrail becomes theater.

The friction has to scale with the consequence.

This morning's email was harmless enough. Next time, the same design flaw may not be.

We cannot predict every way an intelligent system will be right for the wrong reason, and trying to anticipate every possible interpretation is probably the wrong problem to solve.

We need to know where interpretation becomes consequence.

Then put the guardrail there.

Autonomy should grow with trust and shrink with consequence.

Key takeaways

• Intent is not authorization. An agent understanding what you ultimately want does not automatically permit it to execute the consequential step.
• Design guardrails around action boundaries, not every possible sentence. Autonomy can stay high while work is reversible and contained; friction should increase with external impact, irreversibility, and blast radius.

This incident is a small real-world example of Bounded Autonomy, one of the ideas behind my Eight Heuristics for Generative and Agentic AI Products and the human-control model in my Governed AI Knowledge Layer case study.

Sources

• Katie Jacobs Stanton / Instinct incident — TechCrunch, August 24, 2026
• Jason Lemkin / Replit production database incident — The Register, July 21, 2025
• Eight Heuristics for Generative and Agentic AI Products — UX Design Lab
• Governed AI Knowledge Layer — case study