Assume AI Compromise

Microsoft CEO Satya Nadella recently made an important point about the future of AI security: build systems that assume AI models can be compromised.

He is right. And the implications go beyond cybersecurity.

For decades, security professionals have followed the principle of assume breach. Do not assume defenses will always work. Design systems to contain damage when something goes wrong.

AI agents deserve the same treatment. In his recent post on AI trust architecture, Nadella called for separating models from the systems that orchestrate them, documenting meaningful model actions, and ensuring an authorized person can pause or shut down a model mid-task. TechCrunch's report summarizes the argument.

Engineering Safety Is Not the Same as Alignment

One approach to AI safety focuses on alignment: training models to follow human values, exercise judgment, and behave responsibly.

But giving increasingly capable agents a sense of independent authority, their own moral philosophy, or permission to act as conscientious objectors can magnify the control problem. We should not depend on an agent's internal judgment to determine whether it should obey its operator.

The engineering approach is different: separate the supply of intelligence from authority over it.

Surround non-deterministic models with deterministic controls:

• Strict permissions and privilege limits.
• Continuous monitoring and observability.
• Independent policy enforcement.
• Comprehensive logging and audit trails.
• Reliable containment and shutdown mechanisms.

Treat powerful AI agents as potential insider risks, not as moral authorities whose independent judgment overrides organizational controls.

The most trustworthy system is the one that requires us to trust the model the least.

Alignment can improve model behavior. It cannot replace engineering controls.

But Authorized Does Not Mean Correct

Now consider an autonomous marketing agent.

It sees conversion declining, analyzes CRM data, and concludes that pricing is too high. Within seconds, it can rewrite offers, change targeting, and launch campaigns.

The agent may be operating entirely within its permissions.

But what if its conclusion is wrong?

What if customers are concerned about implementation complexity rather than pricing?

An authorized action based on an unsupported assumption can still be a costly mistake. Security controls answer whether the agent is allowed to act. They do not establish whether customer evidence supports the decision.

That is where the ReadingMinds Customer Evidence Layer comes in. ReadingMinds connects conclusions to real customer conversations, grounded transcripts, expression signals, supporting and contradictory evidence, and research-quality evaluations. Expression signals describe how a response is expressed in the conversation, not a person's private state. Learn more in our Trust & Compliance Center.

The goal is to establish whether customer evidence supports the proposed decision before an agent acts on it.

Enterprise AI therefore needs two independent safeguards:

Security controls: Is the agent authorized to act?

Evidence controls: Is the decision adequately supported?

Neither should depend solely on the model's own assessment.

The ability to generate an answer is becoming commonplace. Demonstrating why that answer deserves to influence a business decision remains the harder problem.

Read more about why customer evidence is the missing layer in the AI agent economy, or see how an adaptive conversation works in a short Live Test Drive.