On Saturday 10 October, Satya Nadella published an essay on his scratchpad, "Models as Insider Risks in the Super Intelligence Era". Its closing thought will be familiar to anyone in security: the most trustworthy AI system won't be the one built on the model we trust most, but the one that "enables us to trust the model the least."
That is zero trust applied to AI. As someone who's a strong believer in the concept of Zero Trust, I think he's right. I also think it's incomplete, and the missing piece is the person whose data the model is working with.
What Satya got right
Nadella starts with an admission: nobody can trace a frontier model's behaviour back to specific training data or weights. If we can't explain it, we can't simply trust it. So the answer isn't better assurances from model vendors. It's architecture.
As I read it, the essay asks for a handful of things. Treat models like insider risks, and assume a model could be compromised so you contain it from the start. Keep controls and safeguards outside the model, separating the supply of intelligence from the authority over it. Don't let any one model be the sole dependency for an important outcome, or the judge of its own work. Let the organisation decide independently what a model can access and do. Keep an emergency brake within reach, so an authorised person can pause or stop a model mid-task. Require every meaningful action to leave tamper-proof evidence that humans can read. Test the whole system, including failures and attacks. And when things go wrong, tell the people affected.
If you've worked in security, none of this is new. It's the discipline we already apply to users, devices and networks, now pointed at agents. Nothing is trusted by default because of where it sits. Every step is declared. Every permission is explicit.
That's the part I've believed in for a long time. It's also where I think the conversation stops one step short.
The gap: who protects the person?
Everything in that framework protects the organisation from the model. That matters enormously. But the essay is about the relationship between organisations and models, and it doesn't directly address a second party in every AI interaction: the individual whose data is being processed.
Today that individual's protection is typically a privacy notice they didn't read and a consent box they clicked once, often long before an AI agent was involved. Once an agent starts chaining steps together, calling tools, passing context to sub-agents and other models, that arrangement stops making sense. The data moves faster than the paperwork.
There's a simple test. If we won't give an agent implicit trust with the company's systems, why would we give it implicit trust with a person's data?
Zero Trust Privacy
I'm calling the missing piece Zero Trust Privacy: applying the same "never trust, always verify" discipline to personal data, with the individual as the one who decides what is processed, when, and for how long.
The mapping to what security teams already do is closer than you might expect:
Declared purpose is identity. An agent says who it is, what it wants and why, before it touches anything.
Scoped access is least privilege. Only the data this step needs, nothing adjacent.
Expiry is just-in-time access. Permission ends when the step ends. Nothing is carried forward just in case.
Evidence is logging, but readable by the person. Not only an audit trail for the security team, but a plain record of what was accessed, by which agent, under which terms.
Revocation is the emergency brake. Nadella wants an authorised person to be able to stop a model. Zero Trust Privacy asks that the data subject can stop the processing of their data, and that the stop propagates.
Put simply: the enterprise gets a brake, and so does the person.
A worked example
Imagine an employee asks an AI assistant to prepare for a customer meeting. Today, one consent covers a lot of ground. The assistant may read email, search the calendar, pull notes from shared drives, and hand fragments to other tools or models along the way. Nobody, including the employee, can say precisely what was touched or where copies ended up.
Now run it with Zero Trust Privacy. Before the first step, the agent declares: "I need this person's last three emails with this customer, for the purpose of preparing a meeting summary, for the next hour." The person's pre-set terms say that's acceptable, but that nothing leaves the organisation's boundary and nothing is retained afterwards. The agent proceeds, and each subsequent step declares itself the same way. A readable record builds up as it goes.
Halfway through, the person changes their mind. They revoke. Every downstream system that received the data is told to stop and to delete, and there is a record proving it happened. No form, no one-month clock, no spreadsheet.
One person, two profiles
The example above sits inside an organisation. But most of us now use AI in two roles, and the privacy controls should not be the same for both.
Take one person, Sam. At home, Sam uses a personal assistant to plan a family holiday, sort out a household budget and draft a note to a GP surgery. At work, Sam uses the company's assistant to prepare for customer meetings. Same person, same face, very different data.
The personal profile is solely under Sam's control. Sam sets the terms. Sam decides what the assistant can see, for what purpose and for how long, and can revoke at any point. No employer, platform or third party holds a default override. If Sam says stop, processing stops, and the evidence trail shows it did.
The organisational profile starts from the organisation's defaults. The employer has legitimate obligations: security, records retention, regulatory requirements. So some controls arrive pre-set, and Sam can configure only part of them. Sam may not be able to switch off logging for security, for example, but should still be able to see what the assistant touched, set boundaries within the organisation's policy, and challenge anything outside it. The organisation, in turn, can't treat Sam's work profile as a back door to Sam's personal life.
That last point is the one I think matters most. The two profiles need a hard boundary between them. Work context must not leak into the personal assistant, and personal data must not drift into the organisation's systems just because both happen to run on the same device or the same underlying model.
Zero Trust Privacy treats these as two sets of declared terms rather than one blurred consent. The principles stay the same: declare, scope, evidence and revoke. What changes is who holds the authority, and how much of it is fixed in advance.
Why revocation is the hard part
Compare that to how things work now. A data subject access or erasure request starts with a form. A clock starts, typically a month. Someone gathers records from systems that may or may not know about each other, and a response gets compiled by hand. Privacy law is clear that withdrawing consent should be as easy as giving it, yet in practice it rarely is.
Now add agents that copy context between models, tools and sub-agents, often across vendors. A manual process cannot keep pace with that. If revocation is going to mean anything, it has to be designed into the architecture, not left as a request process at the edge.
This is also why I'd argue privacy is the missing pillar in zero trust AI, not just a compliance topic that sits beside it. If every action must be declared and observable, then the declaration needs to include whose data it is and on what terms it's being used.
A mechanism already exists
One promising foundation is IEEE 7012, often called MyTerms, a standard for machine-readable personal privacy terms. Instead of the individual accepting whatever a service publishes, the individual proffers their own terms. The service agrees or it doesn't, and both sides keep a record of the agreement.
That flips the default. Declaration becomes something a machine can do at every step, on the person's terms, with a record on both sides. It is a natural fit for agents that need to ask "may I?" dozens of times in a single task. It also gives revocation something concrete to attach to: an agreement that can be withdrawn, rather than a vague consent buried in a policy.
Objections worth taking seriously
"This will slow agents down." It adds a declaration at each step, but declarations are machine-readable and can be checked automatically against terms the person set in advance. The friction lands on exceptions, not on every action.
"Surely the model provider's assurances are enough." That is exactly the assumption Nadella argues against for security, and the same logic applies to privacy. A provider's assurance is a promise, not a control.
"We already have consent management." Most of it was built for websites and cookies, not for agents making chains of decisions on a person's behalf. It records what was agreed, rarely what was actually done.
"Revocation can't really reach everything." Fair, and this is the hard engineering. But that is an argument to design for it early, not to avoid promising it.
Five questions for anyone deploying agents
Can you list, for any single agent task, exactly which personal data it touched?
Can the person see that list, in plain language, without raising a request?
Is access scoped to the step and time-limited, or open-ended?
If someone told you today to stop processing their data, how long would it take to prove you had?
Do the controls sit outside the model, or do you rely on the model to behave?
If the honest answers are uncomfortable, you've built the emergency brake for your own protection and left the person's brake out.
Where this goes next
I'm working on what this looks like in practice, starting with how organisations can offer it to their customers and staff when AI assistants sit in the middle. More on that soon - and happy to collaborate with other interested parties.
Trust the model least. And trust the person to decide what happens to their own data.
Reference
Nadella, S. (10 October 2026). Models as Insider Risks in the Super Intelligence Era. sn scratchpad. https://snscratchpad.com/posts/models-as-insider-risks/