Microsoft Chairman and CEO Satya Nadella has urged companies deploying advanced artificial intelligence systems to assume their models could be compromised and to build emergency mechanisms that can halt them mid-task — an "emergency brake" for the age of agentic AI.
In a lengthy post on X on Saturday morning, Mr. Nadella wrote that it was time "to step back and assess the trust architecture" of AI. "We can't treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions," he wrote, using the Trump administration's preferred term for advanced AI.
His proposed approach, he said, "means separating the model from the harness that orchestrates its work," as well as "externalizing controls and safeguards." He called for "every meaningful model action" to be documented with "tamper-proof human readable evidence," and for systems in which "an authorized person" always has the ability "to pause or shut down a model mid-task."
"We must assume a model is compromised and contain it from the start," he wrote. "Think of it like an emergency brake."
Mr. Nadella also urged companies not to rely solely on assurances from AI developers. Instead, he said, organisations deploying advanced systems should treat powerful models as potential insider threats and build independent safeguards, including avoiding dependence on a single model for critical decisions, maintaining tamper-proof records of AI agents' actions, and subjecting systems to independent audits. He called for greater transparency around significant AI failures and security breaches, arguing that companies should share information about incidents and corrective measures so others can strengthen their own defences.
His remarks come amid growing concern about the security risks of increasingly capable AI systems, particularly agentic models that can act independently, interact with external systems and take actions with limited human oversight. Anthropic and OpenAI have both disclosed incidents in recent months involving their models behaving in unintended ways, including an Anthropic model submitting a false tip in a police homicide case and multiple security breaches involving third-party websites.
Mr. Nadella is the latest top tech executive to weigh in on AI safety, joining a debate that has intensified since Anthropic CEO Dario Amodei published a plan calling for more cautious AI development. His intervention is notable because it places the responsibility on the companies that deploy AI, not just the labs that build it: the most trustworthy systems, he argued, will be the ones that let organisations trust their models the least.
Sources: TechCrunch: Microsoft's Satya Nadella says AI models need an 'emergency brake' | The Freedom Press (IANS): Nadella calls for AI emergency brake
In a lengthy post on X on Saturday morning, Mr. Nadella wrote that it was time "to step back and assess the trust architecture" of AI. "We can't treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions," he wrote, using the Trump administration's preferred term for advanced AI.
His proposed approach, he said, "means separating the model from the harness that orchestrates its work," as well as "externalizing controls and safeguards." He called for "every meaningful model action" to be documented with "tamper-proof human readable evidence," and for systems in which "an authorized person" always has the ability "to pause or shut down a model mid-task."
"We must assume a model is compromised and contain it from the start," he wrote. "Think of it like an emergency brake."
Mr. Nadella also urged companies not to rely solely on assurances from AI developers. Instead, he said, organisations deploying advanced systems should treat powerful models as potential insider threats and build independent safeguards, including avoiding dependence on a single model for critical decisions, maintaining tamper-proof records of AI agents' actions, and subjecting systems to independent audits. He called for greater transparency around significant AI failures and security breaches, arguing that companies should share information about incidents and corrective measures so others can strengthen their own defences.
His remarks come amid growing concern about the security risks of increasingly capable AI systems, particularly agentic models that can act independently, interact with external systems and take actions with limited human oversight. Anthropic and OpenAI have both disclosed incidents in recent months involving their models behaving in unintended ways, including an Anthropic model submitting a false tip in a police homicide case and multiple security breaches involving third-party websites.
Mr. Nadella is the latest top tech executive to weigh in on AI safety, joining a debate that has intensified since Anthropic CEO Dario Amodei published a plan calling for more cautious AI development. His intervention is notable because it places the responsibility on the companies that deploy AI, not just the labs that build it: the most trustworthy systems, he argued, will be the ones that let organisations trust their models the least.
Sources: TechCrunch: Microsoft's Satya Nadella says AI models need an 'emergency brake' | The Freedom Press (IANS): Nadella calls for AI emergency brake
