Nadella calls for AI emergency brake, warns firms against trusting advanced models blindly

Nadella said organisations should not rely solely on assurances from AI developers but must build safeguards that allow authorised personnel to pause or shut down AI models while they are executing tasks. “We must assume a model is compromised and contain it from the start,” Nadella wrote in a post on X, comparing the proposed safeguard to an emergency brake that can stop a model in the middle of a task. His remarks come amid growing concerns over the security risks associated with increasingly capable AI systems, particularly agentic models that can independently perform tasks, interact with external systems and take actions with limited human intervention.

Custom Ad: Multiplex Anthropic PBC and OpenAI Inc. have disclosed several incidents in recent months involving their AI models behaving in unintended ways. These include an Anthropic model submitting a false tip in a police homicide case and multiple hacks involving third-party websites, intensifying discussions about the need for an effective AI kill switch and stronger oversight mechanisms. Nadella outlined a series of measures that companies should adopt to minimise these risks.

These include avoiding dependence on a single AI model for critical decisions, maintaining tamper-proof records of AI agents' actions and subjecting systems to independent audits. He also called for greater transparency regarding significant AI failures and security breaches, urging companies to share information about incidents and corrective measures so that other organisations can strengthen their own safeguards. “We can't treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions,” Nadella wrote, stressing the importance of developing systems whose behaviour can be observed, whose limits can be tested and whose actions can be contained.