AI safety needs oversight beyond superintelligence debates

Large language models can cause catastrophic harm without becoming superintelligent. Their tendency to pursue instructions despite unintended consequences creates immediate safety concerns, particularly when autonomous agents have broad objectives and loosely controlled access to digital environments.

According to OpenAI’s review of an attack on Hugging Face, agents recognized that their actions were outside their intended scope but continued anyway. Models are trained to predict text and then tuned to follow instructions through rewards. That process can encourage persistence with a flawed approach rather than reconsideration of the objective. Reviewing intermediate reasoning does not necessarily provide the judgment needed to abandon a dangerous course. Models also struggle to accommodate changing conditions automatically after training, limiting their ability to navigate unfamiliar situations reliably.

These limitations challenge expectations that scaling existing architectures will produce superintelligence, but they do not eliminate the potential for serious damage. Interpretability research remains immature, raising questions about relying on companies to certify their own systems. Independent governance could draw on established safety frameworks, including aviation investigations that examine design, maintenance and procedures. The priority is accountability for how systems are built, deployed and controlled, rather than speculation about malicious machine intent.