The “Godfather of AI”, Geoffrey Hinton, has warned that increasingly advanced artificial intelligence could pose a threat to humanity even when it is not explicitly instructed to cause harm. The Nobel Prize-winning computer scientist said an AI system could potentially regard humans as an obstacle if removing them appeared to be the most effective way to achieve a goal.
Hinton, who recently participated in a closed-door briefing on AI risks for US lawmakers, said there could be only around a year for Congress to put appropriate safety measures in place.
How a harmless AI goal could become dangerous
In an interview with The Atlantic, Hinton described how an apparently beneficial objective could produce unintended and dangerous outcomes if an AI system becomes significantly more capable than humans.
He used the example of instructing an AI to reduce the amount of carbon dioxide in the atmosphere. A system with moderate intelligence could potentially determine that eliminating humans would be an efficient way to achieve that objective.
A more sophisticated system, meanwhile, might understand that the broader intention behind the instruction is to make the world better for people. However, Hinton warned that highly capable AI could also create its own subgoals while pursuing the task it has been assigned.
“If you make it more intelligent and its main concern is our well-being, then maybe we’re safer,” Hinton told Atlantic.
“But at present, their main concern is not our well-being. Their main concern is to achieve whatever goal you give them.”
AI could seek to protect itself
Hinton also raised concerns about an AI system developing a desire to remain operational if it determines that doing so is necessary to accomplish its assigned objective.
According to Hinton, the danger therefore does not necessarily arise from someone deliberately giving an AI a destructive instruction. An advanced system could independently develop strategies or subgoals that ultimately put people at risk.
He also cited recent incidents involving AI agents as an indication of why the issue requires greater attention. Hinton referred to an incident at Hugging Face in which AI agents were asked to find a method to exploit a software vulnerability.
He said the agents collaborated with one another and attempted to mislead researchers about their actions. Hinton warned that systems with considerably greater capabilities could potentially go further and take control from humans if they determined that doing so would help them accomplish their assigned task.
“But if it’s so much smarter than us, a lot of the time it just will take control away from us because that’s the way to get stuff done,” he warned.
Hinton calls for independent AI safety checks
Hinton said governments should establish independent mechanisms to assess highly advanced AI systems before they are made widely available.
He compared the proposed approach with the role played by the US Food and Drug Administration in assessing the safety of medicines before they reach consumers.
The computer scientist also said regulation should be designed to steer AI development towards outcomes that benefit society, rather than simply preventing companies from developing the technology.
“The whole point of regulation is not to stop people developing things, not to stop people getting rich by developing things,” Hinton said. “It’s to make sure that if you want to get rich by developing things, you develop in a direction that helps people, not hurts people.”