As Irving Good realised in 1965, machines with superhuman intelligence could repeatedly improve their design even further, triggering what Vernor Vinge called a 'singularity.'

The quote by Stephen Hawking—“As Irving Good realised in 1965, machines with superhuman intelligence could repeatedly improve their design even further, triggering what Vernor Vinge called a 'singularity.’”—explores the concept of artificial intelligence and its potential to surpass human control. Hawking refers to mathematician Irving John Good, who theorized about the creation of an “intelligence explosion,” in which a machine smarter than humans could redesign itself, leading to a rapid acceleration of technological progress. This self-improvement loop could reach a tipping point, beyond which human understanding and control might no longer apply.

The meaning of the quote lies in its warning about the transformative power of AI. Hawking highlights the possibility that once superhuman intelligence exists, it would not stop at matching human abilities but would exponentially improve itself. This process, described as the singularity by science fiction author Vernor Vinge, envisions a future where technological change occurs so quickly and unpredictably that human society is fundamentally reshaped. Hawking’s words express both awe at the potential and concern about the risks of losing control.

The origin of this quote lies in two intellectual milestones. First, Irving Good’s 1965 essay on the “intelligence explosion,” where he predicted that the first ultraintelligent machine would be humanity’s “last invention,” since it could then create even better machines. Second, Vernor Vinge’s 1993 essay, in which he popularized the term singularity to describe this runaway technological moment. Hawking, building on their insights, brought these ideas into mainstream scientific and public debate, especially in the early 21st century, when AI research began advancing rapidly.

In a broader sense, Hawking’s statement reflects his concern with the existential risks posed by advanced AI. He often warned that while AI could bring enormous benefits, from curing diseases to solving environmental challenges, it also carried the danger of outpacing human control. By citing both Good and Vinge, Hawking positioned himself within a lineage of thinkers urging society to take seriously the long-term consequences of artificial intelligence and to prepare for the possibility of a technological singularity.

Would you like me to also connect this to today’s discussions about AI safety and how Hawking’s warning still shapes debates on regulation and ethics?