The alarm bells have been rung. In May, computer scientist Geoffrey Hinton, known as the “godfather of AI”, quit his role at Google to warn of the “existential threat” posed by artificial intelligence. The Center for AI Safety followed up with an open letter, signed by Hinton and hundreds of others, warning that advanced AI could destroy humanity.
Generally, the people who talk about existential risks reckon that we are on a trajectory towards artificial general intelligence , roughly defined as machines that can out-think humans. They predict that people will invest advanced AIs with more autonomy, giving them access to vital infrastructure, such as the power grid or financial markets, or even putting them at the forefront of warfare – at which point they could go rogue or otherwise resist our attempts to control them.