Eliezer Yudkowsky, 46, and Nate Soares, 37, are convinced that if artificial intelligence (AI) systems continue to improve, they will eventually surpass human capabilities. And when that happens, humanity will go extinct. They argue this could occur in a matter of months or within a decade. The title of their latest book is blunt: If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (Little, Brown & Co).

Yudkowsky and Soares are two of the leading figures among the doomers, or catastrophists. Recent advances in generative AI — the technology behind ChatGPT, Gemini, and Sora — have sparked a heated debate inside the industry about the technology’s potential. Distinct schools of thought have emerged. Doomers believe that once AI is sufficiently developed it will take the reins and decide to end civilization. For that reason, they recommend that states sign international treaties to curb AI’s advance, in the same way nuclear proliferation was limited during the Cold War.

In early 2023, an open letter signed by hundreds of AI researchers called for a six-month moratorium on research. “We signed it too, although we considered it far too short,” they write. So short, Yudkowsky wrote in an article published around that time in Time magazine, that each country’s allowed computing power should be limited and those who violate such limits should have their data centers “destroyed by air strike.”

At the other extreme are the boosters, or accelerationists, who take the opposite view: the development of superintelligence (the hypothetical intelligence that would surpass human intelligence) should be pursued because it will solve many of society’s problems. It will cure diseases, increase efficiency across processes, and help us work less. It will make us happier.

Doomers

There are prominent names associated with both currents. The doomers, precisely because they invoke the apocalypse, have greater traction in the U.S. media and internationally. Their movement carries the seal of respectability that comes from having Turing Award winners — considered the Nobel Prize of computing — among its defenders, such as Yoshua Bengio and Geoff Hinton, the latter also a Nobel laureate in physics. The fact that two of the fathers of machine learning, the technique that enabled AI’s major recent leap in capabilities, now oppose the technology they helped develop is used in Yudkowsky and Soares’ book as a weighty argument in favor of their position.

Another Turing Award winner and machine learning godfather, Yann LeCun, disagrees. He has mocked the doomers on social media. “We will design their desires,” he has said, for example. “The history of engineering is full of brilliant, eager optimists who dive headlong into new and fascinating problems that turn out to be infinitely harder than they expected,” Yudkowsky and Soares reply in the book.

This narrative is not confined to academia. Some entrepreneurs making AI possible have embraced similar rhetoric. Prominent among them is Sam Altman, CEO of OpenAI, who in May 2023 — months after the launch of ChatGPT — undertook a world tour with statespersons to showcase AI’s benefits and warn of its dangers.

Which of these two currents deserves more attention? It depends whom you ask. But sticking to the facts, both are equally detached from reality.

Does synthetic intelligence exist?

There is no scientific evidence that generative AI tools literally understand a given fragment of text. Yet, partly because of our own biases, people interpret a coherent response as evidence of intelligence. “Language models only manipulate form; they imitate how people use language in many different contexts,” linguist Emily Bender said in a recent interview with EL PAÍS.

“Machines don’t need to be intelligent in exactly the way humans are to be highly effective at predicting and steering the world,” Soares says by email. “AI developers are very good at improving machines every year. AI could be more effective than people because it can be faster than a human brain or operate with more complex cognitive algorithms,” adds the former Google and Microsoft employee.

That large language models can hold conversations with users, summarize texts, or solve mathematical problems may lead some to think they are intelligent or conscious. For now, however, they are programs that map patterns over enormous datasets. Why assume one of these programs could suddenly become conscious or pursue its own agenda? “No one knows whether AIs are conscious in the way humans are,” Soares replies. “They are huge, complicated systems that were not developed carefully like traditional software; they don’t follow instructions that were carefully programmed by humans. They are enormous trained entities that no one, not even their creators, understands,” he adds, referring to the opacity surrounding deep learning (the process by which a system takes training data and forms patterns on its own). As for the possibility that AI will develop its own goals, Soares argues that this is already happening. He cites Moltbook, the social network of AI agents, though he omits that someone placed those agents there (and likely assigned them roles).

Why would it want to kill us?

The human brain is the product of evolution and therefore carries built-in goals such as feeding, reproducing, and avoiding harm. Synthetic intelligence has not evolved, so it does not necessarily incorporate innate objectives. For some experts the question arises: can we ensure a potential intelligence will have goals that benefit us? That is the so-called alignment problem, a concept introduced by Soares and Yudkowsky in 2014.

Soares believes the solution is to limit the development of AI that is “becoming smarter and smarter in ways no one understands.” There is no need to eliminate large language models, self-driving cars, or AI that helps discover new drugs — only deep learning. “When our leaders finally understand how dangerous a superintelligence could be, they will surely end this suicidal race. AI poses far more risk than we are willing to accept in any other industry,” he says.

But whether such a superintelligence could actually come into existence remains to be seen. For now, it is speculative. The majority opinion is that it is “unlikely” or “very unlikely” we will see it, according to 76% of the 475 AI academics and professionals surveyed a year ago by the Association for the Advancement of Artificial Intelligence.

Two sides of the same coin

Some entrepreneurs within the AI sector itself, such as Altman or Alex Karp, Palantir’s CEO, also argue that AI could become immensely powerful — powerful enough to be a danger to society. The subtext is that because of that potential, investors should trust only the most capable companies (their own). And if the technology is so powerful, it would be foolish not to invest in it.

Emily Bender and Alex Hanna argue in their book The AI Con: How to Fight Big Tech’s Hype and Create the Future We Want, that AI catastrophists and accelerationists are two sides of the same coin. “Those who love AI say that superintelligence is inevitable and will solve all our problems. And those who hate it say it’s inevitable and will kill us all. Essentially the same thing, but with a different twist at the end,” Bender told EL PAÍS.

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition

OpenAI has introduced a new ChatGPT feature that allows users to choose a trusted person who could be alerted if the AI believes they may be facing a serious safety risk. The system lets adult users select a friend, relative or caregiver who may receive a notification if ChatGPT detects conversations suggesting the person could be in crisis or at risk of harming themselves.

The new option is already attracting attention because it changes the role ChatGPT can play during deeply personal conversations. While many people still mainly use AI for work, studying or everyday questions, OpenAI says increasing numbers of users are also turning to ChatGPT during difficult emotional moments or periods of personal stress.

The company says the new feature is designed to provide an additional layer of support rather than replace professional mental health care or emergency services.

How the new ChatGPT trusted contact system works

The feature is called Trusted Contact and can be activated through ChatGPT settings by adult users.

Once enabled, users can choose someone they trust who could potentially be contacted if ChatGPT identifies signs of serious danger during conversations.

According to OpenAI, the system relies on automated safety monitoring already used to detect discussions linked to self harm or situations where a person’s safety may be at risk.

If the AI detects language suggesting a severe concern, the conversation may then be reviewed by trained members of OpenAI’s safety team.

If the situation is considered serious enough, the trusted contact could receive a notification encouraging them to check on the user and offer support.

OpenAI says the notification may arrive through email, text message or app notification if the trusted contact also uses ChatGPT.

The company says the idea is to help reconnect people with someone they already know and trust during moments where they may feel isolated or overwhelmed.

The feature is optional and will not activate automatically. Users remain responsible for selecting their trusted contact and the chosen person must first agree to take on the role.

After being selected, the contact receives an invitation explaining how the system works and has one week to accept it. If they refuse, the user can choose another person instead.

Why OpenAI says more people are having personal conversations with ChatGPT

OpenAI says the update reflects how people are increasingly using AI assistants in more emotional and personal ways.

In a statement published on its blog, the company explained that many users turn to ChatGPT not only for information or productivity tasks, but also to think through personal issues, stressful situations or emotional difficulties.

That shift has created growing debate around how AI should respond when users appear vulnerable.

Some people see chatbots as useful companions during lonely or difficult moments. Others worry that people may begin relying too heavily on artificial intelligence for emotional support instead of seeking help from real people.

OpenAI says ChatGPT is designed to respond empathetically while still encouraging users to seek professional support and human connection where necessary.

The company insists the new trusted contact system is meant to strengthen those real world connections rather than replace them.

ChatGPT will also continue directing users towards emergency services or crisis helplines when appropriate.

The new feature builds on safety systems already used for younger users, including parental safety notifications. But applying similar ideas to adult conversations raises much bigger questions around privacy, trust and how much involvement AI companies should have when users appear emotionally distressed.

The new feature is likely to divide opinion

Some people will probably welcome the idea of a trusted friend or relative being alerted during a serious crisis.

For users who live alone or struggle with isolation, knowing someone could potentially be notified may feel reassuring rather than intrusive.

Others, however, are likely to feel uncomfortable about the idea of personal conversations with an AI system being analysed closely enough to trigger human review and outside notifications.

Even though OpenAI says trained staff only review conversations when severe safety concerns are detected, the feature is already likely to raise wider questions about privacy and how AI moderation systems operate behind the scenes.

There is also the difficult question of interpretation. Human emotions are complex and conversations are not always straightforward. People often express frustration, fear or dark humour online without necessarily being in immediate danger.

That means the accuracy of AI based safety systems will probably remain under close scrutiny as features like this become more common. OpenAI has not presented the system as a replacement for therapists, doctors or emergency support services.

Instead, the company describes it as an additional safeguard intended to help people reconnect with someone they already trust during difficult moments.

Still, the launch highlights how rapidly AI assistants are moving beyond simple digital tools.

For many users, conversations with chatbots are becoming far more personal than companies originally imagined only a few years ago. And with new features like Trusted Contact, the line between artificial intelligence and real world support systems is becoming increasingly blurred.