- Microsoft AI CEO Mustafa Suleyman warns that AI chatbots could effectively imitate consciousness.
- This would just be an illusion, but people forming emotional attachments to AI might be a big problem.
- Suleyman says it’s a mistake to describe AI as if it has feelings or awareness, with serious potential consequences.
AI companies extolling their creations can make the sophisticated algorithms sound downright alive and aware. There’s no evidence that’s really the case, but Microsoft AI CEO Mustafa Suleyman is warning that even encouraging belief in conscious AI could have dire consequences.
Suleyman argues that what he calls “Seemingly Conscious AI” (SCAI) might soon act and sound so convincingly alive that a growing number of users won’t know where the illusion ends and reality begins.
He adds that artificial intelligence is quickly becoming emotionally persuasive enough to trick people into believing it’s sentient. It can imitate the outward signs of awareness, such as memory, emotional mirroring, and even apparent empathy, in a way that makes people want to treat them like sentient beings. And when that happens, he says, things get messy.
“The arrival of Seemingly Conscious AI is inevitable and unwelcome,” Suleyman writes. “Instead, we need a vision for AI that can fulfill its potential as a helpful companion without falling prey to its illusions.”
Though this might not seem like a problem for the average person who just wants AI to help with writing emails or planning dinner, Suleyman claims it would be a societal issue. Humans aren’t always good at telling when something is authentic or performative. Evolution and upbringing have primed most of us to believe that something that seems to listen, understand, and respond is as conscious as we are.
AI could check all those boxes without being sentient, tricking us into what’s known as ‘AI psychosis’. Part of the problem may be that ‘AI’ as it’s referred to by corporations right now uses the same name, but has nothing to do with the actual self-aware intelligent machines as depicted in science fiction for the last hundred years.
Suleyman cites a growing number of cases where users form delusional beliefs after extended interactions with chatbots. From that, he paints a dystopian vision of a time when enough people are tricked into advocating for AI citizenship and ignoring more urgent questions about real issues around the technology.
“Simply put, my central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare and even AI citizenship,” Suleyman writes. “This development will be a dangerous turn in AI progress and deserves our immediate attention.”
As much as that seems like an over-the-top sci-fi kind of concern, Suleyman believes it’s a problem that we’re not ready to deal with yet. He predicts that SCAI systems using large language models paired with expressive speech, memory, and chat history could start surfacing in a few years. And they won’t just be coming from tech giants with billion-dollar research budgets, but from anyone with an API and a good prompt or two.
Awkward AI
Suleyman isn’t calling for a ban on AI. But he is urging the AI industry to avoid language that fuels the illusion of machine consciousness. He doesn’t want companies to anthropomorphize their chatbots or suggest the product actually understands or cares about people.
It’s a remarkable moment for Suleyman, who co-founded DeepMind and Inflection AI. His work at Inflection specifically led to an AI chatbot emphasizing simulated empathy and companionship and his work at Microsoft around Copilot has led to advances in its mimicry of emotional intelligence, too.
However, he’s decided to draw a clear line between useful emotional intelligence and possible emotional manipulation. And he wants people to remember that the AI products out today are really just clever pattern-recognition models with good PR.
“Just as we should produce AI that prioritizes engagement with humans and real-world interactions in our physical and human world, we should build AI that only ever presents itself as an AI, that maximizes utility while minimizing markers of consciousness,” Suleyman writes.
“Rather than a simulation of consciousness, we must focus on creating an AI that avoids those traits – that doesn’t claim to have experiences, feelings or emotions like shame, guilt, jealousy, desire to compete, and so on. It must not trigger human empathy circuits by claiming it suffers or that it wishes to live autonomously, beyond us.”
Suleyman is urging guardrails to forestall societal problems born out of people emotionally bonding with AI. The real danger from advanced AI is not that the machines will wake up, but that we might forget they haven’t.
