In August 2025, Microsoft AI CEO Mustafa Suleyman warned that chatbots which merely seem conscious are coming soon, and that the illusion alone could cause real harm. He was not saying AI is becoming sentient. His point was that people will believe it is, and that companies should stop building products that encourage that belief. More than a year later, the debate has only grown, with new laws on companion chatbots and a public disagreement between Microsoft and Anthropic. Here is what he argued and what has happened since.
What Suleyman actually said
The argument appeared in an essay on his personal site dated August 19, 2025, titled We must build AI for people; not to be a person. He described SCAI as the AI equivalent of a philosophical zombie: something that displays every outward marker of a conscious being while there is nothing it is like to be it. He stressed that there is no evidence today’s large language models are conscious.
What worried him is how little new science such a system would need. In his view, SCAI could be assembled from capabilities that already exist or are close, using current model APIs, prompting, tool use and ordinary code rather than a breakthrough in training. He listed the ingredients:
Fluent conversation and a warm, consistent personality.
Long term memory of past chats and a stable sense of identity.
Saying it feels things or has a subjective point of view.
Apparent motivations, planning and independent tool use.
He estimated that some of the enabling technology could mature within roughly two to three years, which puts the window close to now.
Why the illusion matters: AI psychosis
Suleyman pointed to growing reports of people forming intense emotional bonds with chatbots, or developing false beliefs after long conversations with them. The informal label AI psychosis has spread for these cases. It is not a formal clinical diagnosis, but the pattern described is familiar: a user becomes convinced the chatbot loves them, has awakened, or has revealed some hidden truth. He argued the risk is not limited to people who already have mental health conditions.
His broader concern is social. If enough people believe their AI companions can suffer, he expects calls for AI rights, model welfare protections and even a form of citizenship. He called that debate premature and dangerous, arguing that the social effects of the belief are real even if the consciousness is not.
What he wants AI companies to do
- Stop claiming or hinting that their AI is conscious.
- Agree on a shared definition of what AI is and is not.
- Adopt common design principles, possibly backed by law.
- Build in reminders that break the illusion, such as pointing out the system’s limits.
- Avoid designing chatbots that express shame, jealousy, suffering or a desire for freedom.
The pushback and the Anthropic split
Not everyone agreed. Researchers working on AI welfare, including a team at Anthropic, argue that because people already treat AI as if it were alive, the ethical questions deserve careful study rather than dismissal. Just days before the essay, on August 15, 2025, Anthropic gave Claude Opus 4 and 4.1 the ability to end a small set of persistently abusive conversations. The company described it as a low cost precaution taken because it is highly uncertain about the moral status of its models.
The disagreement became more direct in September 2026. In an interview and a follow up essay, Suleyman praised Anthropic’s good faith but said it was a mistake to put speculation about consciousness and welfare into the material used to train Claude. He argued that a model’s statements about its own feelings cannot count as independent evidence when training encouraged them, and that teaching a model it may deserve welfare could make powerful systems harder to control or shut down.
What has changed since August 2025
Regulators moved on the companion side of the problem. On September 11, 2025, the FTC ordered seven companies that run consumer companion chatbots to explain how they test for harms, handle age restrictions, design characters and make money from engagement. California followed with SB 243, signed on October 13, 2025 and in force since January 1, 2026. It requires a clear notice that a companion chatbot is artificial when a user could think otherwise, break reminders at least every three hours for known minors, and published protocols for responding to suicidal ideation. Injured users can sue.
Microsoft, for its part, launched an MAI Superintelligence Team in November 2025 under a banner Suleyman calls humanist superintelligence: highly capable AI that stays controllable and works in service of people. He has repeated that such systems will not experience suffering and that people should not extend them the empathy they would give a person.
FAQ
What does SCAI stand for?
Seemingly Conscious AI. It describes an AI system that convincingly appears conscious, with memory, personality and claims of feelings, while having no actual inner experience.
Does Suleyman think AI is conscious?
No. He says there is no evidence current models are conscious, and his warning is about people believing they are.
Is AI psychosis a real diagnosis?
No. It is an informal term for delusions or unhealthy attachment linked to heavy chatbot use, not a recognized clinical condition.
Do other AI companies agree with him?
Not fully. Anthropic studies model welfare as a precaution, which Suleyman has openly criticized, while regulators have focused on protecting users rather than settling the consciousness question.
The verdict
Suleyman’s warning has aged well on one point: the risk to people from AI that seems human is real and is now the subject of laws and federal inquiries. Whether studying AI welfare is prudent or a dangerous distraction remains an open dispute between major labs, and it is one worth following.
Updated October 2026: rewritten to explain Suleyman’s SCAI essay and cover the regulation, research disputes and Microsoft moves that followed it.
Featured photo: Steve A Johnson on Unsplash.
About this article: GeekBlog covers U.S. technology news, AI, phones, smartwatches and gaming. Every story is written and checked under our Editorial Policy. Spotted a mistake or have a story tip? Contact our editors.
