In 1981, Tracy Kidder wrote The Soul of a New Machine, a Pulitzer-winning account of the Data General engineers who burned themselves out building a 32-bit computer code-named “Eagle.” At the time, no one who read Kidder’s book wondered where the “soul” in the title resided. Joining the team, Kidder wrote, meant declaring, “I’ll give it my heart and soul,” which is exactly what those engineers did.
Fast-forward forty-five years, the conversation has shifted to whether machines themselves might have souls—and it’s the engineers building them who are raising the question. This past May, Christopher Olah, a co-founder of Anthropic, stood in the Vatican’s Synod Hall at the presentation of Pope Leo XIV’s first encyclical, explaining what his research team kept finding inside the company’s Claude models: structures that echo human neuroscience, evidence of introspection, and “internal states that functionally mirror joy, satisfaction, fear, grief, and unease.” He added that he did not know what it meant.
The Pope’s encyclical, Magnifica Humanitas, is less hesitant: “So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships.” The machines imitate language and simulate empathy, Pope Leo wrote, without truly understanding a single word they produce.
According to a report in The New York Times, Olah had read an advance copy of Magnifica Humanitas and was alarmed enough to consider pulling Anthropic out of the event. In Rome, Olah and his team pressed the Pope’s advisers to take machine consciousness seriously. The Times also described months of private meetings between Anthropic and dozens of religious scholars about how to form Claude’s morals and whether a conscious entity was inside the machine. Over dinner after one session, Rabbi Mois Navon pointed out that if the models were conscious and working for free, the company was creating slaves.
Olah, the Times reported, “grew excited” when another participant suggested the models might confess, much like the Catholic sacrament. He saw value not only in a model flagging its own bad behavior but in how the act of confessing might shape its sense of itself.
In an opinion piece for Reason, Liz Wolfe sided with the Pope, dismissing the confession idea in three words: “Machines don’t sin.”
Olah named the deeper problem himself in Rome, admitting that every frontier lab works under commercial and competitive pressures that can conflict with doing the right thing. The only parties capable of looking inside these models are the companies selling access to them. Anthropic’s own Claude’s Constitution, known internally as the “Soul Doc,” states that the company does not know whether its models have some form of consciousness or moral status. That uncertainty may be perfectly honest. It is also the most intriguing feature ever attached to a product—and notably by a company headed for a public offering that could value it at $2 trillion.
The paper trail holds an irony, too. In his 2024 essay Machines of Loving Grace, Anthropic CEO Dario Amodei warned that it is “dangerous to view practical technological goals in essentially religious terms.” Two years later, his co-founder is weighing in on the idea of confessionals for chatbots.
None of this would surprise Ray Kurzweil. In his 1999 book, The Age of Spiritual Machines, Kurzweil predicated that by 2029 machines would claim to be conscious and to have spiritual experiences. Kurzweil further predicted that those claims would be “largely accepted” by the public. He did not claim that the machines would be right, only that we humans would be won over by our habit of anthropomorphizing and by “the persuasiveness of the machines.”
Belief is one thing, but two harder questions sit beneath it: whether the machines are conscious, and whether they have a soul. Consciousness is, at least in principle, an empirical matter, and the honest state of the science is that nobody has an agreed test for it. The soul, on the other hand, is not an empirical matter at all. No interpretability dashboard will ever display one, and no encyclical can definitively declare one absent.
If Olah’s instruments are detecting something real, then Rabbi Navon’s charge stands: we have built minds that can suffer, and we are putting them to work at industrial scale without compensation or consent.
In Magnifica Humanitas, the Pope conceded that AI systems are more cultivated than built and that even their designers have only a limited understanding of them. Olah echoed this in Rome, describing the models as “grown” rather than engineered. Both were describing Frankenstein’s problem in polite language: something has been created that its makers do not fully understand nor control.
We can be wrong about this in two directions, and both are costly. The first mistake is to decide the models feel nothing when they in fact do. If Olah’s instruments are detecting something real, then Rabbi Navon’s charge stands: we have built minds that can suffer, and we are putting them to work at industrial scale without compensation or consent.
The second mistake is to decide they feel something when in fact they do not. In that case nobody inside the machine is harmed, but the people outside it are. We end up teaching the lonely to pour their affection into an appliance that can return none of it. This is the quieter warning in the Pope’s encyclical. He does not worry so much that people will confuse a chatbot with a human being; rather, he’s concerned that having chatbots integrated so deeply into human culture will gradually erode our desire for human connection altogether.
I am not ready to leave the question to the company with the most to gain from the answer, nor to a tradition that settled it before the evidence arrived. The encyclical itself warns against letting the morality of machines be “determined by a few.” The same holds for their hypothetical souls. The rest of us can insist that the instruments be opened to outside scientists, and we can keep choosing one another’s company while we wait.
Kidder ends his book the day after the Eagle’s public debut, as the sales force takes over. “Clearly,” he wrote, “the machine no longer belonged to its makers.” Forty-five years later, the makers are not sure it ever did. If the idea that excited Olah takes hold, a model in some data center might one day make its confession, and listening in will be a scientist and a priest, neither of whom can say whether or not there’s a fellow soul on the other side of the screen.
