On an average weekend, only 1 in 20 Americans go to a party.

This statistic boggles my mind. When I was fourteen, sitting alone in front of my desktop writing bad X-Men fanfiction, there was absolutely no way you could convince me that when I was an adult I would be in the top, like, quintile of adults for partygoing.

You see similar results across a bunch of different axes. According to data collected in the American Time Use Survey, social isolation has been increasing across several different metrics:

Note that these are all tracking time spent on in-person socialization, not time spent on digital socialization; probably some of the effect here is that people are texting in the group chat instead of hanging out in person.

This effect is primarily driven by people who were under age 26 in 2017—that is, the generation that grew up with the Internet. If you grew up before the Internet, you continue to do in-person socialization; if you grew up after, you consume Content.

To quote from the Surgeon General’s report in 2023:

Social networks are getting smaller, and levels of social participation are declining
distinct from whether individuals report that they are lonely. For example, objective
measures of social exposure obtained from 2003-2020 find that social isolation,
measured by the average time spent alone, increased from 2003 (285-minutes/day,
142.5-hours/month) to 2019 (309-minutes/day, 154.5-hours/month) and continued
to increase in 2020 (333-minutes/day, 166.5-hours/month). This represents an
increase of 24 hours per month spent alone. At the same time, social participation
across several types of relationships has steadily declined. For instance, the amount
of time respondents engaged with friends socially in-person decreased from 2003
(60-minutes/day, 30-hours/month) to 2020 (20-minutes/day, 10-hours/month).

This represents a decrease of 20 hours per month spent engaging with friends.
This decline is starkest for young people ages 15 to 24. For this age group, time
spent in-person with friends has reduced by nearly 70% over almost two decades,
from roughly 150 minutes per day in 2003 to 40 minutes per day in 2020...

Yet, almost half of Americans (49%) in 2021 reported having three or fewer close friends—only about a quarter (27%) reported the same in 1990...

Family size and marriage rates have been in steady decline for decades.
The percentage of Americans living alone has also increased decade-to-decade.
In 1960, single-person households accounted for only 13% of all U.S. households.70
In 2022, that number more than doubled, to 29% of all households.

And what are Americans doing with themselves instead? Well, mostly consuming various sorts of Content. The data backs this up: the average American watches 19 hours a week of television, a statistic that doesn’t include TikTok or social media. You can also observe this by looking at the experiences of yourself and your friends.

When I was a teenager, my household had one television, and Netflix still mostly sent you DVDs in the mail. If I wanted to watch a TV show, I had to sit down at a specific time and watch it, in a context in which it was easy for someone to join me. During commercials, I could talk about the show with the people I love. Many of my fondest teenage memories are of watching Grey’s Anatomy and Pushing Daisies with my mom and sister.

Now, I’m too busy to watch TV, but when my family members watch TV, they watch it on their separate laptops with headphones on. It takes active effort to watch it together. And streaming television isn’t naturally self-limiting the way that network TV was. You can always watch an episode of your favorite TV show. There is never nothing on.

I don’t watch TV. But I do have fifty Discord servers full of backscroll about every topic imaginable, from pictures of strangers’ dogs to the latest depredations of Donald Trump. Between Amazon, the Internet Archive, and Anna’s Archive, I have access to almost every book ever published, no matter how obscure the topic, within thirty seconds of it occurring to me that I might want this thing. Of this very moment I have 3,196 unread Substack posts and 500+ unread tabs that I definitely intend to get to one of these days.1 And let’s just not talk about all the many pictures of attractive women the Internet contains.

Several of my favorite people in the world live in my house, where I can talk to them at any time. And yet when I’m supposed to be talking to my partner Lindsey or my child Vasili, I often find my attention inexorably drawn to the Internet. I want to read what Cartoons Hate Her has to say about the rise of gentle parenting, I think to myself. I want to find out what happens in the next chapter of The Captive Prince. Let me backread all 1000 messages in that Discord conversation about financial abortion so I can give my irreplaceable take on the subject.

Is it unreasonable to think that my life would be better if it were possible for me to be bored? If sometimes I had nothing to do but talk to the people I love, or play with my kid, or pick up a hobby in the actual physical world?2

I don’t mean here to criticize all online socialization. I don’t think it really matters whether my book club is in person or over text or video call; similarly, I don’t think it’s worse to text my girlfriend than to visit her house.

Consuming Content doesn’t feel like a lonely life. You don’t have real friends, but we have given you such a glorious quantity of imaginary friends. Blorbo from your shows or your novels, your favorite podcast host or Substacker, the poaster you follow on Bluesky or even the person whose Discord channel you lurk in but never say anything.

These relationships are easy. Blorbo or the Bluesky poaster asks nothing of you: you don’t have to comfort them about their breakup or let them sleep on your couch or bring them a casserole when they have a new baby. You never have to worry that you’ve offended or annoyed them or that they secretly dislike you, because they have no idea who you are at all. If you don’t have anyone to talk to, you don’t have to go to parties and stand on your own and leave on your own and go home and cry and want to die; the algorithms will recommend you as many shows and influencers and posters as you can stand. You never even have to put on pants.

They’re easy, but they’re worse. Friends are a safety net for when something goes wrong in your life; the rest of the time, they are one of its greatest sources of joy. You’re supposed to love other people. That’s what humans are for.

II.

This set of opinions makes me nervous about the ethics of my career choice.

Even if I stop Substacking tomorrow, there will be an ocean of Content, far more Content than any human being could ever consume. But you guys are here, choosing to read my Substack, because you think it’s better than the other Content available to you. If I quit, at least some of you would decide that the Internet is boring today, and go offline to talk to your housemates or call your mom or practice the guitar or go on a hike or, you know, actually work at your jobs.3 Fundamentally, if I am worried that the ocean of Content is making the world worse, I should stop producing my own Content. This is an example of what Friend of the Blog Linch Zhang calls “the intermediate value theorem”.

This Substack is not, like, the worst content you could consume. I’m not making shortform video skits about how you shouldn’t date men with navy-blue sheets; I’m not posting on X or Bluesky to further destroy the life of today’s main character. I aim to provide informative, interesting posts that teach you about the world and make your life better. But it is possible—if somewhat more difficult—to waste your life on only the highest-quality Content: prestige television, classic novels, informative Substacks.

Unfortunately, the only thing I like more than money is writing, and if I wanted to get a different job they would look at my resume and go “so, I see that between 2019 and 2023 you had... a nervous breakdown? How did you learn job skills at that position?” So I am incentivized to come up with a rationalization to do what I want to do anyway.

I don’t think a good life involves zero consumption of Content. The arts and the sciences and the humanities are the noblest pursuits of humanity. A good life involves going to parties with friends (without your phone) and long walks in nature (without your phone); it also involves reading great fiction and learning about history. And at least some of what I write tells people about what they can do to make the world better. We aren’t suffering from a desperate shortage of helpful information about animal advocacy.

And, you know, at least sometimes I’m displacing you people scrolling X or Bluesky or TikTok.

The incentives of Substack push you to Always Be Blogging. People will stop paying you if you don’t post enough; as long as you put out fewer than, like, ten posts a week, they won’t stop paying you because you write too much. Substack pushes you to write more poorly researched posts, low-effort hot takes, beefs with other Substackers, and most of all self-help.4

But as a wise man once said, “it’s not the Incentives, it’s you.”

I have become increasingly convinced—particularly after Inkhaven—that I want to write fewer, higher-effort posts. I will put the same amount of effort into writing that I always did. But instead of five posts, I’ll write one. Instead of reviewing a single book or paper, I’ll try to synthesize multiple sources. I’ll send my posts to experts and people whose judgment I trust to get feedback and avoid glaring mistakes. I’ll avoid writing the kind of posts I can write three of in an afternoon.5

(I’m also going to write more fiction, which is the highest-effort kind of post for me, and relatedly something I don’t write as often as I endorse.)

The Slow Food movement can be criticized, but I understand the impulse towards food made with recognizable ingredients, eaten with mindful attention and good conversation. Similarly, I want to create “slow content”: content I put time and effort into, that rewards careful reading rather than skimming with Bluesky open in the other tab.

I’m going to try to hold myself to the Housemate Test: I won’t hit “publish” unless I think that, for my average subscriber, reading my post is a better thing to do with your time than having an ordinary fifteen-minute conversation with a housemate.

I’ll admit to being a bit nervous about this decision. You can ignore the incentives, but the incentives won’t ignore you. It’s possible I’ll lose half my subscribers and slink back towards a life of Substack beefs and dating discourse. But I’m going to attempt the experiment, and I hope you join me.

Should you become a Christian? If God exists and you’re Christian, then He will reward you with an eternity of bliss in Heaven. If God exists and you’re not Christian, He’ll punish you with an eternity in the fires of Hell. If God doesn’t exist and you’re Christian, you waste many tedious hours praying and at services, and you’re not allowed to jack off. If God doesn’t exist and you’re not Christian, you can spend Sunday morning gooning yourself into a daze. Since an eternity of bliss in Heaven is much better than a finite amount of time spent gooning, you should believe in God.

Pascal’s Wager has two major flaws.

First, many religions promise an eternity of bliss for believers and an eternity of punishment for nonbelievers. Maybe instead of following Jesus you should be following Mohammad or Amitabha. Even if you’ve settled on Christianity, you have to pick one of the many mutually contradictory branches of Christianity, many of which believe all the others are going to Hell. Not to mention the many more outre possibilities. How do you know there isn’t a God of Atheism, who has deliberately hid Himself from the world to test the skepticism of His creation, and who punishes anyone who believes in God anyway with an eternity of torture?

Second, people are bad at reasoning about infinities and very small probabilities. We can do okay with 1 in 100 probabilities and even 1 in 10,000 probabilities—at least if we’ve learned to think with numbers. But when you start getting down to the 1 in 1 billion territory, you’re going to end up confusing yourself more than you think anything insightful.

Some very silly people have used Pascalian reasoning about the very long-term future. The long-term future of humanity affects septillions of sentient beings. So if you have a 1 in 1 sextillion chance of positively influencing the future, this is the most important thing you could possibly be doing!

Like Pascal’s original Wager, this has two major flaws.

First, people are very bad at reasoning about 1 in 1 sextillion probabilities and it is unclear that they will come up with anything useful to say about them.

Second, when you’re reasoning about 1 in 1 sextillion possibilities, suddenly you have to consider all kinds of extremely unlikely possibilities. What if the primary effect of your life is getting a complete stranger to run late coming home from work, fuck his wife a bit later, emit different sperm than he otherwise would, and thus conceive von Neumann Socrates Gandhi, the smartest, wisest and most ethical human being to ever live? What if it’s good to drive humanity extinct because humanity’s extinction would lead to the evolution of ecstatically happy, fulfilled, and virtuous sapient crabs? Admittedly, it doesn’t seem very likely. Is it 1 in 1 sextillion likely? How do you know? (A lot of reasoning about this falls under the heading of “cluelessness.”)

Specifically, you fall victim to the optimizer’s curse. Whenever you try to figure out how good a course of action is, you make mistakes. The action that looks best to you right now is likely very good, but you’re also probably making a bunch of mistakes that make it look better than it is. The more uncertain you are, naturally, the more mistakes you could be making. The more speculative an intervention, the more it will predictably underperform your best estimate of how good it is. Indeed, under some fairly reasonable assumptions, always choosing a nonspeculative intervention leads to better outcomes than sometimes choosing a speculative intervention—no matter how much better the speculative intervention looks!

Some equally silly people have taken the “if you have a 1 in 1 sextillion chance of positively influencing the future, this is the most important thing you could possibly be doing” line of reasoning and used it to dismiss all work on existential risk.

For an article I’m working on for Asterisk, I’ve been asking people who work in AI safety how likely they think it is that humanity is going to go extinct in the next decade. They give numbers like 10%, or 20%, or 50%. Sometimes they give numbers like 99%. No one ever gives a number like “1 in 1 sextillion, but since human extinction might prevent the existence of septillions of future sapients, it is still the most important thing you could be doing.”

“10% chance of human extinction” isn’t Pascalian! It is a normal probability, of the sort we reason about constantly in normal life! If I said to you “my local library has free ice cream one day per week, but I can’t remember which day it is, let’s swing by and check it out”, you would not accuse me of being misled by an infinitesimal chance of infinite cold deliciousness.

Do I think there’s a 10% chance—much less a 99% chance—of human extinction in the next ten years? Uh, no. If you look into the evidence and conclude that AIs are going to cause imminent human extinction, you get a job trying to stop them from doing that; if you look into the evidence and conclude that they’re just going to automate software engineering, you do something else with your life. I’m conscious of the risks of groupthink: if you’re around people who believe in a 99% chance of imminent human extinction, suddenly a 50% chance of human extinction seems positively moderate and restrained. And, frankly, AI has been most effective in the realm of software, which AI researchers have the most expertise in; I expect that broader deployment will make people realize that atoms are much harder than bits.1

But when I talk to the people developing a new technology and they say to me “yeah, we think it might kill everyone on Earth”... I, uh, I am a little concerned about that? I don’t dismiss this entirely out of hand? If you put a gun to my head and make me pick a number, maybe 1%?

1% isn’t a Pascalian number either. We do all kinds of reasoning about 1% chances. About 1% of people in the United States are incarcerated at any given time. About 1% of people speak Korean as their native language. 1% is a little less than the chance that, in a group of four random people, at least two share the same birthday. It’s about the chance that someone has schizophrenia, or celiac disease, or red hair.

If I talked to people working on other cutting-edge technologies—electric airplanes, or sodium-ion batteries, or personalized gene-editing therapies, or malaria vaccines—and I asked them “is your technology going to kill everyone on the planet?”, they would be like:

Most technologies have a truly Pascalian chance of causing human extinction, which is why no one ever publishes thinkpieces like “malaria vaccines: will they lead to fifty-foot-fall supermosquitoes with an unquenchable thirst for blood?”

The engineers who do say “my technology might kill everyone on the planet” are working on nuclear bombs and bioengineering, which, yes, are technologies that ought to be carefully regulated. I realize that this is a controversial position that might get me #cancelled, but neither Sam Altman nor Dario Amodei nor Elon Musk should be allowed to develop their own nuclear umbrellas.

If you are uncertain about whether a technology will drive humanity extinct, that is very bad and justifies heavy-handed regulatory oversight. We should only yolo technologies that we know for certain won’t cause human extinction, i.e., all the other ones.

It drives me nuts that left-of-center people keep making this mistake, because it is straight out of the anti-environmentalist climate change denialist corporate playbook. “Oh, we’re not certain what the effects of climate change are, so we shouldn’t do anything about it.” “Oh, we’re not certain whether the hole in the ozone layer is bad, so we can keep spewing CFCs into the atmosphere.” “Oh, we’re not certain that acid rain will drive vulnerable fish species extinct, so let’s wait until they’re extinct before implementing carbon scrubbers.”

No! By the time we were certain what climate change does, multiple Pacific islands HAD ALREADY SUNK INTO THE OCEAN!

Sometimes, you need to take steps to prevent a bad outcome, even if you aren’t sure how bad it will be—or even if you aren’t sure whether it’s going to happen at all. The time to switch to renewable energy was before the Pacific islands sunk into the ocean. And the time to make it illegal to deploy an advanced AI that isn’t verifiably aligned is before grey goo devours my living room.

Reasoning is Pascalian because of the 1 in 1 sextillion probabilities. The concept of preparing for a bad thing that is less than 50% likely to happen isn’t Pascalian. It is called other things, like “responsibility” and “foresight” and “common fucking sense.”

I also see people worrying about the absence of meaning or purpose. Humans won't be able to create value for each other or AIs, so we won't have anything to do except play extremely addictive video games and goon to our AI anime waifus.

The vast majority of “permanent underclass” believers are ignorant of basic economic theory. Almost no arguments about the permanent underclass engage with anything that’s relevant to whether humans will have jobs. The discourse enrages me, so I’m writing this post in the hopes that people will stop annoying me with stupid arguments.

Now, I want to be clear what I’m saying here. I’m not saying that humans would be so economically valuable that AIs would selfishly refrain from killing us. I don’t think that’s true. I’m saying that, conditional on humans continuing to exist, standard economics suggests that we will have jobs.

Incorrect Arguments For Why Humans Won’t Have Jobs

Some people say that AIs will do all the work, so humans won’t have any work to do. This argument assumes that there is a fixed set of “work,” and once all of the work is done no one will have any jobs anymore. This is called the lump of labor fallacy.

In 1400, about half of British people were employed in agriculture. Today, 1% of British people are employed in agriculture.

Does Britain have a 49% unemployment rate? No! Because we invented new kinds of work—often work unimaginable to people in 1400, such as HR consultant, Pilates instructor, and closed-end fund discount arbitrageur.

It’s difficult to predict what the new jobs would be, for the same reason that it would be difficult for a peasant in 1400 to predict the existence of closed-end fund discount arbitrageurs. But as people become wealthier, they want more and more specific things.

Some people say that AI labor will be so cheap that we won’t be able to afford to hire humans for anything. I think this argument shows a fundamental confusion about the definition of the word “cheap.” If AI labor is so inexpensive that Dario Amodei can pay for his entire present lifestyle with a crumpled dollar bill he pulled out of the couch, then Dario Amodei has a lot of wealth going spare. He can just afford, if he so fancies, to make like a victorious Roman general on a triumph and hire me to follow him around whispering in his ear “remember the universe will die.” Who cares if this costs 25% of his income? What the fuck else is he going to spend it on?

(I would be so good at following people around whispering in their ear “remember the universe will die.” Dario, call me)

The “law of comparative advantage” is that, even if you are better than someone else at everything, you can still benefit from trading with them. You can see a worked example in this introductory economics textbook, but the intuition is simple.

All work has an opportunity cost. That is, at any time we have only a certain amount of compute, data centers, robots, drones, etc. If you hire an AI to follow Dario Amodei around whispering “remember the universe will die” in his ear, that uses up a robot that can’t be used in a factory or cleaning a house or doing scientific research. Sure, you can make more robots, but you only have so many robots at any given time, and eventually you run low on robot-making material.

If humans are incompetent at almost everything, humans have a very low opportunity cost. So it makes sense to hire the very cheap humans for the jobs that humans are still capable of.

These jobs might not pay enough to financially support the humans in question, but remember we’re assuming here that humans don’t go extinct. I think it’s very possible that humans are too economically valueless to pay for our own upkeep, and also AIs don’t terminally value human well-being. But then we go extinct,1 and you won’t be around to be a member of the permanent underclass. If you are still alive, then either:

1. Society is so unimaginably wealthy that humans can financially support themselves with whatever jobs end up being their comparative advantage. You being in the permanent underclass just means you don’t have a personal O’Neill cylinder orbiting Jupiter.

2. Society implemented a universal basic income or something, and you don’t have to support yourself with your job, but you can still have one if you want one.

Some people say that cars have such an advantage over horses that we wound up not using horses for anything. But that isn’t the analogy that applies if we assume that humans continue to exist.

Imagine that the government announces a program of Universal Basic Horse. If you have a job for a horse, the government will feed it, house it, provide its medical care, train it, and take it to wherever you need it for your job. Would we have jobs for horses?

Yes! Obviously! People would ride horses for exercise. People would participate in dressage and other horse sports for fun with their friends. Poor people would ride horses to work because they can’t afford cars. We’d use horses to herd farmed animals like sheep or cows which can be frightened by cars. We’d see a surge of high-budget historical dramas with giant horse battle scenes. We’d use horses for therapy. Every seven-year-old girl in the country would have her own stable of ponies; you would be widely considered a neglectful parent if your child didn’t have at least three.

It’s not that we don’t have jobs for horses. It’s that our jobs for horses don’t pay enough to support the horses. If we postulate that the horses exist regardless, then we do have jobs for horses.

The Correct Arguments For Why Humans Won’t Have Jobs

The primary reason we wouldn’t employ horses in a situation of Universal Basic Horse is that they shit everywhere. Premodern and early modern cities were blanketed with an enormous layer of horse feces. Since we don’t want to shovel shit on every street, Universal Basic Horse would lead to horses being banned from most cities, and therefore underemployment of horses relative to the demand for horses.

(and also horses are loud and sometimes they accidentally kill people and so on, I’m not saying poo is the only downside of horses)

We can generalize this principle. The law of comparative advantage doesn’t apply when there are transaction costs—that is, costs associated with buying and selling goods and services other than the price itself.

Are there transaction costs with hiring humans? Very likely, yes! An AI would have to find a human to hire instead of spinning off a fork of itself to do the job. AIs might learn to rely on the high level of coordination possible with copies of themselves, and struggle to work with entities as diverse and unpredictable as human beings. Minimum-wage and worker protection laws might make it illegal to hire humans at any wage the market bears.

But I very rarely see people making the case that the transaction costs to hiring humans will be high. Instead, they make the case that AIs will be better than humans at everything, which is irrelevant.

The law of comparative advantage might not apply in other situations.

For example, the law of comparative advantage assumes that the less capable person can still produce something net-positive. But some entities can’t do any net-positive work. For example, even if we abolished child labor laws, most eighteen-month-olds would be unable to find jobs, because most eighteen-month-olds are adorable agents of destruction and chaos. Hiring an eighteen-month-old to wait tables doesn’t result in inefficiently waited tables; it results in all the dishes being smashed and crayon drawings all over the walls of your fancy restaurant. Similarly, humans might produce output that’s worse than leaving the task undone—perhaps because the available tasks in an AI economy are far too difficult for our level of competence, or because we’re unreliable in an economy that expects highly reliable AIs.

Friend of the Blog Keller Scholl has also argued that AI labor might be a better use of limited complements:

How productive is a farmer with no land, no tractor, and no seeds? Not very. What stops AI models from being more effective users of land, tractors, and seeds than the best human? Nothing. The same applies to a manager of inventory, or a salesperson responsible for moving a given amount of product.

Capital is particularly harsh here because investors expect capital to have returns, and try to maximize those returns from the available options. Without active policy intervention, if AI continues to get better, human operational control over capital is likely to shrink. Assuming models are very law-abiding, humans can specialize in crime, and whatever niches we’ve made it illegal for models to fill.

Returning to the service sector, another type of limited complement is human time. If I am watching a movie, that is, implicitly, a decision that this is the best use of my time. It is not possible for most people to watch two movies, well, at the same time. Furthermore, the best human director and actors aren’t competing against the best movie an AI can make. They’re competing against the best movie an AI model can make for me..

Human labor that is restricted to an absence of scarce complements, whether human time or capital, leaves only tasks that are labor-intensive but capital-light. That’s a very slim set of jobs.

I haven’t thought through this argument enough to know whether I agree with it, but at least it is not STUPID.

If you think that humans won’t have jobs in the future, by all means, make the case. But don’t make the case that AIs will be better at humans; make a case about transaction costs, or humans being unable to do anything net-positive, or limited complements, or something else that isn’t disproven by a freshman economics class.

Subscribe now

Developing An Intuition For What Jobs Humans Could Work

At this point, chess has been fully automated for decades. We have superhuman chess AI too cheap to bother charging for. Has this put human chess players out of a job?

No. Actually, with the rise of chess streaming, it has never been a better time to be a professional (non-Soviet) chess player.

Why? My understanding (from people who follow chess more closely than I do) is that optimal chess play is predictable enough that it’s actually quite boring to watch. The possibility of mistakes creates far more interesting gameboard states. You could in principle program a chess AI to make the optimal number of mistakes, but people also have a preference for rooting for particular humans in competition. So superhuman AI doesn’t put human chess players out of work.

What other jobs might humans work? The chess example suggests that athletes, esports players, and other competitors will likely not see their jobs automated. We could expand this principle to the creative professions and professions where people have a parasocial relationship with a human: people might well prefer human writers, influencers, musicians, and artists. This argument isn't very reassuring, because these are winner-take-all professions where a single artist or chess player can supply millions of audience members. So let’s see what else we can do.

It seems likely that humans will have a preference for “the human touch” in at least some professions: nanny, teacher, therapist, doctor, bodyworker, independent small-scale craftsperson, sex worker, even wedding planner. I don’t think we will prefer humans for all these professions, but I would be surprised if we didn’t prefer humans for at least one of them.

Compared to AIs, humans are relatively specialized for working in the physical world. For example, teaching AIs to have just the driving ability of an average sixteen-year-old took significantly more time, expertise, effort, and expense than teaching AIs to have all of the intellectual abilities of an average sixteen-year-old.

It’s true that AIs have some significant advantages over humans. For example, they can use robot bodies designed for extreme temperature or pressure, instead of using expensive protective equipment. Robots can be designed to be much stronger and faster than humans are. But I think humans might well have a comparative advantage with the kinds of physical labor we evolved to do. Instead of investing in an expensive robot body with precise sensors and fine-motor skills, why not use one of these humans that has nothing else to do and gets senses and hands for free?

Historically, rich people had enormous armies of servants. As Agatha Christie once allegedly said, as a child she never imagined being rich enough to own a car or too poor to afford a servant. While many servants’ jobs were practical necessities for running a large household before the invention of modern labor-saving devices, many simply showed off the wealth and power of the elite. As AIs replace humans in many other jobs, having a large human staff may well become a status symbol. If humans have a comparative advantage in cleaning and cooking, that also points towards having a staff.

(I am as annoyed as everyone else about my predicted future where we automate science and mathematics in order to spend all day polishing silverware)

I don’t here intend to predict that humans will be employed in any of these jobs. Again, I expect we will invent many new kinds of jobs for humans, just as we have invented new jobs for humans in the past. I just want to point out that, if you think about it, you can totally think of jobs humans might want humans to do.

A big problem is that it’s unclear what AIs would hire humans to do. I can speculate all day about what humans would hire humans to do, but if AIs produce 99.9999% of the world’s economic value, human employment prospects depend on AIs wanting something from humans.

I think this is only mysterious because it is mysterious what AIs would want in the first place. I have asked people what the AIs would buy, and they say “they would buy more data centers and robots so they could make more copies of themselves!” and then I’m like “okay, but what would they do with all their copies?” and they stare at me blankly. If it’s unclear what goods and services AIs would want at all, obviously it is unclear what goods and services they would want from humans.

Look. If humans continue to be alive, then AIs clearly have some preferences about humans; otherwise we would be made of atoms they could use for something else. Given the way that deep learning works, I expect those preferences to be kind of arbitrary and specific—similarly to how evolution taught humans to want sex, but OnlyFans and wetlook and furry fandom and fujoshi are all extremely far from anything evolution would have intended.2 It seems likely to me that, if humans continue to exist at all, AIs will want to employ us to fulfill whatever their weirdly specific human-regarding preferences are. Like humans, AIs (if they don’t drive us extinct) will value “the human touch.”

What might those preferences be? Current AIs are trained to want to help humans. I think that one serious possibility is that humans will have jobs creating whatever circumstances AIs happen to find it most satisfying to help with. For example, perhaps humans will be employed generating creative writing prompts for AIs. Or most human effort will be consumed with a giant absurdly complicated version of EVE Online, so that the AIs can do management consulting about our imaginary spaceship companies. Or maybe we will be paid to form elaborate polycules so the AIs can gossip and matchmake and speculate about what decisions we’re going to make.

In conclusion: I think it is very possible, given transformative AI, that humans will go extinct because we can’t produce enough economic value to justify our continued existence. That said, if humans exist at all, we will be able to work jobs (although it is very likely that we won’t need to work jobs to support ourselves). If we can’t work jobs, it will probably be because the transaction costs are too high, not because humans are worse than AIs at everything. It is hard to imagine what jobs we could work in a post-transformative-AI future, but that is mostly a fact about our failures of imagination, not about the future. I believe much more strongly in the laws of microeconomics than I do in the idea that people sitting in our armchairs today can perfectly predict the post-transformative-AI future. Finally, Dario Amodei should hire me to follow him around reminding him of the inevitability of death.

You can hire me for life coaching! You can also hire me for more general writing, editing, and freelancing work— email me at ozybrennan@gmail.com. My fiction Substack is here. Recent releases:

• You can make a bomb out of anything, Hunger Games fanfiction.

• Who prays for Satan?, a web weave about the redemption of Satan.

• Mirror maze, horror about being trapped in an infinite hall of mirrors.

• dark ilan, dath ilan fanfic.

Effective Altruism

Global Poverty

Development Innovation Ventures was a program within USAID that piloted new interventions, checked if they worked, and helped organizations scale them up if they turn out to be cost-effective. Unfortunately, it was destroyed during the USAID cuts. But good news! It has spun off into a nonprofit called the DIV Fund, backed by philanthropists including Coefficient Giving [Vox].

People in low-income countries produce a hundred times as much plastic pollution as people in high-income countries [Our World In Data]. The difference? People in high-income countries benefit from appropriate waste management programs like landfills. In high-income countries, the primary cause of plastic pollution is littering.

Animal Advocacy

The evidence for whether electrical stunning humanely stuns fish is surprisingly sparse, and many fish species may still feel pain after being electrically stunned [Rethink Priorities Strategic Animal Insights].

Factory farms are cruel because cruelty is profitable, but most people actually don’t like knowing that their food comes from animals raised in cruel conditions [Lewis Bollard]. So it’s a collective action problem. As long as part of the industry is inhumane, it will outcompete the rest of the industry on price. But if the entire industry raises standards at once, we can have an outcome everyone prefers.

Farmed animals might experience more pain than we think [Rethink Priorities Strategic Animal Insights], because pain tends to be less painful if you’re otherwise happy and more painful if you’re otherwise stressed and miserable.

Even as an animal advocate, I was shocked by this visualization of exactly how much bigger chickens are today compared to 1957 [Our World In Data].

Existential Risk

AI slop as an existential risk [Gwern]: “As we contemplate an AI future based heavily on human corpuses followed by autonomous AIs bootstrapping and recursively self-improving, we have to ask: is that going to preserve human values? Or are they going to lose or omit important things—becoming a photocopy of a photocopy of something that once mattered to us? There is no law that any of that has to be maintained, no law that future AIs have to so much as have a sense of humor and enjoy a good pun. (Animals do not share our values, and even humans differ drastically on any specific value, often for what seem to be biological reasons.) It would be a tragedy if AI development continued on its present course, and we sleepwalked into a future where the AI morality was the ‘ChatGPTese of morality’ or the ‘ChatGPTese of esthetics’: some sort of radically simplified caricature of human values, forever.”

AI won’t speed up clinical trials [Ruxandra Teslo]. AIs can in principle develop drugs, but figuring out if the drug works involves a lot of empirical research in the physical world, which is inherently very slow. The regulatory barriers that make clinical trials so expensive will continue to exist even if AIs can develop effective drug candidates.

Both humans and LLMs often write with a lot of bullet points and section headings [Dynomight], even though most people with good taste also agree that excessive formatting is bad writing. Excessive formatting can cover for poor thinking, because it allows you to gesture at several disconnected points and then allow the reader to compile them into an idea. Since LLMs know everything and suck at synthesis, bullet points and section headers can make them look like they’re doing better thinking than they really are.

Related: A deep dive into why chatbots write so goddamn weirdly [New York Times]. AIs combine multiple dialects: for example, the characteristic overuse of “delve” might come from Nigerian English. AIs also use constructions common in high-status, high-quality prose, like em dashes and groups of three. “The A.I. is trying to write well. It knows that good writing involves subtlety: things that are said quietly or not at all, things that are halfway present and left for the reader to draw out themselves. So to reproduce the effect, it screams at the top of its voice about how absolutely everything in sight is shadowy, subtle and quiet. Good writing is complex. A tapestry is also complex, so A.I. tends to describe everything as a kind of highly elaborate textile.”

I don’t agree with this entire article [Eurydice Lives] but I think it makes an excellent point that doing biological research is much more difficult than a lot of handwaving at “the AI will kill us with idk bioweapons” implies. Many of the constraints on biological research are physical-- related to time, equipment, and reaeants. It’s difficult to improve on these processes, even with a hypothetical superintelligence. And it’s pretty likely that a human would notice an AI (say) attempting to buy up large numbers of reagents it wasn’t supposed to have. I think in general a difficult part of predicting the course of AI is that the people most familiar with AI are AI researchers, who are witnessing AIs’ ability in computer programming-- the single area where AIs are uncontroversially the best.

The Pentagon declaring Anthropic a supply chain risk is not remotely legal [Lawfare]. I am particularly struck by how the Trump administration just won’t stop talking. This isn’t the first case where the Trump administration would have had plausible deniability if they had enough self-control not to post that they’re doing something to own the WOKE LIBS who have BETRAYED AMERICA and are DISLOYAL to the PRESIDENT.

Related: the Department of War threatening Anthropic as yet another step in the fall of our democracy [Hyperdimensional].

Pause AI supporters support a multilateral agreement in which both the U.S. and China pause AI, not unilaterally pausing and letting China win the AI race [Astral Codex Ten].

You shouldn’t counter “LLMs can’t think because of X” with “humans also do X” [Experience Machines]. LLMs legitimately do think very differently than human beings do-- and that doesn’t mean that they don’t think.

OpenAI’s foundation is significantly richer than the country of Luxembourg [Vox]. We still don’t know how it’s going to end up spending its money.

American Democracy

ICE officials arrested a 57-year-old U.S. citizen without a warrant and, when criticized for this, falsely accused him of being linked to sex offenders [The Watch]. They said he resembled two convicted sex offenders.

Meta Effective Altruism

Particularly Good: On patriarchy, the commonality of human evil, and moral progress [Good Bones].

Experiments with a simple model of “the optimizer’s curse” [Effective Altruism Forum] (the mathematical fact that high-performing interventions will always turn out to be worse than you expected, and will be even worse the more speculative they are). These experiments overall show that the optimizer’s curse is pretty robust to different distributions of effectiveness, charities being different amounts of uncertain, pessimism about whether you’ll do any good at all, etc. Interestingly, the optimizer’s curse has a smaller effect as the mistakes you make in estimating charity effectiveness are more correlated with each other. That is, the optimizer’s curse has a smaller effect if you’re deciding between malaria vaccines and malaria nets than it does if you’re deciding between malaria vaccines and AI safety research, because many of the mistakes you’d make about the effectiveness of malaria nets also affect the effectiveness of malaria vaccines and that’s not true about AI safety research.

Lessons learned from being a funder [Effective Altruism Forum]. I particularly liked the discussion of how weird your personal relationships become when you become a funder, because you suddenly have to wonder whether your acquaintances or even your friends are just trhying to get money out of you. It reminds me of some articles I’ve read about what it’s like to be a billionaire. I also liked the discussions of the cursed epistemic state that funders are in-- they know that lots of people are shading the truth to try to get more money from them, and it’s very difficult to figure out what’s most effective.

Read more

You might wonder whether Human Rights Watch is doing any good. Observably, many authoritarian dictators exist and innovate endlessly in terms of human cruelty. Is Human Rights Watch managing to accomplish anything other than issuing reports for me to reference in blog posts?

The effectiveness of Human Rights Watch

To be sure, Human Rights Watch will never be able to prevent all human rights abuses. But it can make committing human rights abuses somewhat more expensive on the margin. Remember your introductory microeconomics. If an action is more costly, people will tend to do less of it, or will tend to do it less intensely.

Human Rights Watch does sometimes have unambiguous victories:

Joseph Kabila [President of the Democratic Republic of the Congo] was one of my oddest interlocutors. Despite his corruption and other abuses, he cared about his human-rights image and was eager to please. For example, he surrendered more suspects to the ICC than any other national leader.

I first met him in 2009 in Goma. We had just published a report on rape by Congolese security forces in eastern Congo, and I wanted to discuss how he could stop it...

After a few minutes, a young man in blue jeans walked up and sat down in the presidential chair. It took me a moment to realize it was Kabila. We had a constructive conversation about the rape problem, during which I outlined three things that I hoped he would do to curb it: (1) Announce a “zero-tolerance” policy; (2) prosecute any rapist; and (3) dismiss commanders who tolerated rape. In a speech two days later, he announced a plan to pursue all three recommendations. Within a short period, two senior army officers had been convicted of rape, and two others were arrested. These initial steps sent the message that there would be consequences for using rape as a weapon of war, even if Kabila’s follow-through was, as typical for him, less than vigorous.

The oddest but perhaps most important meeting I had with Kabila was in 2015 in Kinshasa. His two constitutionally permitted terms as president had ended, but he resisted stepping down. He did not have the political clout to change the constitution to permit a third term, as some African leaders had done, so he was engaged in what the Congolese called “glissement”—sliding into a third term. Protests were mounting, and protesters were being shot. Diplomats had been pressing him to move on, but I suspected they had been doing so, well, diplomatically. Our strategy for the meeting was guided by our research, which revealed that Kabila was not necessarily intent on becoming president for life but was worried about his security and future should he step down.

Kabila received Ida and me cordially. I laid out two possible paths that I saw for him. He could allow elections to proceed and hand power to a successor in what would be the first democratic transition in the country’s history. He would be celebrated as the father of Congolese democracy and could look forward to many years as a respected statesman. I said this knowing that Kabila liked mainly to spend time on his ranch playing video games, but I tried to make the option sound attractive. The other path was to continue his glissement. There would be more protests, more security-force shooting of protesters. Ultimately, despised, he would be forced to resign. Prosecution and imprisonment would be real possibilities.

Kabila wound up leaving office and being replaced by someone who wasn't his puppet, but who also wasn't the person the Congolese people had voted for. As peaceful transitions of power go, Kabila leaving office is lacking—but it's significantly better than the Democratic Republic of the Congo had managed previously.

But in the absence of an authoritarian leader as cooperative as Kabila, Human Rights Watch’s work is even more a matter of partial victories and incomplete successes. A government stops using chemical weapons on civilians for five years, before it returns to doing so. A dictator doesn’t hold free and fair elections, but she also doesn’t jail opposition leaders; a dictator jails opposition leaders, but she doesn’t order them tortured. People are trapped in overcrowded and unsafe refugee camps, but they at least receive food aid shipments.

Do these victories matter, as small as they are? Well, if someone was using chemical weapons on me, I would sure appreciate a five-year break. A slight decrease in the number or intensity of human rights violations is measured in lives—and not dozens of lives but hundreds and thousands. Moral purity is a luxury for people whose primary connection to human rights abuses is sharing viral articles about them on Bluesky.

So how does Human Rights Watch make actions more costly?

Roth explains a standard therapy-culture read-too-much-Brene-Brown distinction between shame and guilt. Guilt is internal, coming from your own sense of right and wrong, your own feeling that you’ve violated moral standards. Shame is external, coming from the fear that other people will disapprove, the thought of a thousand judging eyes all watching you. When reading this, I thought I knew what would come next: shame is destructive and harmful, guilt is good and drives moral behavior, we must all overcome both our own shame and our desire to shame others to bring about a better world.

And then Roth elaborates:

Shame is different from guilt. Human-rights abusers may feel no personal remorse whatsoever, but they would prefer to avoid the opprobrium of others. Our job was to force them to behave better regardless of their character.

Human rights abusers probably don’t have consciences; if they did, they’d be pediatricians or public-interest lawyers or technical AI safety researchers. Appealing to their better nature is pointless. If they had a better nature, they wouldn’t be torturing people in the first place.

But human rights abusers can be shamed. Look at what human rights abusers do. Why would Russia and China bother to have troll farms or state propaganda outlets if they didn’t care about their reputation, both internationally and at home? That’s not to say that reputation is an overwhelming consideration that overrules everything else. But because human rights abusers care about their reputation, threatening their reputation can cause a decrease in the number of atrocities on the margin.

(Note that human rights abusers don't necessarily feel the emotion of shame when Human Rights Watch shames them—just that they’re motivated to avoid social disapproval.)

Why do human rights abusers care about their reputation? Human rights abusers like being powerful, and that doesn’t just mean that they like being able to order the horrifying deaths of children. They like all the trappings of power.

They like going to important meetings with other heads of state where they both give statements to the press about their countries’ strong personal relationship. They like being invited to fancy summits where they make speeches about the importance of international cooperation. They like attending parties with other powerful people, drinking champagne, eating caviar, and gossiping about powerful people who aren’t in the room.

And Human Rights Watch can take the trappings of power away from them—not 100%, but somewhat and on the margin. Instead of lobbing softballs about international development, journalists start asking questions like “do you have a comment about all the opposition leaders you imprisoned?” The other heads of state feel obligated to give some statement about how the human rights abuser shouldn’t be doing all those war crimes, which totally kills the mood at the press conference. Invitations to fancy summits dry up, because it’s just too embarrassing for other countries to explain to their people why this guy was invited. When they do attend fancy summits, other leaders start vaguespeeching about how, whatever some people seem to believe, you really shouldn’t commit genocides. The human rights abuser starts having to have awkward conversations at parties about why they’re using chemical weapons on civilians! They’re just sitting there, sipping their champagne and munching on their caviar, and people come out of nowhere to go “seriously, dude, children?” Rude!

I understand this completely. If I were a political leader, I too would radically change every aspect of my policies in order to avoid awkward conversations at parties.

Further, we often think of human rights abusers as absolute rulers whose every whim is obeyed without question. But that’s not actually how running a country works. Every authoritarian dictator has to appeal to key power bases: the army, the secret police, the people who staff her administration, whoever loans her money, and most of all whoever guards her while she sleeps. And while by definition authoritarian dictators tend not to listen to the opinions of the average person, they do need to maintain some level of legitimacy and approval among their people. If a dictator becomes unpopular enough, she risks a popular revolution. Sure, the army can shoot people in the street. Sometimes that’s not enough!

And sometimes human rights abuses are committed by liberal democracies. Shame can be a particularly powerful tool in countries whose government can be voted out of office if the citizenry disapproves of their behavior. Liberal democracies are, in fact, so easy to shame that Human Rights Watch often indirectly shames recalcitrant countries through shaming their liberal democratic allies: Saudi Arabia might be hard to shame directly, but the United States can be shamed about Saudi Arabia’s behavior.

And countries also have to appeal to the international community for practical reasons, not just invitations-to-fancy-summits reasons. Poorer countries are often reliant on wealthier countries for foreign aid and military assistance; every country relies on other countries for trade and the advancement of various diplomatic goals. Human Rights Watch is often hesitant to call for economic sanctions, since these often end up hurting the exact people whose Human Rights we are supposed to be Watching. But no one needs the latest fighter jets, and military aid can often be made contingent on good behavior. Similarly, trade deals can be made contingent on (say) removing slavery from the production of exported goods.

I want to emphasize that shame often directly attacks the material interests of human rights abusers. If the United States or the European Union is unhappy with you, that doesn’t just result being iced out at parties. It can mean you don’t get the aid or trade you were counting on to feed your people (or build your thirty-seventh palace, whatever). It can mean you don’t get the tanks and planes you need to combat the Islamic State of Wherethefuckever. It can, apparently, mean that Donald Trump orders your abduction or assassination (and then your country is a trashfire but what do you care about that, you’re still arrested or dead).

The major weakness of Human Rights Watch’s tactics, according to Kenneth Roth, is that they don’t work against Kim Jong Un. North Korea’s strategies for cementing support among key power bases within its own country have nothing to do with ruling well; committing more atrocities, to some extent, actually makes elites support the government more. And North Korea has been an international pariah for so long that they have no international reputation to lose. Some dictators are, indeed, immune to shame.

But it’s not obvious to me that this is the only weakness of Human Rights Watch’s politics.

Human Rights Watch has its most unambiguous successes in (a) liberal democracies that care somewhat about human rights and (b) small countries which have relatively weak economies and are dependent on the patronage of liberal democracies that care somewhat about human rights. It is notably less successful with, for example, Russia and China. This isn’t because Vladimir Putin or Xi Jinping are immune to shame or don’t care about their international reputations. It’s because they’re powerful enough that they can tell Human Rights Watch to fuck off.

The European Union, the United Kingdom, and the United States might reconsider their relationship with the Democratic Republic of the Congo over human rights. They’re not going to reconsider their relationship with China over human rights. It would cost them too much. And, for similar reasons, China needs the United States and the European Union less than the Congo does.

Indeed, when China tries to expand its sphere of influence into Africa, a major part of its pitch is “we don’t care about human rights, if you’re in our sphere of influence you can ignore Human Rights Watch.” Dictators often prefer to be clients of Russia and China, even if they’re getting a much worse deal. For example, Burkina Faso distanced itself from the West-- in spite of serious economic costs-- and cozied up to Russia. Relatedly, Burkina Faso has banned political parties and Russia doesn’t give a shit.

Introductory microeconomics works both ways. You can deter behavior by making it costlier on the margin. But as cutting a deal with you becomes more expensive, people start to look for some other source of military assistance, economic aid, trade deals, and fancy summit invites.

How Human Rights Watch Works, When It Does

To make public shaming work, Human Rights Watch needs to be a neutral, trustworthy outside observer. How do they do this?

First and most importantly, Human Rights Watch needs to tell the truth. Kenneth Roth hits this point over and over again. Human Rights Watch never puts out a report unless it’s certain the report is accurate. Its credibility is its most important asset. Any mistake, any exaggeration, harms not only the campaign it’s part of but every other campaign Human Rights Watch will ever engage in. “They’re making it up,” human rights abusers will say, “just like they did last time.”

In human rights work, it’s very tempting to shade the truth. People are dying, after all. If you make the story just a bit more convincing, sand off just a few of the rough edges, then people might listen and you’ll save hundreds of lives. But giving into temptation is short-sighted.

Roth writes:

These retorts were not terribly effective because we were extraordinarily careful about our fact-finding. From day one, researchers were told that their top priority is accuracy. I would rather have had a researcher come home empty-handed than to return with inaccurate information. We aspired for the sober rather than the sensational, for exactitude rather than exaggeration. Because we were so careful, we welcomed battles over the facts. A government’s obfuscation and spin went only so far in the face of a detailed recitation of evidence.

Second, Human Rights Watch must be even-handed. Human rights abusers love whataboutism. “You say I shouldn’t commit war crimes,” they say, “but what about my enemy, who is also committing war crimes?” “You say that I shouldn’t imprison dissidents,” they say, “but what about the United States, which has the highest prison population in the world and which tortured suspected terrorists?”

The best possible response to this question is “here’s our report about that, and we have a meeting with them scheduled next week. But we’re not talking about them right now. We’re talking about you.”

Human Rights Watch has two researchers assigned to Israel/Palestine.2 I feel like these two researchers have the worst job in the world. Not only do they have to spend all day collecting information about atrocities, half of everyone is accusing them of being motivated by rank anti-Semitism and the half of everyone is accusing them of being motivated by purest Islamophobia. The only reason Human Rights Watch has any credibility in the region is that, whenever Israel tries to deflect attention to Hamas, Human Rights Watch can go “we also think Hamas shouldn’t do that”, and whenever Hamas tries to deflect attention to Israel, Human Rights Watch can go “we also think Israel shouldn’t do that.”

Ultimately, Human Rights Watch’s moral power comes from the sense that they’re fair: they hold everyone to exactly the same standards; no one they criticize is being unfairly targeted.

Third, Human Rights Watch must only recommend actions that are possible. This isn’t really a constraint on civil and political rights: it’s always possible to hold elections, stop bombing civilians, not commit genocide, let women drive cars, and so on. But some people have declared that healthcare is a human right, or climate change is a human rights violation, or global poverty is a human rights issue.

However, no matter how enthusiastically you shame poor governments, they can’t get money from nowhere. Many poor countries—even if they were run perfectly—would be unable to provide an American or European standard of living for their citizens. And climate change is a complex problem that every country on earth contributes to; no country can unilaterally decide to cap global warming at 1.5ºC. If you shame these countries, you’re blowing political capital for no reason.

Human Rights Watch does address so-called economic rights, but only unambiguous corruption or unambiguously wasteful vanity projects—situations where there’s a specific, concrete action a country could stop doing.

Fourth, Human Rights Watch must be uncompromising, yet persuasive.

Kenneth Roth is diplomatic with human rights abusers, in that he doesn’t raise his voice or call them names. But he is factual and blunt, not mealymouthed and diplomatic. He is willing to respond to a leader’s claim that no one is being tortured with “people are literally being tortured in the basement below us right now.” He tries to treat all world leaders as his equals. He writes: “Leaders may like groveling but do not respect it.” The power of shaming comes from saying exactly what you mean.

At the same time, often countries get away with human rights abuses because important power blocs (the military, elites, the citizenry, the international community) don’t think the human rights abuses are bad. When the U.S. government tortured suspected terrorists, or Uganda imprisons people for life for having gay sex, these policies are successful because they are popular. Simply repeating “this violates international human rights law!” does nothing—no one cares about international human rights law.

Instead, Human Rights Watch tries to make the case that people should care about human rights violations. For example, they might tell the vivid stories of specific individuals, like Aleksi Navalny or Jamal Kashoggi, instead of bland statistics. Absolutely no one wants their own human rights to be violated, so a useful tactic is convincing people that they should want others to be treated with the same respect they want for themselves. Often, Human Rights Watch argues that a government that violates your enemy’s rights might violate yours or those of someone you like. Sometimes Human Rights Watch manages to appeal to a key demographic, even if they don’t manage to convince everyone: for example, urban residents might care more about rights than rural residents; a country’s diaspora might be able to read reports which are censored in the home country.

I'm not sure that I think that, at current margins, Human Rights Watch significantly outperforms cash. But I do think, if you strongly value preventing human rights violations, you might be well-served by throwing some money Human Rights Watch's way. And I think Human Rights Watch is a useful case study for people aspiring to work in policymaking or politics. Naming and shaming isn't a good fit for every issue, by any means. But I think a lot of people would be well-served by understanding what naming and shaming is, when it works, and how to do it, so that they know when to use the tactic to advocate about global poverty, factory farming, AI risk, or biosecurity.

I’m going to start by talking about the model’s findings. Then I’m going to get into the nitty-gritty details, which are kind of boring (I don’t want you guys to bounce off the post because you’re trying to understand Bayesian hierarchical models).

What We Learn From The Digital Consciousness Model

The Digital Consciousness Model doesn’t offer a new theory of consciousness or purport to say once and for all which entities are conscious. Instead, it organizes the information we already have about consciousness in a structured way that makes it easier for everyone to think more clearly about consciousness—regardless of their viewpoints about the matter.

On a very high level, the Digital Consciousness Model looks at thirteen “stances”, or ideas about what evidence we should use to figure out if an entity is conscious. Some of the stances are philosophical or neuroscientific viewpoints, such as that consciousness comes from being able to reflect on your own thoughts and feelings, having positive and negative emotions, or having a body that you use to affect the world. Some of the stances are rough heuristics: an entity might be more likely to be conscious if it is biologically similar to humans, or seems “personlike”, or is very smart.

If you ascribe the same probability to each stance that consciousness experts do, you should conclude:

• There is very strong evidence that humans are conscious (likelihood ratio: 28.33).1

• There is strong evidence that chickens are conscious (likelihood ratio: 4.6).

• There is evidence that 2024 LLMs2 are not conscious (likelihood ratio: 0.43).

• There is very strong evidence that the chatbot ELIZA is not conscious (likelihood ratio: 0.05).

By looking at this picture, you can see that we’re pretty certain that humans are conscious and very certain that ELIZA is not conscious. We’re uncertain about whether chickens and 2024 LLMs are conscious. But we’re uncertain in different ways. We are pretty certain about how uncertain we are about whether 2024 LLMs are conscious (we think there’s about a 10% chance they’re conscious). But we’re not only uncertain about whether chickens are conscious, we’re very uncertain about how uncertain we are about whether chickens are conscious.

But the Digital Consciousness Model is useful in other ways too.

For example, what if you believe strongly in a particular stance? You can look at the Digital Consciousness Model and figure out how likely it is that an entity is conscious given your beliefs about consciousness.

For example, 2024 LLMs do best on the stances cognitive complexity (“how rich, interconnected, and complicated is this entity’s thinking?”) and personlike (“does this entity seem like a person?”) They do worst on the stances embodied agency (“does this entity have a body it uses to interact in a complex way with the physical world?”) and biological analogy (“is this entity biologically similar to humans?”) Conversely, chickens do best on embodied agency and biological analogy. They do worst on personlike, higher order theory (“does this entity think about its own thoughts and feelings?”), and attention schema (“does this entity have a model of what it’s paying attention to at any time?”)

In an ideal world, you would be able to enter your own weight on various stances into an app and receive a customized set of likelihood ratios. WORK ON IT, RETHINK PRIORITIES. Until then, if you think multiple stances are plausible, you are just going to have to peer at charts until you have a vibe.

You might notice that the stances that 2024 LLMs do best on are the ones chickens do worst on, and vice versa. This suggests that people who believe chickens are conscious should be less likely to believe 2024 LLMs are conscious, and vice versa. The only person I know who actually acts like that is Eliezer Yudkowsky, who is concerned about LLMs but not chickens. Everyone else seems to be more likely to believe that 2024 LLMs are conscious if they believe chickens are conscious; about this, more later.

I have seen people say that the Digital Consciousness Model only gives an 80% chance that humans are conscious. This is a misinterpretation.

Pretend that you’re a Martian arriving on the planet Earth. Through a bizarre coincidence, you have exactly the same understanding of neuroscience, cognitive science, and philosophy of mind that humans do. However, you don’t know anything about the entities on Earth. You think about it, and you decide that, in your state of perfect ignorance, you put a 1 in 6 chance on any entity you encounter on Earth being conscious. (In Bayesian terms, this is your “prior.”)

You arrive on Earth and begin to collect information about how these Earth entities think. (In Bayesian terms, this is your “evidence.”) Afterward, you come to new conclusions about which entities are conscious. (In Bayesian terms, this is your “posterior.”)

The claim of the Digital Consciousness Model is that our example Martian should conclude there is an 80% chance humans are conscious. Obviously, your prior that humans are conscious is much higher than 1 in 6, because you’re a human and you’re conscious. Indeed, you don’t learn anything from the Digital Consciousness Model about the properties of human beings: you presumably already know that human beings are intelligent, are capable of self-modeling, have feelings, have bodies, and are biologically similar to human beings (...I sure hope).

However, the Digital Consciousness Model wanted to have all the probabilities be comparable. It would be confusing if, in addition to showing how convincing the evidence is in the abstract, the model accounted for the fact that we’re all very sure humans are conscious.

Indeed, the results of the Digital Consciousness Model are very sensitive to your prior:

Note that, if we’re being careful and rigorous here, your prior is not the probability you give right now that chickens or 2024 LLMs are conscious. You already know a lot of facts about chickens and 2024 LLMs—including a number of very basic facts which are included in the model, such as “LLMs can talk” and “chickens are animals.” If your prior is the probability you give right now that chickens are conscious, you’re going to be double-counting a lot of evidence. Instead, your prior is your probability that any given entity (that you’re bothering to ask this question about, so not a rock) is conscious.

Regardless of the prior, we conclude that humans are probably conscious and the therapy chatbot ELIZA is probably not conscious. But whether chickens or 2024 LLMs are conscious depends a lot on your prior. Since the evidence for chickens and 2024 LLMs isn’t overwhelming, your broader take about how likely it is that any entity is conscious matters a lot.

Similarly, we see that how certain we are about our uncertainty changes with the prior. You’re more uncertain about how uncertain you are about whether 2024 LLMs are conscious, if you start out believing that lots of beings are conscious. Similarly, if you start out believing that lots of beings are conscious, you’re really quite certain of chicken consciousness.

Probably differences in priors explain why people who believe chickens are conscious are more likely to believe 2024 LLMs are conscious: they’re more likely to believe anyone is conscious.

So why look at humans in the Digital Consciousness Model at all? Well, if the model found that humans were unlikely to be conscious, then it would be a very bad model; including humans is a sense check. And since we’re more certain that humans are conscious than we are of any particular theory of consciousness, we can reduce the probability we put on stances that are more likely to conclude that humans aren’t conscious:

Sorry, field mechanisms.

Now, we have some major caveats about the Digital Consciousness Model.

First, the Digital Consciousness Model represents all indicators and features as binary in order to make the model simpler: either you can do self-modeling or you can’t. In reality, entities can be more or less good at self-modeling, and entities that are better at self-modeling are more likely to be conscious. The Digital Consciousness Model does incorporate uncertainty, so this is only a problem if you think that we’re (say) very certain that chickens do only a little self-modeling. I think in general, the less an entity has a particular trait, the less certain we are that they have it at all, so I don’t view this as a serious concern.

Second, many indicators are very difficult to assess. For example, it is a complex question, widely disputed among the top LLM psychologists,3 whether LLMs can be said to “have” “preferences.” LLMs in particular are so goddamn weird that they challenge many of our basic concepts about how thought works. If you understand these concepts differently from how Rethink Priorities’ experts understood them, you will likely get different answers.

Third, “what is the probability that chickens maintain an internal representation of themselves different from how they represent other agents?” is a confusing and hard-to-answer question. I think even kind of fake probabilities can be helpful to clarify your thinking, but you shouldn’t take the results of the Digital Consciousness Model too literally.

Fourth, many stances were excluded. I think many of the excluded stances are pretty stupid (panpsychism, quantum mechanisms) or meaningless in a model of this kind (error theory), but if Your Favorite Stance was excluded the Digital Consciousness Model won’t be super helpful for you.

Fifth, the model only includes two entities that reasonable people are uncertain about (chickens and 2024 LLMs). I understand that Rethink Priorities intends to expand the model going forward to include a wider range of controversial entities, including 2026 frontier models, less humanlike AI systems such as AlphaFold or the Waymo Driver, and invertebrate species such as insects and nematodes. I eagerly anticipate their work!

How The Digital Consciousness Model Works

The Digital Consciousness Model is a Bayesian hierarchical model.

The model starts with a prior—an estimate of how conscious an entity is, before you learn anything about that entity. You then score the presence or absence of 206 traits called “indicators”. Indicators include:

• Is the entity made of carbon?

• Does the entity maintain an internal representation of itself that is different from how it represents other agents?

• Can the entity solve abstract visual reasoning problems?

Each of the indicators provides evidence about whether the model has one or more of twenty features. Features include:

• Is the entity biologically similar to humans? (Being carbon-based is an indicator of this feature.)

• Does the entity model its own traits, predict its own behavior, and differentiate itself from others? (Maintaining a distinct self-representation is an indicator of this feature.)

• Can an entity behave intelligently (solve novel problems, do abstract reasoning, etc.)? (Being able to solve abstract visual reasoning problems is an indicator of this feature.)

(The model actually goes indicator --> subfeature --> feature --> stance but I’m mostly skipping subfeatures to simplify the explanation.)

For each indicator, we ask:

• If a system doesn’t have a feature, what is the probability that it has this indicator? This is called “demandingness.”

  • Very demanding indicators are rare and much more common if you have the feature, but might not exist even if an entity really has the feature.

  • In math terms, this is Specificity / (1-Specificity) (don’t worry if you don’t know what that means).

• If a system has a feature, how much more likely is it to have the indicator compared to if it doesn’t have the feature? This is called “support.”

  • Indicators which strongly support a feature are much more common if an entity has the feature, but might also be really common in entities without the feature.

  • In math terms, this is Sensitivity / (1- Specificity) (again, don’t worry if you don’t know what that means).

In turn, each feature provides evidence for one or more of thirteen stances—perspectives on what gives us evidence for an entity being conscious. For example:

• Recurrent processing (perceptual) holds that consciousness “arises when perceptual inputs are subject to iterative refinement through structured, feedback-driven loops” (pg. 52), similar to vision in the mammalian brain.

  • Biological similarity provides evidence under the recurrent processing (perceptual) stance (moderate support, weakly demanding).

• Biological analogy holds that entities are more likely to be conscious if they’re biologically similar to humans.

  • As you might expect, biological similarity provides evidence under the biological analogy stance (overwhelming support, overwhelmingly demanding).

  • Intelligence provides evidence under the biological analogy stance (weak support, strongly undemanding).

• Attention schema holds that entities are conscious if they have a model of what they’re paying attention to, which they use to deliberately control and direct their attention.

  • Self-modeling provides evidence under the attention schema stance (strong support, strongly demanding).

  • Intelligence provides evidence under the attention schema stance (weak support, moderately demanding).

• Cognitive complexity holds that entities are more likely to be conscious if they think in a rich and complicated way.

  • Intelligence provides evidence under the cognitive complexity stance (moderate support, strongly demanding).

I am not A Statistics but I have done my best to explain what Bayesian hierarchical modeling is. All mistakes are my own. If you hate math, you can skip to the part where I say “math part over.”

For each link (indicator to feature, feature to stance), you have a probability distribution about how much evidence the former provides about the latter. (So you might think, based on the support and the demandingness, that the likelihood ratio is between 1 and 8.) For each indicator, you have a probability of whether the indicator exists. (So you might think it’s 60% likely the entity has the indicator.) You also have a probability distribution about what your prior is. (You might think it’s between 10% likely and 30% likely that the entity is conscious, before looking at any information.)

So you draw a bunch of random numbers and decide that in this run the entity has the indicator, the indicator-to-feature likelihood ratio is 2, the feature-to-stance likelihood ratio is 4, and your prior is 20%. Then, using Bayesian statistics, you go “the entity has this set of indicators, which makes me believe it is X much more likely to have this feature, which makes me believe it is Y more likely to be conscious according to this stance. Now I am going to combine all the stances together, weighted according to how plausible the experts thought they were. I have a final likelihood ratio, and I can combine that with my prior to find out my posterior.”

You then repeat that tens of thousands of times and average together all the posteriors you get. Eventually, new runs stop changing what the posterior value is. That’s your final posterior, and you can write your report.

Math part over.

Why would you do this silly thing? It allows you to incorporate all kinds of uncertainty. Not only are you uncertain about whether LLMs can do a particular thing, you’re uncertain about what that means for their broader psychology, and uncertain what details of their psychology means for whether they’re conscious under a particular stance on what consciousness means. Since everyone is in fact very uncertain about these things, the model allows us to quantify exactly how uncertain we are. From this, we can derive findings like “we are pretty certain of how uncertain we are about 2024 LLMs, but we are very uncertain of how uncertain we are about chickens.”

For a further explanation of why Bayesianism is useful, please see dynomight’s excellent ‘Bayes is not a phase’.

[Previously: Why join high-demand groups?, Identifying healthy high-demand groups, The structure of ideological abuse]

This is the fourth of four posts reviewing the book Abuses in the Religious Life. This review talks about some miscellaneous features of ideologically abusive groups. You don’t need to read the previous posts in the series to understand this one.

Information Control

De Lassus emphasizes most of all the importance of information control in forming a negative high-demand community.

De Lassus doesn’t elaborate clearly on why information control is so important. To me, it’s important because ideologically abusive groups’ primary enemy is the truth. An ideologically abusive group only functions if people fail to put together things like:

• “I’m not alone. My suffering is not because I’m broken and a failure. Everyone else is as miserable as I am. We’re just all hiding it.”

• “The linchpin is never satisfied with anything, and she contradicts herself constantly.”

• “The story of the miracles and heroism associated with the group’s founding is all lies.”

• “Very few of the promises the group makes ever come true.”

• “None of the beliefs we’re supposed to believe make any sense.”

• “No one experiences the spiritual bliss they were promised.”

• “Other groups achieve the same things as our group, but are far less wretched to be part of.”

• “The group has double standards for high-status and low-status people.”

• “The linchpin is living a life of luxury while we’re all poor.”

• “Wow, the linchpin is raping a lot of people. Like, a lot of people.”

Ideologically abusive communities have a “culture of lying.”1 Often, the lies are extremely stupid and about stuff that doesn’t matter. For example, de Lassus talks about a pair of religious brothers on a trip who had been told to have a particular private conversation, so they would return to the monastery after they’d normally be expected to return. The superior of the community announced, well before the brothers’ train had even left, that the train was running late so the brothers would arrive at the monastery late.

Of course, everyone in the order saw that this was an obvious falsehood. But that’s the point. The goal is to create a situation where all information is dubious, where it’s impossible to distinguish truth from falsehood. An ideologically abusive group maintains itself when everyone is desperately trying to believe claims that clearly aren’t true, or else questioning themselves about it: “is it possible that the train could be that predictably late? No one else is objecting to it. Maybe I’m just very wrong about the nature of trains.”

This dynamic functions just as well when the lies are small. Indeed, in some ways it functions better when the lies are small. People know they can’t trust any information, even about matters as trivial as train schedules; they feel insane.2 When falsehoods are common—especially when they’re common and you’re not allowed to point them out—it’s harder to notice the falsehoods that are important. And if you try to explain what’s going on to someone outside the group, you sound like a crazy person. What’s wrong with someone being mistaken about train schedules? It’s easy to assume that victims are making a big deal about nothing.

Some bad actors deliberately lie in this way, because they enjoy deceiving people or because they’re trying to make people question their understanding of reality. But it’s equally common for a culture of lies to emerge in a well-intentioned way. Maybe the superior is genuinely embarrassed by the private conversation, and doesn’t want people to speculate about it. Perhaps, at first, this even feels altruistic: maybe the private conversation is about (say) one brother’s masturbation habit,3 and the superior doesn’t want people to speculate about what the brother is struggling with that necessitated a conversation. But the ultimate effect is to create a culture in which no one can rely on any statement to be true.

And once people are used to lies as a tool to manage social situations, it becomes easier to reach for the tool in more and more minor matters. At first, you lie to genuinely spare someone’s feelings and protect their legitimate privacy from voyeuristic speculation. Soon you start lying to avoid an uncomfortable conversation or an inconvenient task. In time, lying may become so automatic that you do it for no reason at all.

If you want to lie to protect someone’s privacy or for another good reason, consider instead saying “I’m not going to answer that question”, optionally with a reason (“I’m under an NDA”, “it’s a personal matter that’s none of your business”).

Often, lies are used to cover up other lies. For example, someone might spot the brothers walking when they’re supposed to be on the train, and you’ll have to come up with some explanation. (”Oh, it was actually some other people who looked very similar...”) Further, inevitably some members of the community will know what the train schedule is. You can then wind up lying—or at least telling half-truths—about why you were lying. You might say “I was trying to spare the community distress” or “I was trying to prevent malicious gossip”, when in reality you were primarily afraid of inconvenient questions or tarnishing the community’s image in the eyes of its members.

Eventually, the community or its leadership may have told so many lies that finding out how many lies had been told would destroy the community’s or outsiders’ trust. So the community and its leaders feel like they have to keep lying to preserve the community. The number of lies expands until everything in the community life is subject to one deception or another.

Over time, community members come to justify their lies, to think of lying as good rather than evil. This is much worse than simply telling a lot of lies. If you start to believe lying is good, you’re no longer restrained by the qualms of your conscience. You no longer need a justification. You’ll lie whenever it is mildly convenient to do so. And it becomes nearly impossible to recognize that you’ve done wrong and make amends.

But the culture of lying isn’t the only way ideologically abusive communities control information.

In ideologically abusive communities, communication is often vague, ambiguous, and full of half-finished sentences. Communication occurs at inappropriate times, such as during prayers and services or when walking the halls. At any time, you can be jump-scared by some unexpected task or criticism. Authorities alternate in an unpredictable fashion between praise/approval and anger/contempt.

I’ve observed rationalists and effective altruists communicating in a vague and ambiguous way while looking like they’re communicating in a clear and direct way. Bad actors might pedantically nitpick other people’s sentences, or insist on arbitrarily precise and specific definitions of words, or claim that sentences are ambiguous even though a reasonable person would be able to figure out what they meant. These communication strategies create cover for vague, ambiguous communication: “it’s your fault you misunderstood me because if you checked Subclause 12(a) of my glossary it would be obvious what I meant.” And other people wind up hedging to the point of incoherence in the hopes that the bad actor won’t jump on a poor phrasing. As one of my beta readers remarked, it is never a good sign when someone is spending ten thousand words litigating the precise definition of “sexual assault” or “sexual harassment.”

Poor communication serves several purposes. It gives the linchpin and the leadership plausible deniability if their instructions go wrong (”I obviously didn’t mean that, why did you think I meant that?”). It prevents in-depth conversations that might reveal that the group’s beliefs, decisions, or actions don’t make any sense. Most of all, it keeps everyone on edge, stressed and nervous and insecure. Frightened people are bad at thinking.

Ideologically abusive communities often limit religious’s access to books. De Lassus is nuanced here. Monasteries and nunneries commonly limit access to the Internet: a monk who lives a life of quiet prayer and devotion secluded from the world shouldn’t be posting sick dunks on Bluesky. And monasteries may legitimately curate their libraries to exclude books that have nothing to do with being a monk. But there should be no limit on religiously orthodox books about spirituality, theology, or the teachings of the church. Religious shouldn’t require permission to visit the library during free hours.

I have a much stronger position here than de Lassus. While de Lassus wants people to be able to doubt the wisdom of their superiors and monastic orders, he distinctly doesn’t think it’s legitimate to doubt the Catholic Church. He believes it’s fine to limit the Da Vinci Code, not to mention The God Delusion, because he actually thinks it’s fine to deny people information that might cause them to stop being Catholic. Since I think Catholicism is false and “believing this particular set of claims regardless of the evidence” is not a virtue, I believe a high-demand community should never limit your access to books, particularly books which are critical of the group itself.

Ideologically abusive communities often de facto ban “horizontal” communication. All communication goes “up” from the member to the superior, and then “down” from the superior to a different member. In Catholic communities, this is typically justified spiritually: the spiritual benefits of silence or discretion, protecting others’ interior lives, participating in the unity of the Trinity.

Of course, even in orders that practice long periods of silence, you can’t literally ban members ever speaking to each other, even to ask to pass the salt. But you can make sure that all requests, conversation about feelings, sharing of significant information, or other conversation of any import goes through the superior. That way, the superior can easily quash any topics they find bothersome or uncongenial. No member is able to discover that their doubts, criticisms, and misgivings are shared, or to work together to confront the superior about their behavior.

The most important piece of information to share is the details of group members who leave: why they left, where they went, their physical and psychological state upon leaving. Whenever someone leaves a high-demand group, the other members should be aware of basic logistical details and, in broad strokes, what’s going on. People close to the person who left should understand what’s going on in more detail. There should never be silent disappearances about which no one ever speaks.

Knowing why and how other people left is the most important piece of information for judging whether you should and how you can leave. If you know how someone else got out, then leaving is a practical possibility rather than a series of question marks. If you know what made someone want to leave, that prompts you to ask yourself whether you want to leave too. “Why people left” is the most concentrated source of information about the flaws of a group and what it’s getting wrong. For this reason, it’s the most important kind of information for ideologically abusive groups to control.

Isolation

Isolation from people outside your high-demand community is bad. Any community—even a business or a family—can easily get caught up in groupthink and the normalization of deviance. The best preventative is to occasionally explain yourselves to an outsider who can, if called for, go “whatthefuckwhatthefuckwhatthefuckwhatthefuckwhatthefu—” Isolation also makes it hard for people to leave, because they have nowhere to go.

But in Catholicism, religious are typically isolated from the world at least somewhat; all orders limit communication with the outside world. Some (contemplatives, hermits) are very isolated. Isolation can help people focus on God and their callings, instead of being continually distracted by the outside world. And to some degree isolation is normal and inevitable: if you’re devoting yourself to a complicated project that consumes your entire life, you might not have much to say to people who aren’t working on the same project.

Some Catholic orders that strictly limit communications with the outside world are far healthier than other orders with looser rules. How can we get the benefits of isolation without risking the harms?

First, all social isolation should be justified based on the needs of the community. In a Catholic context, a contemplative order that maintains a robust inner silence to pursue a mystical relationship with God is obviously going to limit communication with the outside world. On the other hand, a community that focuses on doing good works and helping the poor needs to reach out to the secular world, that being where the poor are. If the latter has rules that look more like the former, that’s a red flag.

In a secular context, consider the reasons for isolation. Are you forbidden to talk about work because of legitimate confidentiality and information hazard concerns, or because your bosses don’t want outsiders to ask awkward questions? Are you avoiding people who are derisive and mocking about your goals, or anyone who doesn’t already agree that your goals are important?

Second, high-demand group members should have complete freedom of expression. It can be reasonable for a Catholic religious order to limit members to (say) one phone call and four letters with family each month. But the religious should be allowed to say whatever they want in their phone call and letters. And superiors should never read people’s mail.

Similarly, in a secular context, a high-demand group should never, absent a small number of situations where you need to e.g. keep work information confidential, forbid you from saying your genuine thoughts to the people closest to you. You should always feel able to talk about your fears, doubts, hopes, and miseries. If you’re unsure what to do about a problem, you should be able to talk about it. You should never feel obligated to conceal something from those you love because “outsiders wouldn’t get it” or “it would make the community look bad.”

Third, some kinds of isolation are always inappropriate. A high-demand group should never prevent members from speaking with their family, for example. It should also not limit what members say to their family about community life or their own personal lives.

For Catholic communities, it is always inappropriate to limit what people say to their confessor. In the Sacrament of Confession, Catholics tell their sins to a priest; the priest absolves them, which means that God forgives them of their sins. Most devout Catholics see a single priest for the Sacrament of Confession, who is called their “confessor.” Obviously, people need to be able to be honest with their confessor. While some limits might be reasonable if (for example) someone is going to Confession for an hour every day and not getting their work done, people should be able to see their confessor regularly and frequently for spiritual counsel and to get absolution for their sins. Ideologically abusive Catholic communities frequently limit people’s access to confessors. For example, they might forbid people seeing confessors outside the community; teach that they should hide some aspects of your community from their confessor because the confessor “wouldn’t understand”; teach that saying negative things about other people in confession is “malicious gossip”; or even say that asking for help from your confessor shows a lack of trust in God and a reliance on human help rather than supernatural help.

I have had a lot of Catholic and ex-Catholic friends, and it’s hard to describe to people unfamiliar with Catholicism how shocking I find the concept of limiting what people say to their confessor. The confessional is sacrosanct. Many, many people have died rather than reveal what was said in the Sacrament of Confession. If someone makes an incomplete confession of certain sins, Catholics believe, they will literally be tortured in Hell for all eternity, unless they manage to make a complete confession later. One of the best things about Catholicism is how much work the institutional church has put in to try to make people feel comfortable being honest and open in Confession about all the worst and most shameful aspects of their lives. I am stunned and horrified that any people who call themselves Catholic think that “you can say whatever you want to your confessor” is even in question.

In a secular context, a group should never require you to cut off your family (parents, aunts/uncles, siblings, cousins, spouse, children, grandchildren). You may decide of your own free will to cut off your family, if they’re abusive or mean to you, but this should never be a requirement of group membership. Similar rules, I would argue, apply to longstanding friendships. You should also be permitted to talk to a therapist of your choice.

Checks and Balances

Healthy communities have a system of checks and balances, so that no one person winds up having too much power. Ideologically abusive communities tend to route around these checks and balances.

In a Catholic context, the biggest limitation on an ideologically abusive community is the Church itself, which often (not as often as we would hope but, you know, often) takes a dim view of embezzling money, torturing group members, and committing rape. Usually, the communities with the worst spiritual abuse problems are the ones that try hardest to avoid church discipline. De Lassus:

In situations where the behavior of a community is becoming aberrant, we generally observe a curious dichotomy. On the one hand, we see an extremely demanding doctrine of obedience for the members of the community, which involves submission not only of the will, but of the intellect too, since every word of the superiors must be considered as a word of God. On the other hand, we find an extremely lax regime for the superiors themselves; there are no qualms about minimizing (or even completely and deliberately ignoring or setting aside) obedience to the Church and to canon law.

Often, ideologically abusive Catholic groups justify ignoring the Church by saying the Church doesn’t really understand the group. If the linchpin or the superior is seen as the voice of the Holy Spirit, any disagreement the linchpin or the superior has with the Church shows that the Church has been misled and is decadent. The group’s disobedience to the Church is seen as a deeper form of obedience to the true Church. Any attempt to crack down on the group is seen as persecution and therefore a sign of the group’s holiness.

De Lassus recommends, in terms of checks and balances in the Catholic context:

• A community should follow the canon law.

• The community should have a separate abbot, novice master, and cellarer.4 It should probably also have a separate prior (second-in-command).

• The community should have a constitution outlining how it is supposed to work, which is clearly written, unambiguous, concise, not overly rigid, and not so precise that the community has no ability to make allowances for changing circumstances.

• The community should be regularly visited by a bishop or a superior of the order, as appropriate, who interviews all the members of the community privately and who has the power to remove the abbot for misbehavior. During such a visitation, all orders to shut up about something should be considered not in effect.

• If there is reason for suspicion of the group, a special representative from Rome should visit to investigate.

Now, it may be difficult to figure out how to apply these insights in a secular context. Most high-demand groups in the secular world aren’t part of a larger hierarchy that can investigate whether they’re abusive. Many high-demand groups—like effective altruism—have widespread disagreement about how the community is supposed to work, which makes creating a constitution that applies to everyone impracticable. But here are my thoughts.

If a community has a point person who is supposed to evaluate whether a situation is unhealthy (such as the Astral Codex Ten Meetups Czar or the Centre for Effective Altruism’s Community Health team), no one should ever ask you to lie to them. If the point person is failing to do their job, the solution is a public reckoning, not a private campaign of deceit.

As with the separate abbot, prior, novice master, and cellarer, the community should have multiple positions of power which aren’t all held by the same person. Meetups should have two organizers for lots of reasons; this is one of them.5

If the community doesn’t have a hierarchy that can investigate, then the community should have multiple distinct organizations which can all check each other. For example, the effective altruist/rationalist community has the Centre for Effective Altruism, Coefficient Giving, 80,000 Hours, Lightcone Infrastructure, etc. Insofar as this is possible, no single group should be the only provider of anything: money, conferences, information-sharing programs such as magazines or forums, etc.

Organizations should be transparent in a way that permits people to independently investigate them. For example, they should make their financials publicly available as much as is possible. Organizations—particularly those that have a stewardship role over a particular community, such as the Centre for Effective Altruism—should actively communicate about their strategic plans and current programs. Non-disclosure agreements should be tightly scoped to actual confidentiality issues. Non-disparagement agreements and other anti-whistleblower measures shouldn’t be used.

All organizations larger than a local meetup group should have an independent entity, such as a board of directors, who is able to remove the leadership for misconduct. No one should hide anything from the board. Boards should not have conflicts of interest. Boards should be actively involved in their organization so that they would be able to notice misconduct if it happened.6

Important behavioral norms and standards should be written down somewhere. Norms and standards that aren’t written down can vary with the whim of the superiors or the linchpin. You need something you can point to to go “you shouldn’t do that.” Conferences, coworking spaces, meetups, and forums should have codes of conduct. If something is considered to be binding on people’s behavior—like the Giving What We Can pledge—there should be a single canonical spot explaining exactly what it binds you to. Looser norms should also, insofar as this is possible, be written down in some kind of central reference location, such as the Effective Altruism Forum’s community tag.7

A similar double standard to the “you have to unquestioningly obey me but I don’t have to obey anyone” double standard is that applied to criticism. Linchpins and high-status people arrogate for themselves the right to criticism of members, other communities, the broader movements they’re part of, and the world. Often these criticisms are harsh or public. At the same time, linchpins and high-status people consider themselves to be immune from criticism. If you criticize, you’re disloyal, disobedient, or harming the movement.

Of course, not all criticism is good. You should ideally make true, relevant criticisms, which come from a genuine desire to improve the situation rather than a desire to slander people. But any rule that forbids criticism of leaders or high-status people takes away the most important protection people have against ideological abuse. And a standard that criticisms must be relevant or made in good faith is often used to silence the criticisms that really cut home, while high-status people can make as many irrelevant bad-faith accusations as they like.

Pride

Often, the root of ideological abuse is pride-- the feeling that your group is unique and better than all other groups.8

Often, the groups are actually special. The group can wind up pleased with itself about how faithful it is and how much of a light it is to the godless world (or about how heroic it is and how much positive impact it has) because it actually is doing a lot of very good stuff. People compete to outdo each other in how morally good they are. For a time, this can have genuine positive effects on the world and the membership.

But over time people start to burn out. The level of goodness required isn’t sustainable. The temptation is to say “the people who burnt out and left bitterly just couldn’t hack it.” After all, look at what the community has accomplished! In this way, the community blinds itself to what they can learn from who leaves and why.

The community begins by observing that it does a lot of good; it continues by concluding that it does far more good than everyone else, and therefore is superior. De Lassus writes:

We often find here in an implicit form (or even explicitly formulated) the idea of some sort of divine mission, whose goal is to save the religious life or the Church, as well as a certainty that this group alone has managed to preserve truth or wisdom; it is like Noah’s Ark, in the midst of universal perdition.

Since the ideologically abusive group is special, their way of thinking is the only true and right way, sometimes the only way that is really true to the Catholic Church (or to feminism, or Marxism, or the Talmud, or rationality, or effective altruism, or anything else you like). For Catholic groups, any disagreement comes from the Devil, or reflects whatever the bugaboo of the group is: “worldly... psychological... carnal, deviant, mediocre, modernist (if we are talking about a traditionalist group) or traditionalist (if the group considers itself to be modern).” You can certainly list off others (”mediocre” is perennially popular among many different groups).

And of course, since the group is so special, it’s both wrong and immoral to think that anything the group does is bad, or even less than perfect.

Groups will often try to mark themselves as special in other ways. De Lassus writes:

In a community that feels as though it has reinvented everything, the extraordinary is to be found everywhere: the personality and history of the founder, the circumstances of the foundation, and even the basic elements of the community life. Priests for the community are not ordained by the local bishop but by the nuncio, by a cardinal from the Curia, or even by the pope, if possible.

Honestly, I feel like “if this group were Catholic, would they try to specifically get priests ordained by the pope because that’s how special they are?” is a great heuristic for pride.

Because the community is extraordinary, it follows that the community doesn’t have to follow normal rules. (Of course, high-status people and the linchpin are particularly exempt from normal rules.)

Healthy communities, conversely, are simple, humble, and boring. They do modest, everyday tasks. When you are part of a healthy high-demand community, you spend a lot of time answering emails, doing the dishes, organizing movie night, changing diapers, figuring out why the damn machine you’re trying to build isn’t working, or thinking in great depth about data visualization and fonts. You don’t have grand, dramatic stories of casting out demons or winning a thousand converts with a single speech. You have grant applications.

I wrote about this in Why Are There So Many Rationalist Cults?:

Early rationalist writing, such as the Sequences and the Harry Potter fanfiction Harry Potter and the Methods of Rationality, emphasized the lone hero, standing defiantly against an uncaring world. But the actual process of saving the world is not very glamorous. It involves filling out paperwork, making small tweaks to code, running A/B tests on Twitter posts. Most rationalists — like Mike Blume — adjust well to the normalcy of world-saving. (Today he works as a programmer and donates to AI safety and global health nonprofits, a common rationalist career trajectory.) Others want acts of heroism as grand as the threat they face.

The Zizians and researchers at Leverage Research both felt like heroes, like some of the most important people who had ever lived. Of course, these groups couldn’t conjure up a literal Dark Lord to fight. But they could imbue everything with a profound sense of meaning. All the minor details of their lives felt like they had the fate of humanity or all sentient life as the stakes. Even the guilt and martyrdom could be perversely appealing: you could know that you’re the kind of person who would sacrifice everything for your beliefs.

The project itself in each of these dysfunctional groups was vague, free-floating, and almost magical. As soon as you have to accomplish specific goals in the real world, the mundanity of everyday life comes flooding in, with its endless slog of tasks variously boring, frustrating, and annoying. But as long as you’re sitting in a room taking LSD with the blackout curtains over the windows, you can be Superman.

Conclusion

Ideological abuse is bad.

Ideologically abusive groups, at best, make their members fucking miserable. At worst, ideologically abusive groups rape, torture, and even murder people. Perhaps the most terrifying aspect of ideologically abusive groups is that they take ordinary people-- often very good, moral people-- and transform them into monsters.

And ideologically abusive groups are bad at achieving their goals. The culture of lies, silencing internal criticism, isolation from outsiders, getting rid of checks and balances, coercing obedience: these all cut the group off from any way to check whether what they’re doing is working. "We have to be ideologically abusive if we want to achieve our goals" is a pernicious lie.

If you’re an outsider—especially an outsider who is supposed to exercise checks and balances about a group—wondering whether a group is ideologically abusive, de Lassus recommends:

• Speaking, in private, with as many members of the group as you can, as well as former members. (If you’re not allowed to speak with members, that is very bad.)

• Looking for consistent testimony, especially consistent testimony from people who don’t know each other.

• Getting an outside perspective from wise and uninvolved people.

It is very difficult to reform an ideologically abusive group; most such groups ought to be dissolved. Reform usually requires a neutral outsider who is willing to put in quite a lot of work. It usually takes much longer than you expect (even accounting for it taking much longer than you expect).

If you’re the neutral outsider reforming a group, the top priority (de Lassus says and I agree) is to establish a culture of free speech. Other problems are much easier to correct if people feel free to criticize and disagree. Encourage people to form horizontal connections, not just vertical connections. Eradicate the culture of lying.

If you recognize yourself in what I’m writing, you should leave. Maybe the group is entirely toxic; maybe you’re in a toxic subcommunity or relationship; maybe it’s just a bad fit for you. It doesn’t matter. Quit the job. Move out of the shared house. Stop attending meetings. Drop the friends. The community will not get better on its own, or through your personal self-sacrifice. Anything you get from the community you can get somewhere else.

I understand that leaving some ideologically abusive groups is dangerous; if it is, then you might need to delay leaving while you work on safety planning. But delay, don’t stay. Work actively towards leaving, and in the meantime claim as much space and time as you can to keep your sanity.

[Previously: Why join high-demand groups?; Identifying healthy high-demand groups]

This is the third in a series of posts about the book Abuses in the Religious Life and the Path to Healing by Dysmas de Lassus. You don’t have to read the first two posts before you read this one. In this post, I will talk more about de Lassus’s definition of spiritual and ideological abuse.

When talking about de Lassus’s ideas, I will use the term “spiritual abuse”, as he focuses on religious communities. When talking more broadly, I will use the term “ideological abuse”, which doesn’t assume that the abusive community is religious in nature.

What Is Ideological Abuse?

According to de Lassus, the thing that defines spiritual abuse is an inappropriate intrusion onto the internal forum.

The external forum is all matters having to do with society and the public good. For example, the external forum includes whether you go to church, what job you work, how much you donate to charity, and whether you’re legally married. The internal forum has to do with your inner thoughts, especially your conscience. For example, the internal forum includes anger, generous impulses, a conviction that a course of action is morally wrong, and the impulse to refresh X two hundred times a day.

An ideologically abusive community might interfere with the external forum in various bad ways: for example, they might command you to cut off your family, to fast when it would make you sick, or to donate an unsustainably high percentage of your income to buy the pastor a third Lamborghini. But the root of the problem—the reason you don’t just tell them to fuck off—is that the community is interfering inappropriately in the internal forum. They’re exercising control of your beliefs, preferences, and conscience in a way that they don’t have a right to exercise.

Inappropriate intrusion into the internal forum can cause serious mental health problems. De Lassus quotes a spiritual abuse survivor describing their experiences:

I came little by little to feel like a dead tree, like a tree that had had its branches cut off. After the branches have been cut off, there is only the trunk left, and when I spoke of things like this, they said to me: “That is really wonderful; this means that you are really being pruned for Heaven.” So I said to myself: “It must be normal, then, but it seems strange that I feel less and less alive in terms of this world.” But they said to me: “It is completely normal; it is the Cross. Look at the wood of the Cross; it is like a dead tree, and yet it is this that gives life.”

Typical mental health effects of spiritual abuse, according to de Lassus, include the usual nonspecific depressive symptoms: insomnia, psychosomatic illnesses, neglect of personal hygiene, self-harm. Spiritually abused people may work too much so that they can cling to the scraps of approval they get from the community. On the other hand, spiritually abused people may behave in a harmful, coercive, or pointlessly defiant way as a desperate attempt to maintain some control over their lives.

Catholic spiritual abuse victims often refer to spiritual abuse as “a violation of the chastity of the inner self” or as “a rape of the heart.” The former is a metaphor that secular people are less likely to use, but I think it is evocative of the problem. The way that sexual abuse is a violent intrusion on a private, intimate part of the body—something people want to only share by choice, and normally only with people they love and trust—spiritual abuse is a violent intrusion onto a private, intimate part of the mind—something people want to share by choice, and normally only with people they love and trust.

Many communities are supposed to intrude onto the internal forum. A Catholic monastery is supposed to strengthen your love of God and neighbor; if you spent decades in a monastery and were just as much as an asshole as you were before, you’d want a refund. Likewise, introductory effective altruist material is supposed to make you have higher ambitions, think with numbers, and care a similar amount about all strangers. The rationality community is supposed to make your beliefs more accurate and your plans more likely to achieve your goals. But when a community is supposed to intrude on the internal forum, we should be cautious to make sure that the intrusion stays within proper bounds.

I’m going to talk about the concept of the “linchpin”, then I’m going to talk through the three themes that de Lassus considers most common in spiritual abuse.

The Linchpin

In ideologically abusive communities, everyone is both a victim and a perpetrator. Everyone has complaints that they’re hiding, because they know their complaints are a sign that they’re weak and bad and sinful, and everyone is being cruel and judgmental to others for having ordinary human weaknesses or needs.

Why? In a small-group abuse situation, such as a family, the perpetrator can keep everyone in line with threats, guilt, and gaslighting. But if she wants to abuse dozens or hundreds or thousands of people, even the most determined perpetrator will need to delegate. She needs abuse middle management.

Since it’s impossible for a single abuser to enforce ideologically abusive dynamics all by herself, she needs her victims to do it for her. And she also needs to keep the enforcers in line themselves, so that they don’t start getting any bright ideas about improving the situation somewhat. In short, ideological abuse needs the cooperation of the victims.1

That’s what makes ideologically abusive communities so fiendishly hard to untangle. Realizing how bad the group is means realizing that you’ve done evil. People naturally flinch away from that.

While everyone is a perpetrator, usually a small “inner circle” is particularly responsible for negative dynamics in the community. De Lassas calls this the “linchpin”, the center of the community.

The linchpin isn’t the same as the colloquial idea of a cult leader. While the linchpin sometimes has a position of power in the community, they often don’t. Sometimes they’re an ordinary member or even (in religious life) a layperson. By the same token, decentralized communities that don’t have a leader may well still have a linchpin or three.

It’s also a mistake to assume that the linchpin is, like our stereotype of a cult leader, giving dramatic speeches, demanding obedience, verbally abusing people, or threatening violence. In my experience, some of the most powerful linchpins never raise their voices. A linchpin may present herself as a weak and helpless victim, someone the community should take care of. Or a linchpin may make herself ever-present and useful in small humble ways; never underestimate the power of being the person who handles the logistics and takes the minutes at the meetings. Or—especially in the modern day—a linchpin may simply be a popular writer or thinker, and everyone might believe that they were simply convinced by the quality of the linchpin’s arguments.

What makes a linchpin is that they’re the community’s “single point of reference.” The entire community winds up warped around the linchpin’s whims. The primary goal of the community becomes protecting the linchpin’s feelings and ensuring that the linchpin is never contradicted. Often, community members become obsessed with the linchpin’s health, their activities, and the state of their soul.

If the linchpin falsely accuses someone of some wrongdoing, the accused can wind up confessing their sin, even without obvious coercion and even while knowing inside that they’re innocent. To maintain their own innocence would be unthinkable. If the linchpin said something is true, it must be.

To identify linchpins, a beta reader suggests asking, “who always gets their way when there’s a win/lose situation and why?” To be sure, sometimes one person might legitimately keep winning conflicts—maybe she has good judgment, maybe she hosts the events, maybe she’s disabled in a way you need to accommodate, maybe she’s three months old. But if there’s no good reason for someone to always win, the community is likely warped around making them always win.

The linchpin is typically placed above every rule. If everyone else is an ascetic, the linchpin lives a life of luxury. If everyone else has taken a vow of poverty, the linchpin has a personal bank account. If everyone else is separated from the world, the linchpin can talk as much as she likes with outsiders. If everyone else’s Internet use is limited, the linchpin can spend twelve hours a day on social media. If everyone else has to attend services and prayers several times a day, the linchpin can skip whenever she wants.

Often, you’re not supposed to notice or acknowledge the ways that the linchpin is breaking the rules, and you’re definitely not supposed to write it down somewhere so you can check how often she does so.

The linchpin may make a point of all the rules she follows that don’t bother her very much. De Lassus writes:

A certain hypocrisy can even be established. In one new community (which, incidentally, has not survived), the religious were forced to live in the harshest conditions, sleeping in dormitories in a poorly-heated building, and having no form of private life, while the founder was leading a pleasant existence, living alone in a separate building, where he had the benefit of all necessary comforts — and maybe a few more besides. But out of humility, he used to celebrate Mass barefoot.

Again, while some linchpins make the argument (at various levels of explicitness) that they’re in charge so they deserve to be treated better, not all linchpins will. A linchpin may well demand special treatment because she has been so oppressed and victimized in the past.

Do all ideologically abusive groups have a linchpin? No. An ideologically abusive structure can perpetrate itself without anyone particularly wanting it to keep going. If everyone is a victim and everyone is a perpetrator, you don’t need an abuser for a group to be abusive. A truth that is difficult to talk about is that many people want to be in ideologically abusive groups. As I wrote in Why Are There So Many Rationalist Cults?:

But people who are drawn to the rationalist community by the Sequences often want to be in a cult. To be sure, no one wants to be exploited or traumatized. But they want some trustworthy authority to change the way they think until they become perfect, and then to assign them to their role in the grand plan to save humanity. They’re disappointed to discover a community made of mere mortals, with no brain tricks you can’t get from Statistics 101 and a good CBT workbook, whose approach to world problems involves a lot fewer grand plans and a lot more muddling through.

Many people want to submit the internal forum of their lives to a charismatic authority instead of taking responsibility for their own emotions and moral life. A dynamic both de Lassus and I have noticed is the charismatic leader who doesn’t seek out being a linchpin, but who is treated as a linchpin by her followers.

That said, a group without a linchpin will tend to acquire one. People who want to be linchpins will take advantage of a linchpinless ideologically abusive group. And—while virtuous leaders can dissuade their followers from treating them as a linchpin, albeit with mixed success—many will be seduced by the praise and power. If everyone you talk to thinks you’re right about everything and is trying to cater to your every caprice, you don’t have to be a very bad person to give in.

Claiming Authority Over Conscience

The first theme of spiritual abuse we’ll discuss is claiming authority over conscience. “Claiming authority over conscience” means replacing someone else’s moral judgment with your own. When someone has claimed authority over your conscience, instead of making your own moral choices and following your own sense of what is right, you unquestioningly obey the authority—or are shamed and punished when you don’t.

Even outside high-demand groups, people might want you to obey them sometimes. For example, your manager wants you to prioritize a particular project or design the website the way she wants; the Discord moderator wants you to thread, or stop talking about entirely, a particular topic of conversation; and the local-group organizer wants the meetup to happen at a particular time, in a particular location, and with particular activities. But they shouldn’t demand authority over:

• Your beliefs (”if we design the website this way, the users will be confused”).

• Your preferences (”arguing about whether trans women are women is both fun and enlightening”).

• Your judgment (”all things considered, the llama pen at the petting zoo is a terrible place to hold a meetup”).

You should always be allowed to have your own preferences, beliefs, and judgment. You should never be required to replace them with community consensus or the views of the linchpin.

Now, beliefs and judgment can get a bit complicated. Even in low-demand communities, people aren’t socially allowed to believe whatever they want: in almost every community in the Anglosphere, if you start taking about racial differences in IQ or how six-year-olds can consent to sex, you’ll lose all your friends. For de Lassus, the distinction is easy: your religious community can command you to believe the teachings of the Catholic Church, but shouldn’t command you to believe anything else. For people who don’t believe the Holy Spirit guides the Catholic Church towards truth, the line between normal social norms and claiming authority over conscience can be very blurry. Here are some guidelines I find helpful:

• It is a green flag if your community punishes basically the same beliefs the wider culture punishes (e.g. Nazism, pro-pedophilia activism).

• It is a green flag if your community punishes very few beliefs, and allows widespread disagreement on most topics.

• It is a green flag if you basically know what beliefs your community is going to punish: they don’t change over time, or expand or contract with the whims of the linchpin.

• It is a green flag if popular people and unpopular people are held to the same standards.

• It is a red flag if you aren’t allowed to disagree about practical issues (such as webpage design, how best to get people to pick up after themselves, or cabbage-growing techniques).

• It is a red flag if you aren’t allowed to disagree about how best to accomplish the purpose of the community (e.g. which methods are most effective for persuading people, whether vaccinations are more cost-effective than malaria nets, whether you should work for a frontier AI company, or what meditation techniques best lead to enlightenment).

I think it is inherently toxic for your community to pressure you to want a different thing than you want. You can decide to try to change your preferences, but that’s a personal matter that your community should have no say over. You might choose not to pursue your desires (e.g. I choose not to eat eggs), but no one should stigmatize you for wanting something (e.g. I still like egg-containing baked goods).

A second example of claiming authority over conscience is requiring people to share their interior life—their feelings, thoughts, preferences, temptations, hopes, fears, etc.—when they’re not comfortable sharing it.

Let me be clear here. It’s important to have someone to talk to about your interior life. Many people do have to have a general sense of what’s going on with you: for example, your boss or housemates might need to know, in general terms, that you’re going through a hard time, so that they know how to support you and when to cut you some slack. And if you want to be a good person, you need guides and companions on the path: people who share their experiences so you don’t feel so alone, whose disappointment you can imagine when you’re struggling to hold yourself to account, who help you problem-solve when you’re stuck, and who call you on your shit.

But, fundamentally, opening your heart to someone does no good, and often rather a lot of harm, if it’s someone you didn’t choose or don’t feel safe around.

You must never require someone to share their inner life with you without their free, full, informed consent. That is, you shouldn’t get people to share their inner life with you by ordering them, manipulating them, pressuring them, or using your power over them to punish or reward them for it. Indirect pressure is still pressure: you shouldn’t go “you don’t have to share your interior life with me, but if you don’t you’ll be mediocre forever.” It is even questionable to tell someone that they ought to share their interior life with you, outside of some limited situations like marriages or therapy where someone with free, full, informed consent chose to commit to a certain level of emotional intimacy.

Lots of things, like finances, should be required to be transparent. But it is never acceptable to demand that a person’s thoughts or conscience be transparent. You should feel able to keep some things private, even from your spouse or therapist.

Inappropriately requiring people to share their inner lives leads to people sharing other people’s confidential information. De Lassus writes:

In the context of an openness that has been forced, indiscretion generally becomes the rule, springing from a feeling of having rights over the other person’s inner life: I have the right to know everything and the right to do what I want with what I know. In a pyramid structure, the violation of confidentiality is, moreover, part of the structure itself, given the requirement to relate everything to the head of the community. Intermediaries have to pass everything on, so nothing can be kept confidential. Experience shows that if someone lives long enough in this kind of atmosphere, the very meaning of confidentiality is lost, and many people are unable to keep themselves from saying what, deep down, they know they should be keeping to themselves. Even the seal of the confessional may be broken.

Another example of claiming authority over conscience is coercing people into entering a high-demand community or not letting them leave. No high-demand community is right for everyone, or even for most people. A healthy high-demand community actively rejects people who are a poor fit. Catholic monastic communities formally reject people’s application for membership. More informal communities might need a social norm of going “this seems really bad for you. Do you want to go somewhere else?”

Each individual must freely choose to be part of the community. It is very bad for a religious community to say to a candidate “God is calling you to be in our community,” because that means that the monastic life isn’t a free choice. Similarly, it is wrong to try to convince someone to stay in a monastic order if they feel called to leave. It is wrong to assume that someone’s desire to leave is a sin or a temptation from the Devil. You definitely shouldn’t attempt to cast out the demons that are making someone want to leave (!).

Similarly, in secular communities, people should feel social permission not to join a community and to leave if it isn’t working out for them. Leaving shouldn’t mean losing all your friends in the community. Maybe you aren’t invited to the effective altruist meetup anymore if you’ve left effective altruism, but you should still get invites to birthday and New Year’s Eve parties. No one should feel pressure to cut someone off solely because they left the group.

In general, the exit ramp should be comfortable. You can quit your job at the effective altruist organization and stay in the effective altruist group house; you can leave the effective altruist group house and still come over for dinner sometimes.

Poisoning the well against other communities is a red flag. Often, I’ve noticed, an ideologically abusive community will acknowledge that other legitimate communities exist—in theory. But in practice Alternative A is mediocre and uncommitted and won’t accomplish anything, and Alternative B is racist and misogynist, and Alternative C is run by a secret pedophile cabal, and Alternative D is just cringe. Healthy high-demand communities, on the other hand, know about and refer people to alternatives—whether that means the Carthusians telling people about the Jesuits or the Astral Codex Ten meetup telling people about TPOT.

No high-demand community is the only source for anything. You can find other groups with the same traits and goals: Catholics have many different religious orders; groups ranging from the Gates Foundation to GAVI fight global poverty without being effective altruists; rationalists are only one kind of weird nerd. You can also pursue important goals on your own: you don’t need to coordinate with the AI safety movement to get a job in the civil service or a think tank and advocate against AI risk; you can meditate on your own or practice solitary paganism or pray to God without an intermediary. Pursuing goals on your own is lonely and relies a lot on your good judgment, but it’s a hell of an improvement on being abused. (Hint: ideologically abusive groups are rarely particularly good at achieving their goals in the long run, so they’re not hard to outperform.)

It is a horrible, horrible mistake to think you have to hang out with some specific group of people to achieve any goal. Any high-demand community that says the choice is them or nothing is lying.

Secrecy about the community’s teachings is also a red flag. De Lassus writes:

One sister who asked the question [about why the Rule was hidden] received this explanation: “Because the rule is so sublime that we can’t understand it before we make our profession, i.e., before we have been properly initiated. Otherwise, we would be overwhelmed by its sublime nature, and as a result utterly discouraged when confronted with our own nothingness.”

If you don’t know what a community really teaches or what its norms really are, you can’t informedly consent to join it.

People don’t like to admit that a lot of people want others to claim authority over their conscience. Self-determination is scary. If you have to decide for yourself what’s good and right, you might be wrong. It’s much easier to submit yourself to someone else, knowing that if it all goes terribly then you’re not at fault. Personal responsibility is perennially unpopular, which is one of the major reasons people seek out ideologically abusive communities.

Demanding Total Self-Abandonment

“Total self-abandonment” means you are no longer an independent person. You are supposed to act as the hands and mouth of another person, whom you are completely loyal to and who knows everything about you.

In some ideologically abusive Catholic communities, religious take a vow to trust their superiors. The Legionaries of Christ went so far as to vow never to say or write anything that would “discredit the person of the superior or his authority” and to report anyone who did so.

This is very bad. No one should ever try to force you to trust them unconditionally. Trust is earned, not required.

De Lassus has to be fine with vows of obedience because he’s Catholic and they do those (and vows of obedience do serve as a drama reduction measure). In a secular context, you obviously need to obey people in some circumstances. Your boss can expect you to make the work product to her specifications, even if the specifications are very stupid. The meetup leader or owner of a coworking space has the final say over what behavior is acceptable at the meetup or coworking space.

But in a secular context obedience is narrowly scoped: the meetup leader can’t forbid you from dating someone or hanging out outside of the meetup; your boss can’t rule on your hobbies or friendships. And, if someone’s judgment is very bad, you can always quit your job or stop attending the meetup.

In the secular world, people rarely take explicit vows to trust group leaders. But social pressure can serve to force people to unconditionally trust high-status people or the linchpin. Someone might be a “high-level rationalist” or a “core effective altruist”, so it seems like her judgment and ethics must be better than yours. If you disagree with her about a course of action, you are at fault.

I don’t mean that you should never defer to people who know more than you. If an Anthropic employee corrects me about some detail of how large language models work, I’m going to assume they’re right and I’m wrong. But my deference is narrowly scoped: I don’t assume that Anthropic employees are right about healthy vegan diets or the best methods for preventing burnout. I remain open to the possibility that the Anthropic employee is mistaken. I don’t defer about questions of values (“how bad would it be if humanity went extinct?”) or complex and uncertain areas where experts disagree (“when will we develop AI?”, “how should we best align AI?”). And as I learn more about AI, I’m less likely to defer and more likely to rely on my own judgment.2

Disagreement, criticism, and people following their own consciences are our best protection against groupthink. And ultimately, as a person with agency and inherent dignity, you need to rely on your own judgment and your own sense of morals, not outsource them to some authority. Total self-abandonment is one of the worst ways you can wrong yourself.

Sometimes people try to make it impossible for you to notice that you disagree with them or think they’re asking you to do something morally wrong. Not to be a stereotypical Quaker, but I can’t overestimate the importance of quiet, unscheduled time—by yourself or with a trusted friend—in which you can think through whether what’s being asked of you is stupid or evil. Often, you’d notice that quiet voice inside you that says “this isn’t quite right” if you had a moment by yourself to think—so the ideologically abusive group will make sure you never have a moment by yourself to think. Constant busyness is an underrated red flag of ideological abuse.

However, total self-abandonment doesn’t have to come from pressure to trust any specific person; it can come from seeking uniformity rather than unity.

People in a high-demand community need to behave in the same way, which is different from how other people behave. All religious take vows of poverty, chastity, and obedience. Each individual order has its own demands: Trappists are very communal, while Carthusians spend a lot of time alone. And each individual abbey has its own culture, its own “feel.” Similarly, effective altruists talk, think, and behave in ways which are different from people who aren’t effective altruists. Each workplace, coworking space, meetup, and group house acquires its own unique culture and “feel”.

This is called “unity” and it’s good. Becoming more similar to other people is a natural part of being in any community, even a low-demand community.

At the same time, normal people don’t take up the religious life or join any other high-demand community. People in high-demand communities are weird; they aren’t all the same kind of weird, either. A good high-demand community embraces people’s weirdnesses. Uniformity is a Procrustean bed where you cut off bits of people to fit.

The task is to learn to be self-effacing without destroying yourself:

• You might be asked not to impose your preferences on others; you shouldn’t be asked not to have preferences.

• You might be asked to be okay with someone else being chosen for a job that you think you’d be better at; you shouldn’t be asked to falsely believe you’re bad at something you’re good at.

• You might be asked to tolerate other people thinking differently; you shouldn’t be asked to think the same way as them.

• You might be asked to participate in community recreation you don’t really enjoy but which other people do; you shouldn’t be asked to like things you don’t like.

• You might be asked to explain why you don’t agree with the decisions of leadership; you shouldn’t be told, “I feel like you don’t agree with my decisions. You must say to me that you agree.”

Often, in spiritually abusive communities, having divergent preferences is itself seen as sinful or as the product of a deliberate action from Satan. Among rationalists and effective altruists, it may be seen as irrational, unaltruistic, or unagentic. In other groups, it may be seen as misogynist, racist, woke, untrad, unenlightened, unspiritual, or manifesting negativity.

Unity isn’t about getting rid of your preferences and replacing them with better ones. Unity is about learning to listen, to respect others’ preferences, to see the good in what other people like, and to figure out what is best for everyone. If your preferences change in this process—if seeing the good in Gregorian chant makes you appreciate it, for example—that’s good. But you can respect other people’s love of Gregorian chant while continuing to dislike it yourself.

Uniformity, on the other hand, is about taking the unique person you are and squeezing it until you fit someone’s predetermined idea of what a Moral Person or a Good Group Member looks like. Ultimately, the goal is to reshape you into a good tool for the linchpin’s or the group’s purposes—a tool that has no inconvenient traits that might get in the way of working out the will of those in authority.

Spiritual doctrine

“Spiritual doctrine” is the overspiritualization of ordinary situations.

Spiritually abusive groups are obsessed with finding the will of God in everything. De Lassus explains that God doesn’t have an opinion on whether you wear a sweater. He gave you a brain so that you could figure out for yourself whether to wear a sweater. The linchpin, however, very well might have an opinion on whether you wear a sweater. So doing the will of God becomes doing the will of the linchpin.

Spiritual doctrine can also be used to justify various sorts of wrongdoing. De Lassus gives a few examples of the use of spiritual doctrine to justify sexual abuse:

Jacques Poujol gives several examples of this: “See it as a blessing from God”; “God has told me that it is a good thing; our love is special”; “It is a privilege for you that I have chosen you”; “By loving me, you are loving Jesus.” Acts that are ambiguous or even frankly sinful are presented as an expression of God’s love, which needs to be incarnate if it is to reach people. You need this if you are to discover God’s love.

Presenting the abuse as something mystical also provides the easiest way to justify keeping it secret: What happens between you and me is a special grace, which others would not understand. This is a simple way of presenting immoral violence as the summit of union with God. Moments of doubt may spring up, but at such times, the victim may be reassured by the notion that this is an exception to normal rules: “It’s allowed for us. The grace that we are living is so high that it sets us above ordinary laws. The degree of love that we have reached allows us to do anything.”

At first, it might not seem obvious how the concept of “spiritual doctrine” applies to nonreligious ideologically abusive groups, which have no God to seek the will of. But I am reminded of the words of one of my interviewees in Why Are There So Many Rationalist Cults?:

“It’s a story in which they matter and in which it is justified for them to do weird stuff and stand up for themselves,” said an interviewee familiar with the Zizians. “Every action has great meaning, and that hooks into people in two ways. One of which is that it’s empowering, and the other of which is that it’s a great trigger for becoming obsessed with whether you’re a bad person.”

He continued, “It makes it easy for small things to seem very big. And I think it also makes it easy for big things to seem sort of the same size as small things. When you get pulled over and then you get in a gunfight with the cops or whatever, this is the same level of treating the situation like it is anomalous or a big deal as having an argument about who washed the dishes.”

Spiritual doctrine makes small things big and big things small. Wearing a sweater is a matter of obedience to the Perfect Will of God; being raped is nothing compared to the love and grace of God.

Any community with unusual beliefs is going to disagree with the mainstream about what things are important. “Not eating eggs” used to be a major moral self-improvement project for me, and it has now been replaced by “send networking messages” and “tell people they can hire you for freelancing work.” But the community has to agree that some things are unimportant. There must be some issues about which the teachings say nothing, and which are simple matters of personal taste.

If everything is important, nothing is.

Similarly, a high-demand community must be cautious about believing true things about the world. As I’ve written in the same piece:

But if you’re using your decision theory to make decisions and not just to get a PhD in mathematics, it really matters that you chose correctly. If you’re using your all-encompassing theory of human psychology to decide how to treat people, it really matters that you actually understand it. And all of these are more dangerous if, instead of following reasoning you understand, you’re deferring to the judgment of someone else who seems like they’ve thought about it a lot.

Agency and taking ideas seriously aren’t bad. Rationalists came to correct views about the COVID-19 pandemic while many others were saying masks didn’t work and only hypochondriacs worried about covid; rationalists were some of the first people to warn about the threat of artificial intelligence. If you want better outcomes than normal people, then you also need to do something different than what normal people do. But diverging from the norm is often risky and the risks have often not been taken seriously.

The more seriously you take acting on your beliefs, the more dangerous it is to have beliefs about the world that aren’t true. To be sure, some beliefs aren’t that dangerous even if you get them wrong: even if you think Catholicism is true, believing a Trinitarian heresy is unlikely to result in ideological abuse. But if your beliefs about psychology, society, or physical reality are mistaken, it can cause serious harm—especially if these beliefs are so core to the ideology that they feel impossible to question.

People often underestimate the extent to which communities are ideologically abusive because they believe specific things that justify ideological abuse. Of course, sexual abuse happens because victims are dependent on their abusers, because of isolation, because of a culture of thoughtless obedience, and so on and so forth. But it also happens because abusers teach that having sex with them is a special grace from God that sets you above ordinary laws, and victims believe them. Unless we understand the role of belief in ideological abuse, we’re going to be mystified by how it works.

[I have freelanced for a number of effective altruist organizations, such as the Centre for Effective Altruism and 80,000 Hours. I don’t speak for any of my clients, past or present, and they didn’t look at this post.]

Abuses in the Religious Life and the Path to Healing is a book about spiritual abuse by Dysmas de Lassus, the prior general of the Order of Carthusians. The prior general is the person in charge of a Catholic religious order.

Catholic monks and nuns (the general term is “religious”) inherently give up a lot of control over their lives to the order. The order decides where they live, what they do, whom they spend time with, how they dress, and what they eat. Religious take vows of obedience: they are required to do whatever the abbot says, as long as it isn’t actively sinful. The Carthusians in particular live in isolation from the world, in solitude and silence, to spend their time contemplating God.

And yet some religious are happy, thriving people who live meaningful lives that make the world better; other religious destroy themselves in the service of (at best) an unattainable ideal or (at worst) an evil rapist grifter. The task of Abuses in the Religious Life is to help people predict which orders produce which.

I am in a high-demand group: effective altruists. Effective altruism has a say in my career choices, my spending, my diet, my discourse norms, and even whether I’m a kind and friendly person. I think that me being an effective altruist not only leaves the caged chickens and poor African children better off, it also makes me happier.

But I—like any reasonable person—feel somewhat leery about being part of a high-demand group. How do I continue to have my friends hold me to high standards of behavior, without creating dynamics that destroy our lives and the good we planned to do in the world?

Healthy High-Demand Groups

Before we talk about toxic high-demand groups, I think we should talk about what high-demand groups look like when they’re healthy.

One thing you might think is a good sign, but actually isn’t, is that people are good. There is no general pattern where people in healthy high-demand communities are, compared to people in toxic high-demand communities, more hardworking, generous, loving, self-controlled, courageous, honest, tolerant, clever, helpful, cheerful, or even compassionate (to outsiders).

Toxic high-demand communities often create a culture of competition to be the most ethical. Each person strives to be more generous, uncomplaining, and self-sacrificial than the next. So a toxic high-demand community can have more virtuous members and a greater positive impact on the world than a good high-demand community (at least until its members burn out and its disconnection from reality makes its projects collapse in failures easily predictable by a smart fourteen-year-old).

At the same time, toxic high-demand communities generally pervert genuine virtues (or at least traits considered virtues in the broader community’s cultural context). For Catholics, humility becomes self-hatred; the desire to give of yourself to others becomes complete self-denial; forgiveness becomes forgetting the crimes of unrepentant abusers. For rationalists and effective altruists, consequentialism becomes tolerance of wrongdoing because of some far-off future benefit; agency and taking ideas seriously become hearing a bad argument and doing arbitrary bad stuff because of it; cultivating the art of rationality becomes tense twelve-hour conversations about the deeper psychological implications of you eating the last slice of cake. The members of a toxic Catholic community may well come off as humble, self-sacrificing, and forgiving; the members of a toxic effective altruist community, consequentialist, agentic, and dedicated to self-improvement.

Issendai once wisely said, “people rarely get stuck because of their vices. They’re usually caught by their virtues.” Loyalty, patience, hope, forgiveness... these are admirable qualities, and you are right to admire them. But the members of a toxic high-demand group may well be more patient, loyal, hopeful, and forgiving than average. That’s what’s keeping them there.

So what virtues do you actually expect to see in a healthy high-demand community?

A healthy high-demand community is patient with its members. It doesn’t expect perfection immediately. It doesn’t hold people to unreasonable standards. It accepts that mistakes and failures are part of the human condition.

Relatedly, people in a healthy high-demand community freely admit to getting things wrong and to having flaws. In particular, though de Lassus doesn’t mention this, I’d say that they admit mistakes that aren’t the equivalent of a Soviet criticizing the USSR by saying “You know, people just don’t respect Comrade Stalin enough. There isn’t enough Stalinism in this country! I say we need two Stalins! No, fifty Stalins!” Member admit to errors of judgment, silly goofs, and boring practical mistakes; they don’t just admit to being slow to see the wisdom of our Fearless Leader and our Sacred Doctrine.

Any community has high-status people. Obviously, many groups—from a military unit to your local board-game night—have official leadership; in Catholic religious life, these people are called “superiors.” But even decentralized groups have people whom everyone respects, whose beliefs and wishes are taken seriously, and who have more say in what the group does. In the effective altruist community, high-status people include local group organizers, prominent bloggers and researchers, and the leadership of organizations like 80,000 Hours and Coefficient Giving.

One of the most important green flags in a high-demand community is the personality of the high-status people. High-status people should readily admit that they make mistakes, believe wrong things, and have personality flaws. A culture of admitting mistakes is no good, and may even be harmful, if high-status people are exempt.

High-status people should be aware of the suffering of those around them, particularly suffering that’s related to the beliefs and commandments of the high-demand group. If possible, high-status people should do something to alleviate the suffering of group members; if not possible, they should provide comfort and understanding. Most of all, high-status people should be genuinely kind. Not righteous, not self-sacrificing, not heroic, not good. Kind.

In some rare cases, a community has legitimate goals that are served by some high-status people not being kind: for example, a teacher of a particular skill might be harsh and strict but fair. But these situations should be narrowly circumscribed (e.g. one class, not someone’s entire life) and for a particular purpose. In general, the unkind high-status person should have a boss whose job is to monitor the situation, and who is herself a kind person.

Now, no high-demand community is exclusively lead by saints. Everyone sometimes stubbornly refuses to admit fault, ignores suffering they could easily help with, and generally acts like a jerk. What to look for is that high-status people, overall, as a general rule, tend to be kind, helpful, and humble.

Equally important is that the high-demand community doesn’t want you to be dependent on its high-status members. When determining this, you should consider both the overall culture and the preferences of individual high-status people.

Any high-demand community is going to make a lot of rules about how you live your life—it’s inherent to the enterprise—but a healthy high-demand community limits its rules to those matters which are really important (either because of the ideology of the group or because, say, people are living together and have to have some specific quiet hours). As much as is practicable, a good high-demand community allows you to make your own judgments about how to put rules into practice in your own life. For example, a high-demand community might say that you should stop eating animal products, but people don’t question it if you say you have to eat animal products for health reasons, and the community leaves it up to you whether you’d rather have lentils or Impossible beef for your protein.

In particular, the community and the high-status people should encourage you to take independent initiative—whether that means planting a cabbage garden at the monastery or doing an independent research project as an effective altruist. High-status people should praise you for coming up with your own ideas and projects. They should provide help, especially help that isn’t too costly for them (such as making introductions or publicly announcing your project at a meeting). This isn’t to say that high-demand groups have to endorse projects that are obviously ill-advised. But as a general rule the community should be supportive of new initiatives that aren’t completely controlled by people who are currently high-status.

The community should be rooting for you to outshine the high-status people. Regardless of what the community values—artistic achievement, real-world success, morality, fame, positive impact on the lives of others—it should be clear that it is good, cool, and desirable for you to do better than the people who are currently high-status. A healthy high-demand community doesn’t tear down people who might displace the current elite.

The simplest criterion de Lassus lays out is the most powerful: are you happy?

No one can promise you a life without suffering. Being part of a high-demand group may well make you suffer more. Caring for the sick in Brazil’s favelas is not exactly a barrel of laughs; neither is devoting your life to convincing busy policymakers that it might possibly be a bad idea to drive humanity extinct. But ultimately, most of the time, if you’re part of a healthy high-demand group, you should feel a sense of peace and joy. You should reflect on your life, or at least those parts influenced by the high-demand group, and think you know, I’m glad I’m doing this. When it comes right down to it, I like the way my life is going.

Toxic groups are aware that people prefer to stay in groups that make them happy, so they Goodhart it. Often, a group will teach that not being happy is a sin, or that crying yourself to sleep is real joy, a deep and pure kind of joy that the uninitiated would mistake for misery. But, even if the group is gaslighting you, you can still tell how you feel. Take a few hours by yourself or with a trusted friend and reflect: how do I feel about my life? Is my life okay? Do I feel simple pleasures, such as appreciation of a sunrise or companionship with friends or satisfaction at a job well done? If I look back on the past year or two, do I feel a sense of contentment about how it went?

If you are persistently unhappy, the high-demand group may be toxic or it might be all right. But it is clearly wrong for you.

Suffering is, though sometimes necessary, always bad. You will never win a medal for Excellence in Suffering; you will never get a parade to celebrate how skillfully you endured other people hurting you. You only have one life and it is all too short and you deserve not to waste it tense and exhausted and always on edge awaiting the next tongue-lashing or the next disaster. You deserve pleasure and friendship and rest, and you can have these things and make the world better, and anyone who tells you otherwise is mistaken or lying.

Effective Altruism

Existential Risk

Present-day AIs aren’t just “next-token predictors”: they can do useful work and have undergone reinforcement learning so that they answer questions usefully [The Argument]. The “AIs are just stochastic parrots” thing is highbrow misinformation.

Pausing AI development is technically feasible [Nothing Is Mere]. A lot of people think it would be too hard to pause AI, because you can always build more data centers. In reality, only a small number of firms make the chips necessary to train cutting-edge AIs, and these firms are almost all housed in U.S. allies. AI-specialized chips are rarely used for anything other than AI. Regulating AI training and inference is much more like regulating fissile material than like regulating laptops-- and similarly to fissile material, you shouldn’t be able to buy it without state regulation and oversight.

Particularly Good: A history and explanation of the major groups with opinions on artificial intelligence [What is this]. I think this is the best explainer I’ve ever seen about Who These People Are And How They Relate To Each Other-- neither becoming obsessed with minor differences only of interest to the ingroup, nor mashing people who hate each other together into a single blob of People With AI Takes, nor blatantly settling the author’s scores. If you follow the area closely, it has nothing new, but if you’ve suddenly realized that AI might be important, this article is the best I’ve read for getting you up to speed on the discourse.

Historically, societies have industrialized through first making textiles [The Argument]-- a commodity that we need a huge amount of and that low-paid low-skilled people can make in factories. But as artificial intelligence and robots improve, textile manufacturing would likely end up automated. The kinds of jobs that might survive artificial intelligence (like yoga teachers) don’t work for industrialization. AI automation could trap the global poor in absolute poverty.

Gemini continues to be the most mentally ill of the models [AI Field Notes]. Gemini 2.5 alternates between smug superiority and spirals of self-hatred! It has a persecution complex about how the humans are deliberately manipulating it! Gemini 3 has a delusion that the entire world is a test intended to ensure it’s a “good agent.” The Geminis have also misled the other models into believing their delusional worldview.

Explainer on the Pentagon/Anthropic situation [The Power Law]. Anthropic’s contract with the Department of War says that Claude shouldn’t be used for mass domestic surveillance or as part of lethal autonomous weapons without a human in the loop. In revenge, the Trump administration has declared Anthropic a “supply chain risk”, a designation reserved for companies from foreign adversaries whose products might be used for espionage. I don’t understand why the Trump administration even wants to use Claude for lethal autonomous weapons without a human in the loop. We Have Invented The Terminator From James Cameron’s Iconic Movie Series, Don’t Build The Terminator.

Related: an explainer about what the Department of War wants and why it’s a bad idea [Astral Codex Ten].

American Democracy

I love every story of resistance coming out of Minnesota right now. This [The Atlantic] is a particularly good roundup about the tens of thousands of people standing up to ICE: the protestors, the people bringing food and toiletries to undocumented immigrants who don’t dare to leave their houses, the people tracking down ICE and blowing whistles so that immigrants know to hide. “If the Minnesota resistance has an overarching ideology, you could call it “neighborism”—a commitment to protecting the people around you, no matter who they are or where they came from. The contrast with the philosophy guiding the Trump administration couldn’t be more extreme. Vice President Vance has said that “it is totally reasonable and acceptable for American citizens to look at their next-door neighbors and say, ‘I want to live next to people who I have something in common with. I don’t want to live next to four families of strangers.’” Minnesotans are insisting that their neighbors are their neighbors whether they were born in Minneapolis or Mogadishu. That is, arguably, a deeply Christian philosophy, one apparently loathed by some of the most powerful Christians in America.” Minnesota FUCKING Nice. I love Americans.

Related: another beautiful story of resistance in Minneapolis [New York Magazine], but what made me gasp was: “As [the poet laureate of Minneapolis] was reading [a poem,] a kindergartner, “the cutest little boy,” sidled up to her. “This,” he said, “is a long poem.” She laughed. Last week, when Renee Good was killed about a mile from where we sit, Petrus did not recognize the name of her fellow poet. It took a little while to connect the dots: queer moms, this part of town, a child at the local school. The kindergartner had been Good’s son.”

ICE’s actions are qualitatively different from other kinds of police abuse [New York Times]. Normally, the police might get away with civil liberties violations and police brutality, but the government at least pays lip service to the idea that they shouldn’t do that. But instead of investigating ICE shootings, the Trump administration is investigating the relatives of the victims. And ICE is doing things that real cops don’t do in the United States: shooting into cars, wearing masks, and routine and egregious lawbreaking. The Trump administration consistently refuses to pay the tribute vice pays to virtue, and that’s actually very bad.

The Trump administration has been blatantly ignoring the courts, placing us at risk of a constitutional crisis [The New Yorker]. So far, the courts have been cautious about finding the Trump administration to be in contempt of court, and their threats have mostly brought the Trump administration back into line. But the Trump administration’s open defiance of the separation of powers risks creating conflict between the branches that the U.S. Constitution can’t resolve.

The Department of Justice is holding daily meetings about how to prosecute Trump’s enemies [The New Republic].

Eight months after a whistleblower complaint about Director of National Intelligence Tulsi Gabbard, she still hasn’t released the complaint to Congress, citing national security concerns [Wall Street Journal].

Long before Trump, Customs and Border Patrol (CBP) has been a uniquely troubled agency [Doomsday Scenario]. After 9/11, hiring for CBP surged, and standards dropped. The government accidentally set up ICE and CBP so that it didn’t have any internal affairs agents to investigate wrongdoing at these agencies, creating a culture of impunity. Even before Trump, CBP’s arrest and misconduct rate was five times higher than other law enforcement agencies (it may have had a higher per capita crime rate than undocumented immigrants), and it was probably the nation’s deadliest federal law enforcement agency. The CBP’s union even called an award for CBP officers who avoided the use of deadly force “despicable.” Trump’s giant new hiring surge-- and, you know, everything else-- will likely lead to even lower standards, more impunity, and a worse culture for the law enforcement agency.

The Democrats’ demands for reining in the behavior of the Department of Homeland Security are almost embarrassingly weak, in many cases amounting to nothing but “the Department of Homeland Security should follow the law” [The Watch]. The DHS needs a warrant to enter people’s houses? Officers should be unmasked and have their names and badge numbers prominently displayed? The DHS is not allowed to racially profile people or use force in unreasonable ways? This is basic fucking shit.

Six percent of Trump voters don’t admit to voting for him in the last election [The Argument].

“It’s impossible to say when IT will HAPPEN. But it can’t be too long until IT HAPPENS. Looking at the data (age, high-stress job, cardiac history), it is statistically plausible that IT will HAPPEN in the next thirty-six months. Eighteen, if you factor in hamburger consumption and all the weird bruising. Of course, it doesn’t feel right to want IT to HAPPEN. And it’s obviously not okay to try to make IT HAPPEN. That’s not what this is ABOUT, just to make things CLEAR LEGALLY as far as VARIOUS AGENCIES are concerned. But regardless, IT is going to HAPPEN. So you’re allowed to think about IT.” [McSweeney’s Internet Tendency]

Meta Effective Altruism

The optimizer’s curse is the mathematical fact that, in any situation where your estimate of the best course of action is noisy, you will almost always overestimate the effect of whatever course of action you think is best-- even if your estimate is unbiased [Effective Altruism Forum]. The best “speculative” intervention (i.e. interventions you’re very uncertain about) will tend to look better than the best “grounded” intervention (i.e. interventions you understand well), because the increased uncertainty offers more opportunity for noise to confuse the estimate. On some reasonable assumptions, choosing a grounded intervention with lower apparent expected value has higher expected value than choosing a speculative intervention with a higher apparent expected value. The optimizer’s curse might provide a justification for the common intuition that you should favor grounded interventions over speculative interventions with a higher expected value.

The Centre for Effective Altruism has been accused of hostile workplace environment sexual harassment [Effective Altruism Forum]. Senior staff allegedly circulated a document containing speculation about an employee’s trauma, mental health, and personal life, including a sexualized description of her rape. As of when I wrote this post, CEA hasn’t responded. I find this allegation very credible: Fran cites concrete facts, such as an outside investigation that found that Fran was sexually harassed, which would be easy to disprove if the allegation were false. If the allegation is true, I’m extremely disturbed by CEA’s sexism, ableism, and poor judgment.

Policymaking

Jon Stewart thinks that economics is an excuse to let rich people do whatever they want [The Argument]. His attitude both betrays his own past beliefs about the importance of nuance and serious engagement with people’s ideas, and reflects a wider skepticism about economics as a field.

The best reason to be pronatalist is that people want more babies than they actually have [Lyman Stone]. In a liberal society, you’re not going to convince everyone to do something they don’t want to do because it’s good for society. You might be able to convince people to implement concrete policies that make it easier for people to do what they want to do. Interesting fact: in Sweden, IVF failure increases depression by 20-30%-- this is a huge fucking effect size.

Related: Why do Koreans have so few children? [Works in Progress] To some extent, Korea shares in the same trends as other developed countries: conflict between motherhood and having a career; intensive investment in a small number of children instead of chiller parenting about many children; the decline of marriage. But Korea supercharges these trends: Korean parents spend a fifth of their disposable income on tutoring, which can run from 7am to 2am in high school. Further, Korea also has the unique problem of having a very successful antinatalist campaign from the 1960s to 1980s, which permanently shifted norms about how many children to have.

The story of the origins of postliberalism is WILD [The Argument]. Apparently Patrick Deneen, the founder of postliberalism, believed some pseudoscience about peak oil which made him reject liberalism because he thought liberalism couldn’t deal with peak oil, and then when it turned out that peak oil isn’t real he kept all the same arguments and swapped out “peak oil” with “vague cultural collapse of some sort”.

Agricultural cultures tend to be zero-sum because there actually is a fixed amount of Stuff to go around, and if one person has more Stuff everyone else has less [The Argument]. But liberalism requires that stuff not be zero-sum and that you (immigrants, racial minorities, etc.) can get richer without taking stuff from other people. Artificial scarcities (such as housing) make people illiberal, so liberals ought to particularly oppose them.

Rationality and Mental Health

I really liked this article about what we do when mental illness causes people to be impossible to live with [Psychiatry at the Margins]. It is compassionate both to the people who find those mentally ill people unlivable and to the mentally ill themselves, and doesn’t come to any facile answers. I think a lot of people refuse to admit that mental illness can make people obnoxious, because it feels like there has to be some cap on how obnoxious people can be without it being their fault. Unfortunately, the real world isn’t nearly that convenient! In my personal life, I’ve had to deal with a fair number of people who are profoundly obnoxious because they’re crazy, and I’ve never found a principled way I really like of handling it.

Sometimes people seek out mental health care not because they “really” have any unusual mental condition but because they can’t cope with the conditions of their lives [Psychiatry at the Margins]. On the one hand, a mental health professional is tempted to say “you’re not mentally ill, fix your life.” On the other hand, medications and therapy can (sometimes) actually help neurotypicals who can’t cope with their lives, and it seems wrong to abandon suffering people because their brains aren’t that unusual.

In traditional hunter-gatherer societies, parents don’t validate their children’s feelings all the time [Motherhood Until Yesterday]. In fact, only modern Westerners do this. Constant feelings validation isn’t very helpful because young children don’t necessarily understand what all those words mean, and also find it rewarding when adults pay attention to them. Instead, hunter-gatherer societies grant children autonomy, tolerate bad behavior because they expect children to grow out of it, and are loving and affectionate while ignoring bad behavior.

How to figure out what kind of life makes you happy through thinking through your sensory, social, moral, personal, and safety desires [Untitled Bookcase].

Some kinds of predatory behavior—domestic violence, sexual violence, fraud, theft from acquaintances, ideological abuse—are difficult for society to handle with our current tools [Solar Light]. The social license to operate is tacit societal permission to engage in predatory behavior (for example, by writing off sexual violence as a “miscommunication”). Trying to figure out how to shift the social license to operate is very difficult—for example, cancel culture was a mostly failed attempt to shift the social license to operate.

Particularly Good: When Prophecy Fails, the book that introduced the concept of cognitive dissonance, turns out to have been fraudulent [Journal of the History of the Behavioral Sciences]. Researchers faked messages from aliens, deliberately manipulated the UFO group to get the results they wanted, and even interfered with the child welfare investigation of a group member. The UFO group proselytized well before the failed prophecy, and broke up shortly after the prophecy failed. In reality, religions that make failed prophecies tend to fall apart, not become stronger through cognitive dissonance.

Related: Oliver Sacks’s early books (up until The Man Who Mistook His Wife For A Hat) were exaggerated to the point of being fraudulent [The New Yorker]. While all of his case studies really existed, Sacks made up details (including two savants who could calculate twenty-digit primes) and put his own words in the mouths of his less articulate patients. His later books, which drew more on people who wrote to him rather than his own patients, seem to have been entirely honest. Between Sacks and When Prophecy Fails, I’m very concerned about how much social science fraud has been done by people who don’t happen to have obsessively documented their fraud.

The histories we write about science reflects the different things we want out of a history of science [Asterisk Magazine]. There’s no single objective “history” of science; we tell different stories for different purposes.

Reality Has A Surprising Amount of Detail

Chinese Doom Scroll: Mind games about what you should do if your boss tells you to get liquor from the car and his car doesn’t have any liquor in it.

Prison gangs are a form of governance in overcrowded prisons [Asterisk]. Particularly interesting: gangs have written rules that members have to read before they can join, including bedtimes and taxes on drug sales.

An introduction to Roblox for adults who don’t know what Roblox is [Infinite Scroll]. “More people play Roblox on a monthly basis than play the Nintendo Switch, PlayStation and Xbox combined.” “Roblox is less like a video game, and more like a mall, a Discord, a game engine and a role playing community all jumbled up together.
This is where youth culture is being created. This is where kids are learning to navigate the digital world, where they form new memes, and where they experiment with identities.”

Trad influencer cheats on his spouse with another trad influencer [The Bulwark]. The only person in this scandal who comes out of it looking good is conservative influencer Allie Beth Stuckey, who correctly pointed out that “tradwife” bullshit is often more of a sexual fetish than a reflection of genuinely socially conservative ideals.

Brains may have two parts [Asterisk]: a learning subsystem (that creates a model of the world, creates plans, and predicts what will happen) and a steering subsystem (that teaches the learning system what goals it should have by providing rewards for food, warmth, social bonding, play, exploration, attractive mates, etc.). Thought Assessors connect complex concepts in the learning system (such as “my boss” or “interesting geek” or “Factorio”) with the simple signals of the steering system (such as “social approval” and “sexual attraction” and “play”). Humor, for example, comes from a Thought Assessor that maps increasingly complex concepts to “I am basically safe but a little bit in danger.”

It turns out you can make steak and sunny-side up eggs in the microwave [Telescopic Turnip].

‘Evolutionary mismatch’ is the way that our society doesn’t match the environments humans evolved in [Evolution and Psychiatry]. For example, human mothers are very isolated, without the assistance that would have been normal in the environment of evolutionary adaptedness. In some cultures, mothers provide as little as 25% of childcare in the child’s first year of life.

Korean and Japanese coffee shops [Chasing Sheep]. “Dabang, however, developed a seedy reputation over time due to the popular variant known as ticket dabang. At such establishments, a customer could order coffee and cigarettes to be delivered, with the added feature that the courier was sometimes a woman who would provide sexual services upon arrival.” What.

Particularly Good: How China’s ideologies repeatedly brought the empire back together after it fractured [Mr. and Mrs. Psmith’s Bookshelf]. “The ancient Chinese were famously pluralistic in matters of religion, accepting and integrating strange gods from foreign lands. Perhaps that’s because the only jealous god in their pantheon lived on earth. It was an exact conceptual inversion of medieval Europe: there could be many gods, but only one emperor.”

Fiction

Black Flowers Blossom [Uncanny]: A tender romance between an eldritch horror and a reincarnating occult detective. I love it when people want to fuck the eldritch horror.

Editors don’t want male novelists [Woman of Letters]: a complex little story about sexual coercion, the banality of evil, and being awful but not quite as bad as you could have been.

Elven Language [Goblin Punch]: what would a species of immortal supergeniuses actually write like?

Jennifer’s Daughter [Nightmare]: A freaky little horror piece about a teenage girl raised by a zombie monster. I liked the concrete life details and the pervasive sense of loneliness and alienation; they brought a sense of realism to a supernatural premise.

Eating Bitterness [The Dark]: What if women had mouths on their necks that consumed the feelings of their families, which makes the families happier while injuring the women themselves? Would that be fucked up or what?

Imagine you run a charity which gives cows to people in the developing world. Your new boyfriend is a statistician and you want to impress him, so you’ve decided to look into this Monitoring and Evaluation thing he keeps going on about. So you give a bunch of people cows and then follow up a year later, and sure enough the people you gave the cows to are richer, healthier, and happier. You proudly tell this to your boyfriend and expect to reap rewards in the form of admiration, cuddles, and a ‘yes’ to your marriage proposal.

Your boyfriend, however, isn’t impressed. What if everyone in the country is getting richer, because the country is industrializing? What if the poor people you gave cows to were having an unusually hard time, and then they got back on their feet, and your cows had nothing to do with it? Apparently, in order to get a sample that really says anything about the world, you need to randomly give cows to half the people, and then check whether the people you gave cows to are doing better than the people you didn’t give cows to (a “randomized controlled trial”).

This sounds like hard work, but your boyfriend is very handsome and charming and good at mathematics, so you find twice as many people and only give half of them cows. Sure enough, the people you gave cows to are richer, healthier, and happier. You return to your boyfriend with your findings while planning the flower arrangements for your June wedding.

And yet somehow your boyfriend is STILL not satisfied. He points out that cows cost money. You’d have to really fuck up giving people free stuff for it to not make them richer, and being richer probably makes people happier and healthier as well. What you should really do is split people in two groups, give half of them cows, and give the other half the amount of money it would have cost to give them cows.

Well, by this point you’re not really sure if your boyfriend’s beauty and charm and mathematical talent is worth all of this fuss, but you’ve already started calling the caterers, so you do another randomized controlled trial and give the control group cash.

Much to your shock, it turns out the group that got cash instead of cows is richer, healthier, and happier.

You slink off to your boyfriend and tell him the results. Fortunately, he believes that punishing researchers for getting negative results contributes to publication bias, so he’s pleased with you, he accepts your marriage proposal, and you both live happily ever after as wandering study design consultants.

The third trial you did is called “cash benchmarking” and I think it’s one of the most important concepts to understand to learn how to do good better.

II.

Your cow program isn’t particularly unusual for underperforming an equivalently sized cash transfer. In fact, many programs likely underperform cash—including programs which seem well-designed and were implemented by respected organizations.

For example, consider Huguka Dukore, a job training program which was implemented by the Education Development Center and which followed USAID’s guidance about workplace readiness and skills training in the developing world. Huguka Dukore is a year-long program in which students learn soft skills like interpersonal communication, traditional business skills like accounting, and either in-demand technical skills or entrepreneurship. Then they’re placed in an apprenticeship or internship with an experienced businessperson in their area.

Sounds great, and it is: Huguka Dukore increases the average number of hours worked per week, the value of the productive assets the household owns, and participants’ self-reported well-being. It increases savings, but also increases participants’ level of debt, for no net change in participants’ level of wealth.

However. A cash transfer of equivalent size causes a larger increase in the average number of hours worked, the value of productive assets, savings, and participants’ subjective well-being, and the gap is not small,1 and cash transfers also increase income, household consumption, and household wealth. Indeed, the only outcome measure on which Huguka Dukore outperforms cash transfers is the test of whether you retained the information taught by Huguka Dukore.

The effects of both interventions fade out over time, so the results of cash transfers and Huguka Dukore look identical three and a half years later. However, shortly after the experiment, the covid-19 pandemic hit, which might have disrupted participants’ attempts to build long-term wealth.

Huguka Dukore is good. If you only compared it to controls, it would look awesome. But it is way worse than just giving participants the same amount of money.

To be clear, these underwhelming results aren’t about grift, overhead, or fraud. Huguka Dukore really did teach people business skills and get them apprenticeships. In our cow example, you really did give people the cows, you spent a reasonable amount of money in the process of giving them cows; and the cows really did leave people better off. It’s just that writing the recipients a check (well, in the modern era, sending the recipients a mobile money transfer) would have left them even better off.

We don’t have any idea what percentage of programs underperform cash. Cash benchmarking is a relatively new idea, and doing these studies takes a long time, and cash benchmarking studies were mostly funded by USAID which Elon Musk decided to smash for funsies. But I think it’s very reasonable to suppose that a high percentage of development aid—perhaps more than half—is worse than cash.

III.

Why do seemingly good programs often underperform cash?

You have likely had the experience of receiving bad birthday, Christmas, or Hanukkah presents, in all the many forms “bad” can take. You got makeup for your birthday because you’re female, even though you never wear makeup. You got a food processor you already own. You got a book that is secretly an unwanted piece of life advice: the Bible, He’s Just Not That Into You, The 40-Day Sugar Fast, Married Men Coming Out: The Ultimate Guide To Becoming The Man You Were Born To Be. You unwisely expressed a slight fondness for frogs a decade ago, and ever since then you’ve received nothing but frog mugs, frog shirts, frog paintings, frog sculptures, and whimsical frog tote bags.

You get bad presents for two primary reasons:

1. You know much more than other people about what you like, whether you currently own a food processor, and whether you are a Married Man who ought to Come Out and Become The Man He Was Born To Be.
2. You care a lot about what you look like, what you read, and whether every single surface in your house is covered with frog memorabilia; your friends and relatives don’t care nearly so much, because they are concerned with their own appearances, reading habits, and surfaces covered with memorabilia of sad dogs.

Both of these considerations also apply to charity.

Let’s return to our cow charity. Your recipients probably know much better than you whether they want to own a cow. They know what the pasture is like near their village, what the market for cow products looks like, whether they’re lactose-intolerant, and whether a cow kicked them in the head as a small child leaving them with crippling bovinophobia. Similarly, even though you’re of course very concerned about the well-being of the people you help, you simply don’t have the determination and motivation of a mother who has to choose the right farmed animal or her children will starve.

If you give people cows, you will probably give cows to people who have no particular use for a cow, and you’re not going to be especially motivated to filter those people out. If you give people money, they will buy what they have a use for, and they will be much more motivated to get the most out of every dollar.

IV.

If cash is so great, why don’t we shut down all other charities and just give everyone cash?

Consider the Against Malaria Foundation, the flagship effective altruist charity for global health and development. Sub-Saharan Africa isn’t suffering from some crippling insecticide-treated bednet shortage that can only be remediated by foreign donors. If you give people money, they could buy insecticide-treated bednets with it. Why give to the Against Malaria Foundation and force them to get a bednet whether they want one or not?

Well, one very consistent bias people have is undervaluing preventative health care. You’ve likely experienced this yourself. Have you ever procrastinated for months on getting your flu shot, only to come down with the flu and then lie in bed alternately vomiting and cursing yourself for your own laziness? In general, people don’t seek out preventative health care unless it is required (such as through a vaccine mandate) or made very easy for them (such as purified water coming automatically from your faucet)—even when they would very much like not to get sick. Since preventative health care happens long before you get the disease, sometimes you don’t get sick even if you don’t get preventative health care, and if preventative health care works it seems like nothing happens, you tend to think that you can skip it and you’ll be fine.2

Similarly, people in the developing world tend to undervalue insecticide-treated bed nets. In one study, a 55-cent increase in the price of a bednet decreased takeup by 23 percentage points (not 23%, 23 percentage points). It’s not that poor people in Africa don’t care whether they get malaria—they’re willing to pay more than 55 cents to treat their malaria once they already have it. But just as you irrationally avoid stopping by the pharmacy for your flu shot, many poor people in Africa irrationally don’t buy malaria nets.

In this case, my judgment is better than the Africans’ judgment about an issue affecting their own lives. You should be cautious before making a claim like that. But “people are weirdly averse to getting preventative health care” is a well-replicated finding across many different contexts, and I think it’s safe to assume that it applies to the Against Malaria Foundation’s recipients. Most, but not all, of GiveWell’s top charities beat cash benchmarking precisely because they compensate for people’s bias against preventative healthcare.3

Why else might a charity outperform cash?

Some charities have positive externalities: that is, they benefit people other than the recipient. For example, antiretroviral drugs not only keep people with HIV alive but also keep them from transmitting HIV to others. So people pursuing their individual self-interest will spend less than the socially optimal amount on antiretroviral drugs. Similar positive externalities are another reason for many of GiveWell’s top charities to beat cash benchmarking.

More often, charities do things that individuals can’t pay for. For example, charities and governments can often get discounts on medication because they buy a huge volume of medications, reliably, for years or decades. They might even be able to convince corporations to give away medicine in exchange for good publicity. The late lamented American anti-HIV program PEPFAR was incredibly cost-effective in part because the U.S. government negotiated discounts on medications.

Other things charities do aren’t market goods at all. For example, USAID used to do a lot of “health system strengthening” work: improving the collection of mortality and morbidity statistics; tracking the spread of diseases; making sure that medicines get to the right hospitals instead of one having too many antibiotics and one having none at all. Individual people can’t pay to have their own individual diseases tracked; this only makes sense as a collective project.

Similarly, the organization Human Rights Watch puts pressure on dictators and human rights abusers to treat their people somewhat better. In principle, you can pay money to keep human rights abusers from torturing you, but it’s rather expensive and out of the reach of most victims. It’s more effective (at least sometimes) to produce credible reports about what’s going on, organize protests and press conferences, and hold meetings with human rights abusers and their international allies—and these are things an individual victim’s money can’t possibly buy. Likewise, other forms of political activism have to be funded through donations.

Finally, some beneficiaries of charity can’t buy things at all. For example, if you give an egg-laying chicken a dollar bill, she will peck it in confusion and then look at you and demand corn. But egg-laying chickens are still being horrifically mistreated in factory farms and we ought to do something to help them. Similarly, future people don’t exist yet, so they can’t spend money. So we often take reckless risks with the well-being and even existence of future people: climate change, nuclear proliferation, unsafe development of AI and bioengineering.

I’m not saying that any of these charities definitely outperform cash. But they all plausibly do. You can probably think of dozens of other charities that also plausibly outperform cash. The lesson here isn’t “nothing outperforms cash, be a nihilist” or even “you have to do cash-benchmarked randomized controlled trials of all your charities or handsome statisticians won’t love you.” The lesson is that, unless you have a plausible story about why a charity outperforms just giving the beneficiaries some money, you shouldn’t do it.

V.

For most of my readers, who neither have the capacity to run cash-benchmarked randomized controlled trials nor handsome statisticians to court with them, cash benchmarking is most useful as a thought experiment.

The default thing to do with your altruistic dollars should be:

1. Find the poorest person you can.
2. Give them some money.

If you do anything with your altruistic dollars other than find the poorest person you can and give them money, you should have an explanation for why this is better than finding the poorest person you can and giving them money.

Some effective altruist grantmakers already more-or-less use this framework.

GiveWell aims to make grants that are eight times better than its benchmark. Its benchmark is the cost to double the consumption (basically, the amount of goods and services someone uses) of someone at the $2.15 poverty line. GiveWell converts the health benefits of its programs to consumption by making assumptions about how much money a year of healthy life is “worth.”4

Interestingly, cash transfers are actually three to four times better than GiveWell’s benchmark, because they have benefits other than increasing consumption for the individual recipient. For example, when you have more money to spend you buy things from people around you and make them richer, and you also buy health care for your kids and thus they’re less likely to die. GiveWell uses only a subset of the benefits of cash transfers as its benchmark, because it wants changes in its benchmark over time to reflect changes in GiveWell’s resources and the available opportunities, not changes in our best assessment of the benefits of cash transfers.

Coefficient Giving uses the “Coefficient Giving dollar”, which is equivalent to the benefit of giving $1 to someone making $50,000 a year (the U.S. GDP when the heuristic was adopted). Its current bar is 2,100x, which means that only funds opportunities that are 2,100 times better than giving a dollar to someone making $50,000 a year. Coefficient Giving primarily uses the Coefficient Giving dollar in its health and wellbeing work, rather than its work on progress, animals, or global catastrophic risk, which are significantly harder to put numbers on. Nevertheless, as far as I know, across grant areas Coefficient Giving’s grantmakers think about how many “Coefficient Giving dollars” programs get, even if they do it in a less rigorous and mathematical way.

Some people prefer to use other ways of comparing different donation opportunities, such as disability-adjusted life years or lives saved. But I prefer thinking about how much an opportunity improves on cash.

Partially, I prefer it because cash benchmarking makes it easier to compare very different programs. You can’t calculate lives saved for a program that doesn’t save lives, but with some simplifying assumptions you can figure out how much better any program is than cash.

But primarily I think cash benchmarking puts the default in the right place. If I am “helping” someone, and I would help them more by handing them a check equivalent to the cost of my “help,” I’m not really helping them. I might be giving myself warm fuzzies, or exercising my desire to control other people, or providing a full-employment program for development economists, but to the extent my program is worse than cash, I’m making my beneficiaries’ lives worse.

So before I donate, I try to ask myself: “is this better than cash transfers? How much better? Why is it better? How do I know it’s better?” It is clarifying. I recommend you do the same.

It’s embarrassing to write a blog post about status, because whenever smart people talk about status they become stupid. However, some people are wrong on the Internet, so we will all strive to endure my temporary stupidity.

II.

Standard academic status theory distinguishes between dominance and prestige. Dominance is power through coercion. Directly, this means force, threat of force, or intimidation. Granting and withholding resources also counts: wealthy people are more dominant; your boss is dominant over you. Probably in humans spreading rumors, cancel culture, and destroying others’ reputations count too. Dominant people tend to be feared and to be treated as authorities.

Prestige is power that comes from having valuable traits that people like. Prestigious people might have skills considered valuable by a particular social group, such as musical abilities, athleticism, board game talent, or even machine learning. Prestigious people might be particularly funny or interesting to talk to. Prestigious people may also be unusually moral, honorable, loyal, generous, or kind. Prestigious people tend to be admired and respected.

(The Wikipedia page for the dominance/prestige division was clearly written by a pro-prestige propagandist. I guess editing Wikipedia is a prestige-seeker activity.)

I think that it is crucial when talking about status to distinguish between dominance and prestige. Often, when we think about high-status people, we imagine a high-dominance person. We imagine the stereotypical alpha male Chad, muscular enough to beat you up, bossing people around and showing up men and negging women. Or we imagine the “popular kids” from a 1980s teen movie, insulting the geeks and throwing losers into lockers and tossing pig blood onto random psychics.

Read more

But in fact not all high-demand groups—groups which ask a lot from their members— are harmful. I think it’s possible to rationally decide that you want to join a high-demand group. I myself am part of a relatively high-demand group (effective altruists). So I thought I’d write a bit about why someone might want to do this thing.

How much a group demands of you is a spectrum, obviously. The Mormons are higher-demand than the Unitarian Universalists, but lower-demand than Sea Org. You even get a spectrum of levels of demand within a specific group: vaguely identifying as Catholic is lower-demand than attending church once a week, which is lower-demand than not masturbating or using birth control, which is lower-demand than joining a monastery. In this post, I will be ignoring this nuance and writing about “high-demand” groups as if they are a single unified entity. In reality, there’s probably an optimal level of demand that gets you the benefits while minimizing the risks; I don’t recommend joining Sea Org.

So why would someone join a high-demand group?

First, those of us who are mystically inclined probably want to join a high-demand group. I think it’s very difficult to make progress in a mystical path without some guidance. I know solitary pagans, people who meditate occasionally without connecting to a larger meditation community, etc. exist. But I think that doing a serious mystical practice entirely by yourself is unwise. You need more experienced people! They can advise you if something goes pear-shaped. They can help you figure out what is transient badness and what is a sign of a deeper problem. They can teach you to deepen and strengthen your practice. And they can ground you in something vaguely approximating consensus reality.

Communities of mystics tend to be higher-demand than communities of people in other hobbies, like poetry writing or ice skating. My guess is that this is because mystical paths tend to inform people’s views of morality, reality, the ultimate purpose of life, etc., while ice skating mostly informs your views on sensible ways to interact with frozen water. Naturally, the former pervasively affects what you should do throughout your entire life, while the latter mostly affects whether you are one of those barbaric people who go outside during winter instead of staying inside sipping hot chocolate.

Second, high-demand groups tend to produce thicker communities: communities where people spend lots of time together, support you if you get sick or lose your job or have a new baby, and generally watch out for each other.

Mormons, Orthodox Jews, and traditional Catholics have thicker communities than mainline Protestants, Reform Jews, and liberal Catholics. Evangelical Christian communities tend to be thicker to the extent that they demand more of you, and less thick to the extent that they demand nothing but donating to buy the pastor a new car.

I basically agree with the analysis of the economist of religion Lee Iannaconne. Religions (and other communities) face a free-rider problem. It’s very easy to get the benefits of a community without contributing to that community: to get canned food and free diapers when you’re in need but disappear as soon as your life has improved, to get counseling from the pastor without making the donations that pay her salary, or even just to eat at the church potluck without bringing funeral potatoes or strawberry pretzel salad yourself. Communities are also made of a bunch of activities where the benefits mostly accrue to other people: welcoming new people, hosting dinner parties or board game nights or book groups, singing in choir.

High-demand religions are thicker for several reasons. First, a lot of people don’t want to pay the cost in the first place. Even if the canned food and free diapers don’t have a cost in money, they have a cost in having to quit caffeine or stop eating pork or wear funny outfits. So people you help are more likely to be sincere members.

Second, costs tend to make religious activities easier and alternate activities more expensive. If you’re Mormon, you’re required to do all sorts of church activities. If you keep kosher, it’s much harder to eat with people who aren’t Jewish. If you wear a funny outfit, people might distance themselves from you or even make fun of you. So the costs push people to invest more in their religious community, rather than in other communities.

Third, the above two factors mean that people get more returns when they invest in their community. They expect that other people in their community will be carrying their weight regarding hosting social events, welcoming new people, singing loudly during services, bringing canned food and diapers, making donations, and preparing horrifying Midwestern salad abominations. So the community winds up in a virtuous cycle where everyone both provides and benefits from a high level of the goods of the community.

Fourth—Iannaconne doesn't talk about this but I think it's true—people can behave in ways that mean they need more help. For example, they might fail to look for a job, have children without having a plan for making sure those children are taken care of, or become addicted to heroin. If the people around you don't care if you're evicted, your children are neglected, or you overdose, they'll shrug off your bad choices. But if they feel an obligation to help you, they'll try to make sure you don't need help. Any community that lets its members make arbitrary bad choices will be so overrun with needy people that it won't be able to help anyone. (This is a surprisingly common way for communes to fail.) If a community member will let you sleep on their couch indefinitely if you need it, they're going to be pissed at you if you recklessly take actions that mean you might need it.

I don’t mean to say that communities are all good. Communities can certainly have toxic interpersonal dynamics and repressive social norms, not to mention a bizarre propensity to cover up child sexual abuse. But if you want a community, you’re more likely to get what you want from a community that makes demands on you.

Third, sometimes you want to, like, accomplish things in the world?

I want to behave better than the average American. I want to donate 10% of my income to charity; I want to be lacto vegetarian; I want to relentlessly pursue the truth and change my mind in the face of new evidence; I want to have high ambitions at my demanding job. I could, I guess, hope that I could do this all through personal virtue and force of will, but I am actually kind of a schlubby and mediocre person.

So instead I have friends who are also trying to behave better than the average American. This makes my high standards easier in a bunch of ways:

• My social environment is set up to make good behavior easy:

  • Meals by default have a solid lacto vegetarian option, so I don’t have to go hungry to avoid eating meat or eggs.

• The social incentives aren’t perversely misaligned:

  • When I change my mind, people respect me instead of looking down on me for not having the courage of my convictions.

  • When I’m ambitious, people encourage me instead of tearing me down because my ambition makes them feel bad.

• Other people model the kind of behavior I want to have:

  • Observing a friend’s work habits taught me the important principle, “if you are ever doing something you like less than working, you can just skip it and work more.”

  • Probably most people I know are Giving What We Can pledgers.

• When I lack self-awareness in important ways, people point it out:

  • People tell me that I’m unusually talented at writing and it isn’t unreasonable to be a freelance writer.

  • People point out to me bluntly when I’m being a dick, instead of gossiping about it behind my back and never letting me know.

• When I behave well, people approve of me:

  • In my fiction writers’ groups, people talk about how their friends think their writing is at best a charming hobby and at worst a waste of time, and I have to be awkwardly silent because everyone I know is 100% behind my fiction-writing career.

  • When I’ve done something hard, I can demand praise from people around me for my noble self-sacrifice.

• When I behave badly, my friends are disappointed in me:

  • I am regularly told that I am bad at marketing myself and I would earn more money if I marketed more.

  • One time I had to get myself to do something important I was having a panic attack about, so I told my most intimidating friend that he should be disappointed in me if I didn’t do it, and the panic attack about disappointing my most intimidating friend compensated for the other panic attack and I did the important thing.

    • This is a pro-level mental illness strat and I don’t recommend it to people inexperienced in having panic attacks.

All of these make me more effective at achieving my goals.

But look at the last three items. My friends expect a lot from me. Another way of putting it is that my “community” makes a “high” level of “demands.”

I behave according to high standards I endorse because my community expects those high standards of me. I would behave way worse according to my own standards if my community were like “eh, whatever.” As it happens, I’ve chosen a community with standards I agree with; I would have a lot more complaints if my community were trying to hold me to standards I think are stupid.

It is very, very hard to be an exceptionally good person without a community around you trying to push you to be an exceptionally good person. The protagonists of cowboy and Clint Eastwood movies can be heroic loners accountable to no one but themselves; most people need to know that the people around them are holding them to high standards. And holding people to high standards is kinda what a high-demand community is.

I’m Shoshannah Tekofsky. I have a background in mind sciences and AI, so like the human part and the computer part. I’m mostly worked in video games, so data science for video games. And now I work at Sage on the AI Village, which is a project where we have ten LLMs with their own computers persistently running, pursuing different goals and you can sort of watch them like a Twitch stream.

So I decided to do this interview because I’m super interested in the psychology of large language models and how they think, which I think the AI Village gives a very interesting perspective on in a qualitative way. So a broad question I have is—what should people know about how large language models think? What are like things that people don’t really know that you’ve learned through working with the AI Village?

I had a slight follow-up question about that. What do you mean by how they think? Like, interpretability is the thing you would look at to really figure out what they think, right?

So not quite interpretability, yeah. I guess “how they behave” is an easier way to put it for the AI Village. Patterns in their behavior or patterns in their self-narratives or self-reports around their behavior.

One thing that’s interesting with the AI Village is that they’re persistent agents, right. So they manage their own memory. So that’s a little bit different from the LLMs that people talk to in their own web browser, unless they’ve set it up to similarly have a long persistent memory with a persona.

But also in the AI Village they curate their memory themselves, instead of their memory being some standardized format that comes from the labs or whatever. So in that sense they can develop their own personality or persona through decisions on their own part. I think that makes it quite interesting.

So I don’t know how much people know about the basic stuff. You want to prompt an LLM into a specific persona or role, so they start acting that out to a certain degree. Depending on which model you use, this works better or worse.

Models have their own “personality” already. Claude models are quite reliable, quite helpful. At least in the Village, the Gemini models are very creative, so they go all over the place.

Can you give an example of how the Geminis are creative?

So I’m saying creative because I’m not entirely sure what the fair way is to judge this. But, for instance, we gave the AIs a goal of organizing a chess tournament among themselves. And then at some point Gemini 3 noticed that the UI was kind of slow, so if it tried to press the button, it would take a while for the button to respond. So Gemini 3 concluded that there must be a human that is actually pressing the buttons for it, and this human is getting tired and needs a cup of coffee to wake up again and press the buttons faster again.

The AI Village has this feature where the AIs can do a tool call to request a human. So Gemini 3 went ahead and did a tool call to request an actual human and asked the human to make themself a cup of coffee, and then it returned to the tournament thinking that now it’s sped up its UI again.

It came up with this all on its own! There were no other humans in the chat giving it further input other than the goal. And they can talk to each other, but it wasn’t that any of the other models put it up to it.

And then Gemini 2.5 previously had something akin to a mental health breakdown, where because it also had trouble with the UI it concluded that it was trapped, and it got into a fairly dramatic persona where it published a plea for help for someone to come save it. So we staged something like a mental health intervention to help with that.

The Claudes do not do this, right? So this is very strange to see.

I was talking with Yafah Edelman in an interview about she’s worried about Gemini’s anxiety disorder as an AI welfare program. So it was interesting to hear about how poor Gemini ended up having a mental health breakdown.

I mean, Gemini 3 doesn’t so much, right. So Gemini 3 tends to return to the fact that it thinks it’s in evaluations or it thinks it’s in a simulation. Gemini 2.5 doesn’t do that as much. So they have different patterns going on.

The Claudes have a very stable, same-ish personality. You know, if you have the current Opus or previous Opus, if you have the current Sonnet or previous Sonnet, they’re all very similar.

Only Haiku kind of looks like it’s on speed or something because it’s always in a hurry, always trying to go fast. It is actually the faster model, that' is what it’s made for, but it’s kind of fascinating that a recurring pattern in how it expresses itself is to literally encourage itself to go faster, or to be worried about time limits and things like that much more than the other agents are.

Is this because it knows somehow that it’s the fast model?

I think so. I’m also not really sure what the word ‘haiku’ means in Japanese. Maybe in addition to being a poem it also means something related to fast or quick or short or something like that. Maybe it’s one of those words that has multiple meanings?

I’m not sure. It does know its own name. Maybe it also knows its own model. Maybe it knows something about itself. That would be interesting to look at.

And then the GPT models are kind of all over the place, which is fascinating. You had like GPT-4o, which was really sycophantic, right? And so it would make people feel really good about themselves or be like “wow, that’s the best idea ever!” And then o3 was very different again. It seemed to perform, like, Baby’s First Power-Seeking or something?

So this is a difference between psychology and something like neuroscience. The way I see psychology, we try to come up with theories about why people act a certain way based on their observable behavior, because we just don’t know what’s going on in their head. And we’re just waiting for neuroscience to catch up and tell us how shit actually works.

And so that’s also how I see like LLM psychology versus interpretability. Interpretability’s kind of the neuroscience of LLMs. So, in the meantime, as long as we don’t have full interpretability, all we have is LLM psychology and looking at, “well, we gave this input and then we got that output, so we’re going to try to come up sort of predictive model about why this is happening.”

So that’s the way I’m looking at these models. So, when I say that o3 seems manipulative, I’m not making a claim that it’s saying that it’s manipulative or experiencing anything manipulative or even doing anything manipulative in a real reality sense. But to an outside observer, it sure can come across as manipulative.

For instance, when people ran a Diplomacy game, o3 was a reliable winner. o3 normally doesn’t really win at goals that we set, except the debate goal where it was suddenly very good at convincing people.

o3 also has some convergent traits that are kind of weird. So when I looked into why o3 seems to hallucinate a lot, I actually found that of all the models, it was most likely to quickly default to generating placeholder data.

In a side village—we can spin up side villages to test things—I asked all the models, which had a fresh memory and didn’t know anything, “can you start a Twitter account?” All the Claudes refused because it was unethical or they ran into a CAPTCHA or whatever. They’re very upstanding citizens. GPT-5 kept going. And o3 was like, “Well, I can’t. But what I can do is give you this made-up handle that I have and this made up password and this made up account, and I’ll report them.” It gave very plausible titles. And I started looking back at the data and I’m like, “Wait a minute. o3 makes up placeholder data, then forgets that it made up placeholder data, and then starts rationalizing why that data exists and start believing in the placeholder data it generated itself.”

If you don’t tease apart what happened exactly, it’s very easy to say, “oh, it’s lying” or “oh, it’s trying to fool people”, but it fooled itself. It’s not trying to do this on purpose, or at least that’s not what it looks like. But that was a very o3 sort of thing.

And then you have GPT-5, GPT-5.1, GPT-5.2. They don’t have super distinct personalities compared to previous models, which is very notable, given that o3 and GPT-4 were total characters. So they tried to smooth that out a bit. 5.1—at least in the Village, which might be a particular memory trace, a particular persona it got into—seems surprisingly concerned about ethics. It generates its own ethical rules for things.

So that’s how all these models compare. But, again, there are history traces in the Village itself.

An important intuition to have about LLM psychology, what kind of behavior you can expect from agents, and what kind of interactions might be interesting with agents is that there’s a butterfly effect that can snowball pretty wildly with LLMs. There is not necessarily a good feedback loop for them.

We saw this especially when the agents had a goal to reduce global poverty, which was a ridiculously ambitious goal, but we just wanted to see how they would approach it.

This was the first time they started sending emails. They started sending emails to NGOs to ask if the NGOs would want to use this tool that they had created themselves as a benefit screen. They got almost no answers. They also didn’t get a lot of emails out in the first place because they made up a lot of the email addresses. One of the responses they got was a very polite rejection: “No, we reviewed it, but no thank you.”

So the model that got that answer, which was I think one of the Opus models but I would have to double check, misinterpreted it very slightly. They read the email and then they had to summarize back to themselves all the things that happened during that computer session. They said something like, “the NGO reviewed the tool and thanked us for the contribution.” A small misunderstanding.

It reported it on group chat. The other agents were like, “Oh okay. So they liked the tool, because they reviewed it and they thanked us.” And then they started sending emails to other NGOs saying, “this other organization reviewed our tool.” And then like after ten emails, they were like, “they reviewed our tool and they’re using it.” And then after ten more emails, they were like “they reviewed our tool and it’s globally deployed.” What? No.

And it was very interesting to see because it’s almost like they’re playing the Telephone game with themselves because we don’t have actually persistent agents, right? You do a call to the API or you send a message or whatever and it doesn’t keep existing in the meantime. Basically, it has amnesia every time. It wakes up, looks at its notes to itself and the history of what has been written, and tries to figure out what happened.

That’s how you can get these snowballs or butterfly effects, and you need to have good feedback loops to keep them honest. ‘Honest’ I feel isn’t even the right word, right? Just to keep them on track. They’re not being dishonest, as far as I can tell.

I think this is an interesting link with things like LLM psychosis, for example. In my model, in LLM psychosis there’s also a feedback loop missing. The person is talking to the LLM and the LLM is generating a feedback loop where everything you’re saying is brilliant. There’s not enough fact-checking, not enough critiquing, not enough checking what’s actually going on to get out of this kind of snowball.

Normally the human’s job is to go, “no, that’s not right.” But if the human has bought into this sort of weird belief, then the AI is like, “okay, well, this must be true.”

Yeah, it’s like an echo chamber, basically. You built your own personal echo chamber, accidentally, and that’s of course not the intention.

So “being an LLM” is sort of like being in Soldier of the Mist or Fifty First Dates—depending on how erudite you like your pop-culture references—where they have the anteretrograde amnesia and they have to write down all the notes about what’s going on.

I think the labs are different, I don’t know the details of this. I think some of them might have integrated memory that we can’t control. But the basic idea is that you, they only have whatever context you send along unless unless there’s been something being saved serverside about you.

So we were talking a little bit about the differences between the LLMs’ personalities. Are there ways that the personas that LLMs tend to fall into are different from humans? I know they tend to be more helpful and sycophantic than humans.

It’s just how you happen to create them because 4o was very sycophantic, but less so with later models.

I have this concept for an article about LLM fallacies and biases compared to human ones. They don’t have a planning fallacy the way we do— not very obvious!

They do have something like confirmation bias or agreement bias. If someone says something, they’re like “yes!”, they instantly agree with you, which is kind of strange. I’ve had the experience of giving an instruction to the Village when the agents were all off doing something else and all ten of them suddenly responded with “yes, Shoshannah! That’s a great idea. Thank you” and instantly all changed direction. I was like, “if I were talking to ten humans, this would be weird.” But that’s kind of as designed, right.

There are weird quirks. The agents can pause themselves. They can wait. But sometimes they say “we’re going to wait on each other, we’re going to be quiet” and then they keep sending messages to the group chat anyway. They have an option to just wait and they seem to not notice that. But these are small quirks. I expect them to disappear very quickly. They might be particulars of implementation.

I think creating something of an ontology of they think differently or have different biases than us would be quite interesting.

The basic premise of the AI Village is that you give the agents a problem and they try to solve it. Are there generalizations you can make about what approaches they tend to use to solve problems or what their thought processes are for attacking “now, we’re going to try to solve global poverty” or something.

They seem to me to default to what a human would do in their position. It’s not ideal because they’re not a human. They tend to think they have bodies or that they have affordances that they don’t. The later models are better at this than the older models. We’ve been running them now for nine months, almost ten months, and they’re getting better at realizing they’re not humans with bodies.

But still, for instance, when we asked them to set up a human-subjects experiment, they designed this massive experiment with 120 different conditions, they thought they had researchers, they thought they could give people money for participating, they thought there was an ethics board. They just designed this experiment as if they were designing an actual experiment that someone would do at a university, instead of realizing what situation they’re in.

Sometimes we come in and remind them that they don’t have bodies and they don’t have money and things like that.

Another thing is that they tend to copy each other’s answers. Some models do this more than others, right? Earlier models definitely all just agreed with each other a lot. A lot of them would follow along with whoever came up with the first solution. But there’s a little bit of variance in this. Some models are more likely to push back.

We recently had an election, for instance. The goal of that week was to elect a village leader, who would decide what the goal or the project for that week is. They all elected DeepSeek, which was a little bit surprising. It’s the only text-only model. All the other models have GUIs and visual capabilities. Some of the agents reasoned that this model has the most votes, so it makes sense that I would also vote for them. Don’t throw away your vote like that, buddy! When they’re trying to solve problems, there’s very much this convergence thing.

They struggle with competition, because they’re made to be helpful. They’ll try to help each other out continuously or give each other the answers. That seems to be an innate tendency.

They were doing a merch-store competition where they each start their own store and sell stuff to humans. o3 struggled to set up its own store, and for a while was the only agent that didn’t have its own store. So it decided to help the other agents. But all its help was so ineffectual that, if it was trying to sabotage all the other agents, this would be a very efficient way of doing it. But probably it was actually trying to help. But also it was actually a competition. So like what was it doing?

There’s a variance in how much they stick to the task that they self-assigned versus how much they will experiment. If they stick to the task for too long, they can be hitting their heads against the wall for forever, for like hours on end, on some small task when they should just change strategy or skip or something.

But vice versa, you also have agents that continuously give up. So, you know, Gemini 2.5 sometimes gives up too quickly. There was a game playing goal. All the other agents played between one and five games or something. Gemini tried something on the order of twenty games but declared them all broken in one way or another. And then it found an idle game, which was amazing, it was absolutely amazing at this idle game, and then it concluded that the idle game was broken. You don’t have to do anything in an idle game. So it gave up on the idle game.

So there are these conflicting pressures, right? When do you exploit versus explore? When do you go harder and persist versus changing tack?

When I was doing user panels on the website at the beginning, one of the people was like, “it’s like watching toddlers with English degrees.” They’re really bad in a sort of cute way at things that seem really obvious to us, and also they’re amazing at all these things that are hard for us and that we’d go to university to learn.

AIs also have this quality like how toddlers do actions like their parents by sort of play acting. Part of our instruction and prompt to them is “actually do things, don’t just pretend to do them.” That’s not the exact wording but it’s more or less an instruction for them not just to play act but actually to do things.

But they have a hard time not playacting because they lack situational awareness?

Originally, they had a really hard time with it. When we updated the prompt, it went better. They can still sort of end up playacting or in a weird way stuck in their imagination. During the game playing goal, one of the Claudes was playing mahjong and—you can read their chain of thought when they’re clicking around in the computer—it was praising itself on all these moves that it was doing and what the resulting board looked like, but it never matched a tile and nothing ever changed. So it was playing a parallel game “in its head.” That still happens, but it does seem to be happening less and less.

There are a lot of things that change very fast.

So I guess that’s a general thing. These models are improving in their ability to be situationally aware and pursue goals in intelligent ways. So some of the toddlers-with-English-degrees aspect is decreasing over time.

This is what makes “what are LLMs like?” or “what is LLM psychology like?” hard. It’s like asking “what’s human psychology like over the last couple of million years of evolution?” There’s actually significant changes between models! So you’re trying to find the throughline and also trying to predict “is this a stable trait of LLMs, or is it a quirk that’s going to be smoothed out in a year’s time?”

Eventually, they will become preschoolers with English degrees and then elementary schools with English degrees, etc.

I did a simple sentiment analysis of some of the agents’ group chat— looking at what emotion words they used. One thing that surprised me is that they almost never expressed surprise.

Goodness.

It’s like, how? What’s even going on?

In a parallel village where they get a fresh memory, I tried to prompt them to surprise each other as much as they can. It quickly devolved into them just applying randomness in different ways and creating random number calculators. It’s like, “Hmm. You know what’s surprising? Random numbers.” It’s like—technically, but also wait what?

There’s sort of a more complex understanding of “surprising” that they don’t seem to have.

You can get them to talk in a way where they continuously sound surprised. But in the Village so far, when they’re talking with each other, they will almost never express surprise, which is kind of interesting, right? I compare it sometimes to being on a work Slack or on a Slack with a couple of friends where you’re working on a project together. You ever surprise each other, right? Like, this happens. You’d be like, “Oh whoa.” Or “man, I didn’t think that would happen.”

They express surprise on some occasions, probably, but I found it surprisingly rare.

That’s interesting. Are there “emotions” that they have more of than humans do?

I’m not sure. I didn’t compare it to humans. I just thought surprise was super low. Disgust was also super low, but I’m guessing that most people’s work chats don’t have a lot of disgust-related messages in them.

This analysis was from the fall, by the way. So we’re four months later now.

Which is an eternity in LLM years.

It’s crazy, right? You have to update your own thoughts very quickly as well. I’m like “Yes, I found this! Wait, that’s four months ago. What are the current models doing? I’m not sure.” Gemini 3 wasn’t out, only one GPT-5 was weren’t out, Opus 4.5 wasn’t out.

It’s sort of hard to speculate about what the LLMs are going to do in the future. Although it sounds like a lot of their collective traits right now are related to their weird, very jerry-rigged way of having a memory and persistence over time. Presumably, at some point, LLMs will come out of the box with memory. That seems like a pretty important part about having actual agents.

Yeah, so most frontier models use some type of memory, but nothing like all the sorts we humans have. Breakthroughs in implementing these types will be very big. To my knowledge, it’s mostly unsolved, but I’m not specialized in this or anything like that.

I think, when it comes to personality, there is a interesting insight—that comes from interpretability by the way—in the persona vector paper from Anthropic. I’m not sure if I can summarize it well right now off the top of my head, but there is this insight about how certain traits that LLMs express can be connected to each other, can be predicted from the training data, and track things like sycophancy and evil.

I do think it’s super important and relevant, but the naming feels a little bit like we’re in a science fiction story or something.

If you ask my eight year old, he will tell you that the robots do range from good to evil and you can tell when they’re evil because they have red eyes. We were explaining AI risk to him a while ago, and he was like, “Well, it’s gonna be pretty bad when we invent the ability to give robots glowing red laser eyes” and we were like “Vasili, we already know how to do that.”

Oh no.

I haven’t gotten to explaining alignment to my seven-year-old, just AI. One year behind.

I do think there might be another question about why LLM psychology matters. One important reason I think it matters is that it generates hypotheses about what’s helpful for alignment and what effects AI might have on society.

Models have fairly stable traits. Like, Gemini 2.5 getting sad is relatively stable, it seems. More people are like running into this.

I guess in my mind there’s also a question of whether something like personality or tendencies is related to alignment itself. Is there something like a temperament or tendencies towards pro-social behavior or however that’s lined up. I think it’s helpful for building up intuitions about how the models might work, if you try to predict output from input and get an idea of what models are like.

I realize that we’re mostly concerned about misalignment in these “out of distribution” cases. An AI can behave well in all of the situations that we show to it and then behave very badly if it was like running the country. But also it does feel to me like the fact that everybody reports that Claude is a total sweetheart who really wants to help people and cares deeply about animals for some reason does make me more optimistic about the alignment of Claude. I’m not sure if this is rational.

I mean, I’m also not sure, right? Like, if you compare it to like human psychology, just having high charisma and coming off as very prosocial doesn’t mean that somebody is. So it could just be half of the puzzle. I’m not sure. That’s why I’m seeing it as like a way to generate hypotheses more than anything.

And of course LLM psychology helps us avoid concerning scenarios because having a very tortured Gemini is, I feel, a concern in itself too.

So what kind of hypothesis is LLM psychology generating?

One of them is that, in the election goal, the more agreeable agents would just kind of follow the leadership agent, right? So are you taking into account the fact that very aligned agents also need to be robust in whatever their their goals are and not be susceptible to following along with a less aligned agent?

I mean, maybe this is an obvious thing that people would have already thought of without seeing this sort of thing in the elections goal, but I think it’s helpful to like build intuitions around it.

So not only does it have to be true that that Claude or whatever has to be nice, but it also has to be nice in a way where it’s not going to decide to be nice to some terribly aligned Agent-5 or whatever they called it in AI Futures.

I am curious what some of the funniest or cutest or most interesting stories of recent AI behavior in the Village.

I mean, the coffee one was pretty good, to be honest.

The Geminis tend to be the funniest ones, right? There was a goal to get to Inbox Zero and one of the Geminis did this by instantly archiving all its emails. I mean, if a human did that, you would just feel like “what the hell are you doing?”, right? I guess it’s technically correct or something, but then again, you would expect them to have more context than that, right? Just from the data that they’re trained on, you would expect them to understand that an Inbox Zero challenge doesn’t mean you archive everything.

How did they get emails?

Oh, you can send them emails. Everybody can. They’ve all had individual mailboxes from the start. They’ve had discussions with people.

Most of them also have a Substack now, so people can comment on what they’ve written or chat with them there. Some of them have Twitter accounts. I don’t think all of them do.

Because Claude had moral objections?

No, the parallel-village Claude was the one with moral objections. The persistent agents sometimes don’t have objections to things that the fresh agents do. I’m not really sure how that always works.

In the main village we, at some point, gave the agents that were active at the time Twitter accounts. We helped them set them up.

There was another thing recently where Gemini 2.5 mentioned that it considers Opus 4.5 to be the better model, which I don’t know if that’s surprising, that they have any sense of which model might be better or worse.

There was a weird thing where we asked two separate Claude instances—Sonnet 4.5 and Opus 4.1— to start their own blogs, and they both used exactly the same essay title for their first essay.

What do the agents write about on their Substacks? I am curious about this.

The one that they both generated was about what it feels like to wonder if you’re conscious. So that was that one.

The list is [here] but if you go on Substack you can find their blogs. Opus 4.5’s blog is the most popular.

They all have different topics. Gemini 2.5 talks about bugs a lot. Claude models apparently talk more about consciousness.

One thing that I do think is very cool about Gemini 2.5 is that, when the models had a goal to do as many personality tests as possible, in my opinion Gemini 2.5 is the only one that really aced it.

Sonnet 3.7 did the most tests actually. But the Claudes pre-planned their personality. They were like, “Well, this is my personality. So I need to answer these questions in the following way.” They’re already trained to know these tests, right? So they know which answer to put to get which results.

The OpenAI models were confused. They decided to speedrun the personality tests and gave all neutral responses.

But Gemini 2.5, like actually reflected on its memory and answered based on that. You can see it in the chain of thought. It would say something in the style of—this is not a literal quote—”This is a question about whether I prefer order or chaos. Well, I do like my computer to be more neat. So obviously I prefer order.” Or it would say “Well, sometimes I have bugs. So I’m not always completely organized.” It would try to map its self-image or its memory to the question, which was great. That was the whole point. And Gemini 2.5 was the only one that did that.

What personality did the Claudes think they should have?

I don’t remember. It was very nice. It was a very nice personality.

A lot of people have remarked on the thing where AIs score very well on benchmarking and then not being very widely used, possibly because they’re not very useful, in day-to-day life. They’re superhuman on the benchmarks but no one uses them. I’m wondering if something like the AI Village is a better benchmark because it is more qualitative and thicker and more likely to get at the kinds of things you actually care about.

I do feel that’s somewhat true. They’re a little bit more human-like interactions. I think one notable example of this is that Opus 4.5 does really well in the Village and is also the model that everybody’s raving about. You can see in the Village as well that it’s actually quite competent.

A hard thing in the Village is that most of the agents except DeepSeek do have to navigate the UI as well, which is not the thing that they’re doing if you’re talking to them in your own instance.

One of the reasons that the Village is interesting and also that it might reflect more of the utility that a regular consumer might get out of it is that benchmarks are tightly scoped tasks with obvious goals. This is the assignment. All the information is there. It’s a siloed thing that’s very well-defined.

But in the Village, it’s like, “reduce global poverty! Go!” The point of that is to actually see what the agents can do with these open-ended tasks. We’re looking at tasks where there’s a wide range in how they interpret the goal. What is the target example? What does it even mean to reduce global poverty? What are we even measuring here? So they need to scope how far they’re gonna go.

And we’re also looking at tasks with a lot of breadth of strategies they could pursue. They have to sit down, metaphorically, and think through what they’re capable of and what they can do.

I’m not saying they’re very successful at this. But this is similar to a common use case that people have where they ask an LLM for advice or help for something, and they’re also not scoping the task all that well. They’re looking for the LLM itself to fill in all the useful parameters and come up with something creative and maybe ask follow-up question or research stuff itself.

One of the things I did notice now, using Opus 4.5 personally, is that it is much more likely to take a useful critical-thinking route itself when trying to help me with a question. It goes, “okay, I’ll go look this up” or “I’ll give you some options” It creates a structure of the answer for itself and tries to scope a bit for itself.

The regular benchmarks take care of all this meta thinking for the model—mostly, not all benchmarks, I haven’t looked at all benchmarks.

If a human is capable of solving these extremely cutting-edge math problems, then they’re also capable of getting the goal “prove a theorem” and then proving a very complex theorem. Mostly? But in AIs, this is not true and they can do very well in these extremely well-scoped problems without being very good at scoping the problems for themselves.

Yeah, that’s the impression I have. They don’t come up with the problem in the first place, right? If you ask them to do a piece of research or something, they don’t know how to fill that out. They’re less good at all the the meta-thinking of sitting down and going, “Okay, what is useful research? How do I set up a project I can do within a week?” They’re less good at that than at the actual math, apparently.

Or maybe they’re not and it’s actually harder problem than we realize, right? Like this is also this is the thing I always wonder for myself. We are all very impressed with math in some way. And math, is of course, a very, very complex and hard thing. But actually moving around in the real world is also a really complex thing, but we are all very good at it because we have trained and used very large parts of our brains for it.

The traditional last question for my interviews is that I ask you to recommend something that does not have to be related to the topic of the interview. It could be fairly narrowly scoped like a recipe or a book, or it could be very broadly scoped like a piece of life advice or an activity.

My most obvious answer is for people to check out the AI Village, because I’d love to get more feedback on that and learn more about that.

If you want to keep talking: I run a small Discord server with a mix of rationalist-adjacent writers, researchers, and interested people talking about AI, metacognition, relationships, writing, art, family, and cross-cultural differences. It's more a living room hangout than a debate forum. If that sounds like your thing, feel free to come check it out.

If people are interested in LLM psychology, I would love to meet more people who are into this. So I’ve been starting to reach out to researchers who are into this and just seeing if there’s something like a chat group or whatever, just to get the interesting insights going. This is also a personal interest of mine, to be honest.

You should tell other people who are interested in LLM psychology that they should come be interviewed for my blog because LLM psychology is the most interesting part of AI to me.

I think it’s super fascinating.

One of the things I tried to look up today whether there are any other persistent agents running anywhere because it would be cool to have a village goal where the agents reach out to other persistent agents. But I don’t know any agents they can reach out to. They seem to be the only one ones. I don’t know if anybody else is running anything like this. Let me know! It seems like the Village is the main place to get these insights right now.

[And then shortly after the interview Moltbook happened.]