Information control, isolation, and ideological abuse

[I have freelanced for a number of effective altruist organizations, such as the Centre for Effective Altruism and 80,000 Hours. I don’t speak for any of my clients, past or present, and they didn’t look at this post.]

[Previously: Why join high-demand groups?, Identifying healthy high-demand groups, The structure of ideological abuse]

This is the fourth of four posts reviewing the book Abuses in the Religious Life. This review talks about some miscellaneous features of ideologically abusive groups. You don’t need to read the previous posts in the series to understand this one.

Information Control

De Lassus emphasizes most of all the importance of information control in forming a negative high-demand community.

De Lassus doesn’t elaborate clearly on why information control is so important. To me, it’s important because ideologically abusive groups’ primary enemy is the truth. An ideologically abusive group only functions if people fail to put together things like:

• “I’m not alone. My suffering is not because I’m broken and a failure. Everyone else is as miserable as I am. We’re just all hiding it.”

• “The linchpin is never satisfied with anything, and she contradicts herself constantly.”

• “The story of the miracles and heroism associated with the group’s founding is all lies.”

• “Very few of the promises the group makes ever come true.”

• “None of the beliefs we’re supposed to believe make any sense.”

• “No one experiences the spiritual bliss they were promised.”

• “Other groups achieve the same things as our group, but are far less wretched to be part of.”

• “The group has double standards for high-status and low-status people.”

• “The linchpin is living a life of luxury while we’re all poor.”

• “Wow, the linchpin is raping a lot of people. Like, a lot of people.”

Ideologically abusive communities have a “culture of lying.”¹ Often, the lies are extremely stupid and about stuff that doesn’t matter. For example, de Lassus talks about a pair of religious brothers on a trip who had been told to have a particular private conversation, so they would return to the monastery after they’d normally be expected to return. The superior of the community announced, well before the brothers’ train had even left, that the train was running late so the brothers would arrive at the monastery late.

Of course, everyone in the order saw that this was an obvious falsehood. But that’s the point. The goal is to create a situation where all information is dubious, where it’s impossible to distinguish truth from falsehood. An ideologically abusive group maintains itself when everyone is desperately trying to believe claims that clearly aren’t true, or else questioning themselves about it: “is it possible that the train could be that predictably late? No one else is objecting to it. Maybe I’m just very wrong about the nature of trains.”

This dynamic functions just as well when the lies are small. Indeed, in some ways it functions better when the lies are small. People know they can’t trust any information, even about matters as trivial as train schedules; they feel insane.² When falsehoods are common—especially when they’re common and you’re not allowed to point them out—it’s harder to notice the falsehoods that are important. And if you try to explain what’s going on to someone outside the group, you sound like a crazy person. What’s wrong with someone being mistaken about train schedules? It’s easy to assume that victims are making a big deal about nothing.

Some bad actors deliberately lie in this way, because they enjoy deceiving people or because they’re trying to make people question their understanding of reality. But it’s equally common for a culture of lies to emerge in a well-intentioned way. Maybe the superior is genuinely embarrassed by the private conversation, and doesn’t want people to speculate about it. Perhaps, at first, this even feels altruistic: maybe the private conversation is about (say) one brother’s masturbation habit,³ and the superior doesn’t want people to speculate about what the brother is struggling with that necessitated a conversation. But the ultimate effect is to create a culture in which no one can rely on any statement to be true.

And once people are used to lies as a tool to manage social situations, it becomes easier to reach for the tool in more and more minor matters. At first, you lie to genuinely spare someone’s feelings and protect their legitimate privacy from voyeuristic speculation. Soon you start lying to avoid an uncomfortable conversation or an inconvenient task. In time, lying may become so automatic that you do it for no reason at all.

If you want to lie to protect someone’s privacy or for another good reason, consider instead saying “I’m not going to answer that question”, optionally with a reason (“I’m under an NDA”, “it’s a personal matter that’s none of your business”).

Often, lies are used to cover up other lies. For example, someone might spot the brothers walking when they’re supposed to be on the train, and you’ll have to come up with some explanation. (”Oh, it was actually some other people who looked very similar...”) Further, inevitably some members of the community will know what the train schedule is. You can then wind up lying—or at least telling half-truths—about why you were lying. You might say “I was trying to spare the community distress” or “I was trying to prevent malicious gossip”, when in reality you were primarily afraid of inconvenient questions or tarnishing the community’s image in the eyes of its members.

Eventually, the community or its leadership may have told so many lies that finding out how many lies had been told would destroy the community’s or outsiders’ trust. So the community and its leaders feel like they have to keep lying to preserve the community. The number of lies expands until everything in the community life is subject to one deception or another.

Over time, community members come to justify their lies, to think of lying as good rather than evil. This is much worse than simply telling a lot of lies. If you start to believe lying is good, you’re no longer restrained by the qualms of your conscience. You no longer need a justification. You’ll lie whenever it is mildly convenient to do so. And it becomes nearly impossible to recognize that you’ve done wrong and make amends.

But the culture of lying isn’t the only way ideologically abusive communities control information.

In ideologically abusive communities, communication is often vague, ambiguous, and full of half-finished sentences. Communication occurs at inappropriate times, such as during prayers and services or when walking the halls. At any time, you can be jump-scared by some unexpected task or criticism. Authorities alternate in an unpredictable fashion between praise/approval and anger/contempt.

I’ve observed rationalists and effective altruists communicating in a vague and ambiguous way while looking like they’re communicating in a clear and direct way. Bad actors might pedantically nitpick other people’s sentences, or insist on arbitrarily precise and specific definitions of words, or claim that sentences are ambiguous even though a reasonable person would be able to figure out what they meant. These communication strategies create cover for vague, ambiguous communication: “it’s your fault you misunderstood me because if you checked Subclause 12(a) of my glossary it would be obvious what I meant.” And other people wind up hedging to the point of incoherence in the hopes that the bad actor won’t jump on a poor phrasing. As one of my beta readers remarked, it is never a good sign when someone is spending ten thousand words litigating the precise definition of “sexual assault” or “sexual harassment.”

Poor communication serves several purposes. It gives the linchpin and the leadership plausible deniability if their instructions go wrong (”I obviously didn’t mean that, why did you think I meant that?”). It prevents in-depth conversations that might reveal that the group’s beliefs, decisions, or actions don’t make any sense. Most of all, it keeps everyone on edge, stressed and nervous and insecure. Frightened people are bad at thinking.

Ideologically abusive communities often limit religious’s access to books. De Lassus is nuanced here. Monasteries and nunneries commonly limit access to the Internet: a monk who lives a life of quiet prayer and devotion secluded from the world shouldn’t be posting sick dunks on Bluesky. And monasteries may legitimately curate their libraries to exclude books that have nothing to do with being a monk. But there should be no limit on religiously orthodox books about spirituality, theology, or the teachings of the church. Religious shouldn’t require permission to visit the library during free hours.

I have a much stronger position here than de Lassus. While de Lassus wants people to be able to doubt the wisdom of their superiors and monastic orders, he distinctly doesn’t think it’s legitimate to doubt the Catholic Church. He believes it’s fine to limit the Da Vinci Code, not to mention The God Delusion, because he actually thinks it’s fine to deny people information that might cause them to stop being Catholic. Since I think Catholicism is false and “believing this particular set of claims regardless of the evidence” is not a virtue, I believe a high-demand community should never limit your access to books, particularly books which are critical of the group itself.

Ideologically abusive communities often de facto ban “horizontal” communication. All communication goes “up” from the member to the superior, and then “down” from the superior to a different member. In Catholic communities, this is typically justified spiritually: the spiritual benefits of silence or discretion, protecting others’ interior lives, participating in the unity of the Trinity.

Of course, even in orders that practice long periods of silence, you can’t literally ban members ever speaking to each other, even to ask to pass the salt. But you can make sure that all requests, conversation about feelings, sharing of significant information, or other conversation of any import goes through the superior. That way, the superior can easily quash any topics they find bothersome or uncongenial. No member is able to discover that their doubts, criticisms, and misgivings are shared, or to work together to confront the superior about their behavior.

The most important piece of information to share is the details of group members who leave: why they left, where they went, their physical and psychological state upon leaving. Whenever someone leaves a high-demand group, the other members should be aware of basic logistical details and, in broad strokes, what’s going on. People close to the person who left should understand what’s going on in more detail. There should never be silent disappearances about which no one ever speaks.

Knowing why and how other people left is the most important piece of information for judging whether you should and how you can leave. If you know how someone else got out, then leaving is a practical possibility rather than a series of question marks. If you know what made someone want to leave, that prompts you to ask yourself whether you want to leave too. “Why people left” is the most concentrated source of information about the flaws of a group and what it’s getting wrong. For this reason, it’s the most important kind of information for ideologically abusive groups to control.

Isolation

Isolation from people outside your high-demand community is bad. Any community—even a business or a family—can easily get caught up in groupthink and the normalization of deviance. The best preventative is to occasionally explain yourselves to an outsider who can, if called for, go “whatthefuckwhatthefuckwhatthefuckwhatthefuckwhatthefu—” Isolation also makes it hard for people to leave, because they have nowhere to go.

But in Catholicism, religious are typically isolated from the world at least somewhat; all orders limit communication with the outside world. Some (contemplatives, hermits) are very isolated. Isolation can help people focus on God and their callings, instead of being continually distracted by the outside world. And to some degree isolation is normal and inevitable: if you’re devoting yourself to a complicated project that consumes your entire life, you might not have much to say to people who aren’t working on the same project.

Some Catholic orders that strictly limit communications with the outside world are far healthier than other orders with looser rules. How can we get the benefits of isolation without risking the harms?

First, all social isolation should be justified based on the needs of the community. In a Catholic context, a contemplative order that maintains a robust inner silence to pursue a mystical relationship with God is obviously going to limit communication with the outside world. On the other hand, a community that focuses on doing good works and helping the poor needs to reach out to the secular world, that being where the poor are. If the latter has rules that look more like the former, that’s a red flag.

In a secular context, consider the reasons for isolation. Are you forbidden to talk about work because of legitimate confidentiality and information hazard concerns, or because your bosses don’t want outsiders to ask awkward questions? Are you avoiding people who are derisive and mocking about your goals, or anyone who doesn’t already agree that your goals are important?

Second, high-demand group members should have complete freedom of expression. It can be reasonable for a Catholic religious order to limit members to (say) one phone call and four letters with family each month. But the religious should be allowed to say whatever they want in their phone call and letters. And superiors should never read people’s mail.

Similarly, in a secular context, a high-demand group should never, absent a small number of situations where you need to e.g. keep work information confidential, forbid you from saying your genuine thoughts to the people closest to you. You should always feel able to talk about your fears, doubts, hopes, and miseries. If you’re unsure what to do about a problem, you should be able to talk about it. You should never feel obligated to conceal something from those you love because “outsiders wouldn’t get it” or “it would make the community look bad.”

Third, some kinds of isolation are always inappropriate. A high-demand group should never prevent members from speaking with their family, for example. It should also not limit what members say to their family about community life or their own personal lives.

For Catholic communities, it is always inappropriate to limit what people say to their confessor. In the Sacrament of Confession, Catholics tell their sins to a priest; the priest absolves them, which means that God forgives them of their sins. Most devout Catholics see a single priest for the Sacrament of Confession, who is called their “confessor.” Obviously, people need to be able to be honest with their confessor. While some limits might be reasonable if (for example) someone is going to Confession for an hour every day and not getting their work done, people should be able to see their confessor regularly and frequently for spiritual counsel and to get absolution for their sins. Ideologically abusive Catholic communities frequently limit people’s access to confessors. For example, they might forbid people seeing confessors outside the community; teach that they should hide some aspects of your community from their confessor because the confessor “wouldn’t understand”; teach that saying negative things about other people in confession is “malicious gossip”; or even say that asking for help from your confessor shows a lack of trust in God and a reliance on human help rather than supernatural help.

I have had a lot of Catholic and ex-Catholic friends, and it’s hard to describe to people unfamiliar with Catholicism how shocking I find the concept of limiting what people say to their confessor. The confessional is sacrosanct. Many, many people have died rather than reveal what was said in the Sacrament of Confession. If someone makes an incomplete confession of certain sins, Catholics believe, they will literally be tortured in Hell for all eternity, unless they manage to make a complete confession later. One of the best things about Catholicism is how much work the institutional church has put in to try to make people feel comfortable being honest and open in Confession about all the worst and most shameful aspects of their lives. I am stunned and horrified that any people who call themselves Catholic think that “you can say whatever you want to your confessor” is even in question.

In a secular context, a group should never require you to cut off your family (parents, aunts/uncles, siblings, cousins, spouse, children, grandchildren). You may decide of your own free will to cut off your family, if they’re abusive or mean to you, but this should never be a requirement of group membership. Similar rules, I would argue, apply to longstanding friendships. You should also be permitted to talk to a therapist of your choice.

Checks and Balances

Healthy communities have a system of checks and balances, so that no one person winds up having too much power. Ideologically abusive communities tend to route around these checks and balances.

In a Catholic context, the biggest limitation on an ideologically abusive community is the Church itself, which often (not as often as we would hope but, you know, often) takes a dim view of embezzling money, torturing group members, and committing rape. Usually, the communities with the worst spiritual abuse problems are the ones that try hardest to avoid church discipline. De Lassus:

In situations where the behavior of a community is becoming aberrant, we generally observe a curious dichotomy. On the one hand, we see an extremely demanding doctrine of obedience for the members of the community, which involves submission not only of the will, but of the intellect too, since every word of the superiors must be considered as a word of God. On the other hand, we find an extremely lax regime for the superiors themselves; there are no qualms about minimizing (or even completely and deliberately ignoring or setting aside) obedience to the Church and to canon law.

Often, ideologically abusive Catholic groups justify ignoring the Church by saying the Church doesn’t really understand the group. If the linchpin or the superior is seen as the voice of the Holy Spirit, any disagreement the linchpin or the superior has with the Church shows that the Church has been misled and is decadent. The group’s disobedience to the Church is seen as a deeper form of obedience to the true Church. Any attempt to crack down on the group is seen as persecution and therefore a sign of the group’s holiness.

De Lassus recommends, in terms of checks and balances in the Catholic context:

• A community should follow the canon law.

• The community should have a separate abbot, novice master, and cellarer.⁴ It should probably also have a separate prior (second-in-command).

• The community should have a constitution outlining how it is supposed to work, which is clearly written, unambiguous, concise, not overly rigid, and not so precise that the community has no ability to make allowances for changing circumstances.

• The community should be regularly visited by a bishop or a superior of the order, as appropriate, who interviews all the members of the community privately and who has the power to remove the abbot for misbehavior. During such a visitation, all orders to shut up about something should be considered not in effect.

• If there is reason for suspicion of the group, a special representative from Rome should visit to investigate.

Now, it may be difficult to figure out how to apply these insights in a secular context. Most high-demand groups in the secular world aren’t part of a larger hierarchy that can investigate whether they’re abusive. Many high-demand groups—like effective altruism—have widespread disagreement about how the community is supposed to work, which makes creating a constitution that applies to everyone impracticable. But here are my thoughts.

If a community has a point person who is supposed to evaluate whether a situation is unhealthy (such as the Astral Codex Ten Meetups Czar or the Centre for Effective Altruism’s Community Health team), no one should ever ask you to lie to them. If the point person is failing to do their job, the solution is a public reckoning, not a private campaign of deceit.

As with the separate abbot, prior, novice master, and cellarer, the community should have multiple positions of power which aren’t all held by the same person. Meetups should have two organizers for lots of reasons; this is one of them.⁵

If the community doesn’t have a hierarchy that can investigate, then the community should have multiple distinct organizations which can all check each other. For example, the effective altruist/rationalist community has the Centre for Effective Altruism, Coefficient Giving, 80,000 Hours, Lightcone Infrastructure, etc. Insofar as this is possible, no single group should be the only provider of anything: money, conferences, information-sharing programs such as magazines or forums, etc.

Organizations should be transparent in a way that permits people to independently investigate them. For example, they should make their financials publicly available as much as is possible. Organizations—particularly those that have a stewardship role over a particular community, such as the Centre for Effective Altruism—should actively communicate about their strategic plans and current programs. Non-disclosure agreements should be tightly scoped to actual confidentiality issues. Non-disparagement agreements and other anti-whistleblower measures shouldn’t be used.

All organizations larger than a local meetup group should have an independent entity, such as a board of directors, who is able to remove the leadership for misconduct. No one should hide anything from the board. Boards should not have conflicts of interest. Boards should be actively involved in their organization so that they would be able to notice misconduct if it happened.⁶

Important behavioral norms and standards should be written down somewhere. Norms and standards that aren’t written down can vary with the whim of the superiors or the linchpin. You need something you can point to to go “you shouldn’t do that.” Conferences, coworking spaces, meetups, and forums should have codes of conduct. If something is considered to be binding on people’s behavior—like the Giving What We Can pledge—there should be a single canonical spot explaining exactly what it binds you to. Looser norms should also, insofar as this is possible, be written down in some kind of central reference location, such as the Effective Altruism Forum’s community tag.⁷

A similar double standard to the “you have to unquestioningly obey me but I don’t have to obey anyone” double standard is that applied to criticism. Linchpins and high-status people arrogate for themselves the right to criticism of members, other communities, the broader movements they’re part of, and the world. Often these criticisms are harsh or public. At the same time, linchpins and high-status people consider themselves to be immune from criticism. If you criticize, you’re disloyal, disobedient, or harming the movement.

Of course, not all criticism is good. You should ideally make true, relevant criticisms, which come from a genuine desire to improve the situation rather than a desire to slander people. But any rule that forbids criticism of leaders or high-status people takes away the most important protection people have against ideological abuse. And a standard that criticisms must be relevant or made in good faith is often used to silence the criticisms that really cut home, while high-status people can make as many irrelevant bad-faith accusations as they like.

Pride

Often, the root of ideological abuse is pride-- the feeling that your group is unique and better than all other groups.⁸

Often, the groups are actually special. The group can wind up pleased with itself about how faithful it is and how much of a light it is to the godless world (or about how heroic it is and how much positive impact it has) because it actually is doing a lot of very good stuff. People compete to outdo each other in how morally good they are. For a time, this can have genuine positive effects on the world and the membership.

But over time people start to burn out. The level of goodness required isn’t sustainable. The temptation is to say “the people who burnt out and left bitterly just couldn’t hack it.” After all, look at what the community has accomplished! In this way, the community blinds itself to what they can learn from who leaves and why.

The community begins by observing that it does a lot of good; it continues by concluding that it does far more good than everyone else, and therefore is superior. De Lassus writes:

We often find here in an implicit form (or even explicitly formulated) the idea of some sort of divine mission, whose goal is to save the religious life or the Church, as well as a certainty that this group alone has managed to preserve truth or wisdom; it is like Noah’s Ark, in the midst of universal perdition.

Since the ideologically abusive group is special, their way of thinking is the only true and right way, sometimes the only way that is really true to the Catholic Church (or to feminism, or Marxism, or the Talmud, or rationality, or effective altruism, or anything else you like). For Catholic groups, any disagreement comes from the Devil, or reflects whatever the bugaboo of the group is: “worldly... psychological... carnal, deviant, mediocre, modernist (if we are talking about a traditionalist group) or traditionalist (if the group considers itself to be modern).” You can certainly list off others (”mediocre” is perennially popular among many different groups).

And of course, since the group is so special, it’s both wrong and immoral to think that anything the group does is bad, or even less than perfect.

Groups will often try to mark themselves as special in other ways. De Lassus writes:

In a community that feels as though it has reinvented everything, the extraordinary is to be found everywhere: the personality and history of the founder, the circumstances of the foundation, and even the basic elements of the community life. Priests for the community are not ordained by the local bishop but by the nuncio, by a cardinal from the Curia, or even by the pope, if possible.

Honestly, I feel like “if this group were Catholic, would they try to specifically get priests ordained by the pope because that’s how special they are?” is a great heuristic for pride.

Because the community is extraordinary, it follows that the community doesn’t have to follow normal rules. (Of course, high-status people and the linchpin are particularly exempt from normal rules.)

Healthy communities, conversely, are simple, humble, and boring. They do modest, everyday tasks. When you are part of a healthy high-demand community, you spend a lot of time answering emails, doing the dishes, organizing movie night, changing diapers, figuring out why the damn machine you’re trying to build isn’t working, or thinking in great depth about data visualization and fonts. You don’t have grand, dramatic stories of casting out demons or winning a thousand converts with a single speech. You have grant applications.

I wrote about this in Why Are There So Many Rationalist Cults?:

Early rationalist writing, such as the Sequences and the Harry Potter fanfiction Harry Potter and the Methods of Rationality, emphasized the lone hero, standing defiantly against an uncaring world. But the actual process of saving the world is not very glamorous. It involves filling out paperwork, making small tweaks to code, running A/B tests on Twitter posts. Most rationalists — like Mike Blume — adjust well to the normalcy of world-saving. (Today he works as a programmer and donates to AI safety and global health nonprofits, a common rationalist career trajectory.) Others want acts of heroism as grand as the threat they face.

The Zizians and researchers at Leverage Research both felt like heroes, like some of the most important people who had ever lived. Of course, these groups couldn’t conjure up a literal Dark Lord to fight. But they could imbue everything with a profound sense of meaning. All the minor details of their lives felt like they had the fate of humanity or all sentient life as the stakes. Even the guilt and martyrdom could be perversely appealing: you could know that you’re the kind of person who would sacrifice everything for your beliefs.

The project itself in each of these dysfunctional groups was vague, free-floating, and almost magical. As soon as you have to accomplish specific goals in the real world, the mundanity of everyday life comes flooding in, with its endless slog of tasks variously boring, frustrating, and annoying. But as long as you’re sitting in a room taking LSD with the blackout curtains over the windows, you can be Superman.

Conclusion

Ideological abuse is bad.

Ideologically abusive groups, at best, make their members fucking miserable. At worst, ideologically abusive groups rape, torture, and even murder people. Perhaps the most terrifying aspect of ideologically abusive groups is that they take ordinary people-- often very good, moral people-- and transform them into monsters.

And ideologically abusive groups are bad at achieving their goals. The culture of lies, silencing internal criticism, isolation from outsiders, getting rid of checks and balances, coercing obedience: these all cut the group off from any way to check whether what they’re doing is working. "We have to be ideologically abusive if we want to achieve our goals" is a pernicious lie.

If you’re an outsider—especially an outsider who is supposed to exercise checks and balances about a group—wondering whether a group is ideologically abusive, de Lassus recommends:

• Speaking, in private, with as many members of the group as you can, as well as former members. (If you’re not allowed to speak with members, that is very bad.)

• Looking for consistent testimony, especially consistent testimony from people who don’t know each other.

• Getting an outside perspective from wise and uninvolved people.

It is very difficult to reform an ideologically abusive group; most such groups ought to be dissolved. Reform usually requires a neutral outsider who is willing to put in quite a lot of work. It usually takes much longer than you expect (even accounting for it taking much longer than you expect).

If you’re the neutral outsider reforming a group, the top priority (de Lassus says and I agree) is to establish a culture of free speech. Other problems are much easier to correct if people feel free to criticize and disagree. Encourage people to form horizontal connections, not just vertical connections. Eradicate the culture of lying.

If you recognize yourself in what I’m writing, you should leave. Maybe the group is entirely toxic; maybe you’re in a toxic subcommunity or relationship; maybe it’s just a bad fit for you. It doesn’t matter. Quit the job. Move out of the shared house. Stop attending meetings. Drop the friends. The community will not get better on its own, or through your personal self-sacrifice. Anything you get from the community you can get somewhere else.

I understand that leaving some ideologically abusive groups is dangerous; if it is, then you might need to delay leaving while you work on safety planning. But delay, don’t stay. Work actively towards leaving, and in the meantime claim as much space and time as you can to keep your sanity.

1. I want to caveat that de Lassus is particularly anti-lying, because in Catholicism deliberate lying is a mortal sin (a sin that gets you sent to Hell). Catholics are naturally prone to accept claims that it is prudentially wrong to commit mortal sins. I think his analysis about lying is still insightful.

2. This is what “gaslighting” really means, if you’re curious.

3. Masturbation is a sin according to Catholicism

4. The cellarer is in charge of food and drink, and it’s not clear to me why they’re one of the three Most Important Guys.

5. I’m not sure if there’s some deep Chestertonian wisdom in having one of them be in charge of food.

6. I like Holden Karnofsky’s reflections on boards here.

7. If you notice a norm that hasn’t been described, try writing up a piece yourself!

8. This is another of those “of course the Catholic is against pride” things-- pride is the greatest of the seven deadly sins-- but I still think it’s an insightful analysis.