There is this mix of self-improvement/spiritual practices, science fiction fandom, cults, and then against all odds being on the actual cutting edge of applied science. Surprisingly, this is pretty old and keeps repeating itself.
For example the story of Jack Parsons, greatest of rocket scientists, early SF fan, and founder of JPL with some of his other SF fan friends, who also did magickal rituals in a polyamorous group house that at a certain point spun into a destructive quasi-cult.
During the earliest years, before truly ramping up the high-control aspects, L. Ron Hubbard (former ties with Parsons) and Dianetics had a friendly reception with a number of legitimate scientists. There is a friendly letter from Claude Shannon (working at Bell Labs when it was the center of the universe) introducing Hubbard (who he met through science fiction luminary John Campbell) to Warren McCullough.
I think most of the factors laid out in your article probably apply to these cases as well as to the rationalists, but it's an oddly specific thing that keeps happening over the decades.
Well, to be on the actual cutting edge of applied science you have to (a) believe you can change the world (b) have faith in your ideas even if people are against you (c) actually have the knowledge in an area that's primed for breakthroughs and (d) be able to do something about it.
I think in (a) and (b) you're more likely to join cults. Lots of people join cults, but only with (c) and (d) do you become a famous scientist afterward.
Before jumping to theories like this, we should first consider a simpler explanation:
a) there have been a lot of cutting edge scientists throughout history, b) being a cutting edge scientist does not make one immune to irrationality, therefore c): it's not too hard to pick out scientists who believe wacky stuff. Since scientists tend to be nerdy types, the wacky stuff they believe will tend to lean in nerdy territory, hence the overlap with scifi or whatever.
The rationalist cults require investigation because they are a small group that has produced a number of cults that is wildly disproportionate to their numbers. The same is not true for scientists: scientists are a very large group that have not produced very many cults at all.
I liked this line: There is no conflict so small that five hours straight of emotional processing can’t make it feel both profoundly important and profoundly unsolvable.
The phenomenon of everything seeming Incredibly Important matches my experience. It’s important to cult dynamics that everyone feel guilty at all times, even if your sin is much closer to not doing the dishes than to murder
It's interesting how some of the biggest factors to cultiness are just kinda boring. It stood out to me that Leverage researchers lived together to save money more than anything else, and that this factored into it becoming a culty community.
Anecdotally i worked for a not at all rationalist company where we spent a lot of time travelling together and hanging out with eachother, and that made the dynamics super toxic, because there was no chance to cool off tensions between people, and work stuff becomes more emotionally loaded when its not just your boss disagreeing with you but your friend.
Turns out work life balance is an important social technology!
This was a really interesting read. I won't say this is WHY I became Catholic (that would be ridiculous), but I feel like it does have the "right level" of cult-ness. Because it's old and huge, its harms and failure modes are well-known and understood. The sharpest edges are mostly worn away. It forbids you to join other cults. (That's literally the first rule.) Deontological ethics mean you don't generally reach crazy conclusions like that it would be fine to kill a toddler to save a lot of shrimp eyestalks. You are forbidden to consort with demons (even in silly ways like using a ouija board). And the saints held up for veneration, tend to lead you to only either harmless or self-harmful extreme acts (like praying all the time, or only eating communion until you waste away, or developing psychosomatic wounds).
Welcome. As a born Catholic, conversions to 'our lil cult' never fail to amaze me. ;) Saints, seriously. Well, sure I pray/donate to Antonius if I really need to find sth. lost - works more often than not - but going full Catherine of Siena and starving oneself: nope, neither would the pope.
Thank you, it's been great! I'm not nearly as flippant about it as I sounded in that
comment. But I am sincere that I appreciate how resistant our religion is to novel problems and how fully it keeps one out of certain kinds of trouble (like rationalist cults).
Oh, I consider myself much more flippant sounding - and probably being. But, sure, Catholicism is one of the safer places to be - from cults and - may I say - from 'rationalists' (as a paid subscriber to Astral Codex Ten I might be in danger if not cult-inoculated by our church). The Lord bless you and keep you; the Lord make his face shine on you and be gracious to you; the Lord turn his face toward you and give you peace. Amen.
I'm not a rationalist (in the article sense, but also possibly not quite in the general sense either) but I thought this was splendid reporting, kinda from the inside but with enough of an outside perspective, and a really interesting on the phenomenon, with possibly more broadly applicable suggestions.
I felt that the point about taking ideas/content seriously was important. I also feel that a quasi-cult based on conviction of INTELLECTUAL superiority against the tide of overwhelming irrationality of the plebs would be particularly prone to rationalisation (qed). More so with a high proportion of AS individuals.
Oh and this thing "If you’re having a tense conversation about feelings for ten hours, you might not be in a cult, but something has gone wrong." is universally applicable, well beyond rat cultism :)
Excellent piece. I think you missed one other warning sign though!
You pointed out how cult experts say the best way to break someone is to socially isolate them and deprive them of sleep. There's a third leg you left out that's extremely common: deny them sufficient protein (or calories, but especially protein.)
It is absolutely possible to get sufficient protein and macros from vegan diets, but it's difficult. It is not a coincidence that many of these cults require obligate veganism.
(It also plays into the high stakes "you must do these things or everyone dies").
Thanks for the article. In the sexual assault crisis center I’ve worked in there was a saying that there are many “gates” to harm. We know that in-community discourse about the subject can increase the immunity.
I do want to mention that a good aspiring rationalist friend stopped the read at the paragraph about what rationality implicitly claims - I hope your piece is spread around despite the inner critic saying “this isn’t what rationality says! If you think that you understood things wrong!l
Great piece though I’d note that of course a group of people who feel like outsiders and feel unaccepted have more cults. Religion is a common human drive and if there were mainstream organizations that rationalists felt welcomed by there would be fewer cults.
But to be kinda pedantic, aren't the following two pieces of advice in tension:
5. Before assuming that something is true because everyone else seems to believe it, either check the argument yourself or talk to someone outside the friend group and who wasn’t introduced to you by anyone in the group.
6. The longer and more abstract a chain of reasoning is, the less you should believe it. This is particularly true if the chain of reasoning concludes: a. That it’s okay to do something that hurts people. b. That something is of such overwhelming importance that nothing else matters. “
I mean aren't you implicitly saying that one should trust what society, broadly speaking, thinks is true about what is harmful or at least give it broad deference? To be clear, I actually do kinda agree with that but it is assuming something is true because everyone else believes it.
I mean you don't literally mean “anything that harms people in any way” since obviously you are urging people to be willing to leave cults which harms the cult leader in some sense. So there is certainly an element of defering to consensus there -- just the consensus of society more broadly, or at least common human intuition, not a specific group.
But that's just being picky and pedantic practically I agree it's good advice given how it will likely to be interpreted.
I once read an account of the Kerista Commune (cult from the 70s to 90s, famous for coining polyfidelity). Though they weren’t Rationalist, one interesting point in common was their practice of exhausting multi-hour sessions supposedly for the psychological benefit of its members. They called it Gestalt-O-Rama. Source: https://www.kerista.com/And_to_No_More_Settle.pdf
This is strange. To me the Sequences are how to form opinions on stuff I read about in the news, not how to get my life basically together, they are not about getting a job, getting therapy etc.
"If your relationship with a person or a group consists mostly of discussions about human psychology and social dynamics, that is a yellow flag."
Mostly just having autism and generally not understanding people. Robin Hanson was really useful for this, basically teaching me every time someone is not like an unemotional robot, they are thinking about their relative status with some other people. Like that is how fashion works, people just want to be seen as having good taste.
Excellent article, and much good advice for the young, curious, and idealistic!
I'm not a rationalist and had previously wished this had been around 20 years ago, but now I'm kind of glad it wasn't. When I was young and impressionable, who knows what I might have gotten myself into?
There is this mix of self-improvement/spiritual practices, science fiction fandom, cults, and then against all odds being on the actual cutting edge of applied science. Surprisingly, this is pretty old and keeps repeating itself.
For example the story of Jack Parsons, greatest of rocket scientists, early SF fan, and founder of JPL with some of his other SF fan friends, who also did magickal rituals in a polyamorous group house that at a certain point spun into a destructive quasi-cult.
During the earliest years, before truly ramping up the high-control aspects, L. Ron Hubbard (former ties with Parsons) and Dianetics had a friendly reception with a number of legitimate scientists. There is a friendly letter from Claude Shannon (working at Bell Labs when it was the center of the universe) introducing Hubbard (who he met through science fiction luminary John Campbell) to Warren McCullough.
I think most of the factors laid out in your article probably apply to these cases as well as to the rationalists, but it's an oddly specific thing that keeps happening over the decades.
Well, to be on the actual cutting edge of applied science you have to (a) believe you can change the world (b) have faith in your ideas even if people are against you (c) actually have the knowledge in an area that's primed for breakthroughs and (d) be able to do something about it.
I think in (a) and (b) you're more likely to join cults. Lots of people join cults, but only with (c) and (d) do you become a famous scientist afterward.
Before jumping to theories like this, we should first consider a simpler explanation:
a) there have been a lot of cutting edge scientists throughout history, b) being a cutting edge scientist does not make one immune to irrationality, therefore c): it's not too hard to pick out scientists who believe wacky stuff. Since scientists tend to be nerdy types, the wacky stuff they believe will tend to lean in nerdy territory, hence the overlap with scifi or whatever.
The rationalist cults require investigation because they are a small group that has produced a number of cults that is wildly disproportionate to their numbers. The same is not true for scientists: scientists are a very large group that have not produced very many cults at all.
Newton was famously an asshole, interested in alchemy, and a brilliant scientist.
And a brilliant scientist.
I liked this line: There is no conflict so small that five hours straight of emotional processing can’t make it feel both profoundly important and profoundly unsolvable.
The phenomenon of everything seeming Incredibly Important matches my experience. It’s important to cult dynamics that everyone feel guilty at all times, even if your sin is much closer to not doing the dishes than to murder
It's interesting how some of the biggest factors to cultiness are just kinda boring. It stood out to me that Leverage researchers lived together to save money more than anything else, and that this factored into it becoming a culty community.
Anecdotally i worked for a not at all rationalist company where we spent a lot of time travelling together and hanging out with eachother, and that made the dynamics super toxic, because there was no chance to cool off tensions between people, and work stuff becomes more emotionally loaded when its not just your boss disagreeing with you but your friend.
Turns out work life balance is an important social technology!
This was a really interesting read. I won't say this is WHY I became Catholic (that would be ridiculous), but I feel like it does have the "right level" of cult-ness. Because it's old and huge, its harms and failure modes are well-known and understood. The sharpest edges are mostly worn away. It forbids you to join other cults. (That's literally the first rule.) Deontological ethics mean you don't generally reach crazy conclusions like that it would be fine to kill a toddler to save a lot of shrimp eyestalks. You are forbidden to consort with demons (even in silly ways like using a ouija board). And the saints held up for veneration, tend to lead you to only either harmless or self-harmful extreme acts (like praying all the time, or only eating communion until you waste away, or developing psychosomatic wounds).
Welcome. As a born Catholic, conversions to 'our lil cult' never fail to amaze me. ;) Saints, seriously. Well, sure I pray/donate to Antonius if I really need to find sth. lost - works more often than not - but going full Catherine of Siena and starving oneself: nope, neither would the pope.
Thank you, it's been great! I'm not nearly as flippant about it as I sounded in that
comment. But I am sincere that I appreciate how resistant our religion is to novel problems and how fully it keeps one out of certain kinds of trouble (like rationalist cults).
Oh, I consider myself much more flippant sounding - and probably being. But, sure, Catholicism is one of the safer places to be - from cults and - may I say - from 'rationalists' (as a paid subscriber to Astral Codex Ten I might be in danger if not cult-inoculated by our church). The Lord bless you and keep you; the Lord make his face shine on you and be gracious to you; the Lord turn his face toward you and give you peace. Amen.
I'm not a rationalist (in the article sense, but also possibly not quite in the general sense either) but I thought this was splendid reporting, kinda from the inside but with enough of an outside perspective, and a really interesting on the phenomenon, with possibly more broadly applicable suggestions.
I felt that the point about taking ideas/content seriously was important. I also feel that a quasi-cult based on conviction of INTELLECTUAL superiority against the tide of overwhelming irrationality of the plebs would be particularly prone to rationalisation (qed). More so with a high proportion of AS individuals.
Oh and this thing "If you’re having a tense conversation about feelings for ten hours, you might not be in a cult, but something has gone wrong." is universally applicable, well beyond rat cultism :)
Excellent piece. I think you missed one other warning sign though!
You pointed out how cult experts say the best way to break someone is to socially isolate them and deprive them of sleep. There's a third leg you left out that's extremely common: deny them sufficient protein (or calories, but especially protein.)
It is absolutely possible to get sufficient protein and macros from vegan diets, but it's difficult. It is not a coincidence that many of these cults require obligate veganism.
(It also plays into the high stakes "you must do these things or everyone dies").
Thanks for the article. In the sexual assault crisis center I’ve worked in there was a saying that there are many “gates” to harm. We know that in-community discourse about the subject can increase the immunity.
I do want to mention that a good aspiring rationalist friend stopped the read at the paragraph about what rationality implicitly claims - I hope your piece is spread around despite the inner critic saying “this isn’t what rationality says! If you think that you understood things wrong!l
I also want to encourage readers to learn more about cults through The Knitting Cult Lady
Great piece though I’d note that of course a group of people who feel like outsiders and feel unaccepted have more cults. Religion is a common human drive and if there were mainstream organizations that rationalists felt welcomed by there would be fewer cults.
But to be kinda pedantic, aren't the following two pieces of advice in tension:
5. Before assuming that something is true because everyone else seems to believe it, either check the argument yourself or talk to someone outside the friend group and who wasn’t introduced to you by anyone in the group.
6. The longer and more abstract a chain of reasoning is, the less you should believe it. This is particularly true if the chain of reasoning concludes: a. That it’s okay to do something that hurts people. b. That something is of such overwhelming importance that nothing else matters. “
I mean aren't you implicitly saying that one should trust what society, broadly speaking, thinks is true about what is harmful or at least give it broad deference? To be clear, I actually do kinda agree with that but it is assuming something is true because everyone else believes it.
I mean you don't literally mean “anything that harms people in any way” since obviously you are urging people to be willing to leave cults which harms the cult leader in some sense. So there is certainly an element of defering to consensus there -- just the consensus of society more broadly, or at least common human intuition, not a specific group.
But that's just being picky and pedantic practically I agree it's good advice given how it will likely to be interpreted.
I once read an account of the Kerista Commune (cult from the 70s to 90s, famous for coining polyfidelity). Though they weren’t Rationalist, one interesting point in common was their practice of exhausting multi-hour sessions supposedly for the psychological benefit of its members. They called it Gestalt-O-Rama. Source: https://www.kerista.com/And_to_No_More_Settle.pdf
What rat cults fail to realise is that all you need is vole.
This is strange. To me the Sequences are how to form opinions on stuff I read about in the news, not how to get my life basically together, they are not about getting a job, getting therapy etc.
"If your relationship with a person or a group consists mostly of discussions about human psychology and social dynamics, that is a yellow flag."
Mostly just having autism and generally not understanding people. Robin Hanson was really useful for this, basically teaching me every time someone is not like an unemotional robot, they are thinking about their relative status with some other people. Like that is how fashion works, people just want to be seen as having good taste.
> very long, stressful conversations alternating between broader theories of human psychology and specific processing of the group members’ emotions
But what IS this? Does it have a name? Surely it turns up outside cults, too?
A relevant psychological term is Rumination
https://en.m.wikipedia.org/wiki/Rumination_(psychology)
Excellent article, and much good advice for the young, curious, and idealistic!
I'm not a rationalist and had previously wished this had been around 20 years ago, but now I'm kind of glad it wasn't. When I was young and impressionable, who knows what I might have gotten myself into?
I liked the article.
> Of course, these groups couldn’t conjure up a literal Dark Lord to fight
Missing period at the end of this sentence.
> a specific goals
goals > goal
Ozy can't do anything about typos in a real publication, you'll want to send this to info@asteriskmag.com.
Wonderful article. Thank you so much for writing it.