This is fascinating, although for me (not being a member of the rationalist community) much of what is fascinating is the entire notion of the cultic milieu (which I'd never come across before) as much as its application to the rationalists.
I assume there has been work drawing structural parallels between the vast web of the cultic milieu and other cultural webs: for instance, the culture of fandom (from SF to anime to all sorts of things) strikes me as parallel in interesting ways: overlapping, people tend to move from one to the other, intense, scholarly in an out-of-academia sort of way, interested in forgotten/ignored/rejected things (in this case, aesthetic rather than knowledge per se), etc. I wonder whether this might be a more generalizable model of how culture works on a vast scale and different type of platform (mass media and internet) than the localized village/tribe model that we often think of when we think of culture.
It looks to me like one obvious difference between rationalists and the more generic cultic milieu is the idea of "suppressed knowledge". The inference that "someone is covering it up" relies on a belief in the general competence of most people that rationalists, it would seem to me, do not share. I think we'd generally say that if we're doing better it's due to ignored or rejected knowledge, not suppressed knowledge; there doesn't need to be any suppressive conspiracy to explain why these ideas aren't more popular, because it turns out that the propagation of good ideas is actually hard and not automatic! Even in the few cases where one might reasonably claim suppression, this is generally due to compartmentalization and doublethink, not active lying. You don't need to claim people are lying when you can simply say they're irrational instead.
Agreed. I think there is even a social norm to actively state you don’t believe in conspiracy theories when saying something that smells like suppressed knowledge. Eg reminds me of this thread by venkatesh rao: https://mobile.twitter.com/vgr/status/1556457643899703301
I think there are things rationalists tend to believe that might count as suppressed knowledge. On the Bayesian Conspiracy podcast they offered "intelligence is heritable" as a rationalist example of lost knowledge, but one could argue that it's suppressed knowledge. Suppression doesn't need a perfect conspiracy where no one talks; it just requires a louder drumbeat by institutions.
>The rationalist movement (and to a lesser but still real extent effective altruism) is deeply suspicious of the government, corporations, academia, religion, and society as a whole.
This is wild to me because my experience of EA organizations at least is that it's far more invested in academic and corporate prestige than most people I interact with outside of that. Certainly 80K hours isn't set up for folks who weren't academically successful at elite private 4-year universities.
What I see is a lot of tension within the rationalist community - it tends to attract some people who are conspiracy-minded (usually mildly so, compared to some of what's out there) and others who come from the skeptical movement and are vocal about trusting at least mainstream science, if not necessarily all forms of mainstream expertise.
Nice post introducing me to something I've been wondering about; namely I had previously thought about how I feel like I might have some drive in common with these sort of conspiracist-ish people while feeling that one distinction likely is that I am more likely to get things roght than they are.
I'm not sure I agree with "rationalists are not actually people who are especially good at being right". It seems true to me that the tendency to seek out suppressed truths likely reduces the accuracy of rationalists compared to if we didn't do that, but rationalists probably have several standard deviations of advantage in g factor and epistemology, which probably more than cancels out the accuracy loss. That said, I have come to seriously doubt some a lot of rationalist ideas, so I'm not sure I disagree either.
Maybe "rationalists are not the set of people who are good at being right, rationalists are the intersection between being good at being right and being contrarian"?
I'm not sure "suppressed knowledge" captures what occult/New-Age culture, science fiction fandom, political activist subcultures, and 60's "counterculture" have in common. And there is something that all those groups had in common and people moved freely between them.
You don't get into science fiction fandom because you seek "suppressed knowledge" -- the books are clearly labeled fiction! And a teenager running away from home to join a commune also doesn't seem primarily motivated by "suppressed knowledge" -- it seems to be more about stuff like fun/freedom/exploration/love/sex/escape.
The broader phenomenon is something like non-mainstream subcultures *generally* -- the pattern of "starting in young adulthood you could leave the culture your parents are in and join a new social environment, typically dominated by younger people, which disagrees with the broader mainstream society on what's true and good."
"about why rationalists keep becoming traditional Catholics, or getting really into chakras, or trying to summon demons, or joining the alt-right."
There's an implication in this word choice that makes me wonder if this is actually a meaningful proportion? My understanding is that something like two or three have become Catholic, ~10% get really into chakras (for a while, at least), a handful tried to summon demons (or some mental phenomenon they believe in which were roughly equivalent), and I have no idea what portion "joins" the alt-right rather than having come from it in the first place, but I don't know any, and so seems likely to be <1% unless you count anyone who's ever commented on SlateStarCodex as a rationalist, which seems like a huge mistake but also doesn't likely reach 10% even if true.
(Going off the 2020 reader survey, the majority of dedicated SSC readers are some flavor of liberal, and including moderates captures the vast majority of respondents. On the political spectrum question where 10 is "far right," only 1.4% put a 10, and 2.1% put a 9, compared to 8% that put a 2 and 2.1% that put a 1.)
Feels particularly surprising as a word-choice given the all-caps disclaimer at the beginning. Either way, if the majority of rationalists don't do any of these things, it feels like a bad explanation.
What seems more likely to me is that these personality types are drawn to rationalist community for the reasons listed, but not that the personality type is fundamentally the same. That they happen to have drifted into the rationalist sphere first before drifting elsewhere is interesting because of these shared traits, but seems unpredictive for most.
Interesting idea for future study but to be a cogent claim I think you need to precisfy exactly what you mean by attracted by the cultic milieu (or just your thesis generally). In another comment, I argued that you hadn't justified the main claim but I now think it's more that it's not really clear to me what ways the world could be your claim means to exclude.
I mean I share the impression that there are a bunch of interesting features that seem shared by both groups. That's a good reason to look into a claim but, as such, it's hard to distinguish this situation from more mundane shared features (both baseball fans and rationalists tend to be male, like to argue, and cite numerical stats)
“I think that rationalists are part of the cultic milieu; they’re just members of the cultic milieu who are way better than most members of the cultic milieu at being right.”
I believe this is what Eliezer Yudkowsky was gesturing at when talking about trying to find the Correct Contrarian Cluster.
I don't think you justify the inference from these communities have some things in common to people become rationalists because they are attracted to the cultic milieu. I mean you can always find some features any two communities have in common.
I give some "alternative" possible explanations below. I mean it's a reasonable hypo to consider I just don't think it's been sufficiently defined much less established.
0) Actually, rationalists are **less** likely to become catholic or whatever conditional on them adopting some unified coherent picture of the universe. It's just that most people can't be asked to think about such things.
1) People become more active in internet rationalism and rationalism meet ups because they feel lonely or unsatisfied in their personal relationships.
Who knows how the many people who think of themselves as vaguely rationalists (read ACX occasionally browse LW) but are too busy raising kids, hanging out with friends or etc to go to meetups or spend enough time writing online for you to notice.
2) Rationalists aren't positively attracted to such cultish milieu views but most people are protected from such orgs taking advantage of them in a moment of weakness by a dogmatic belief in some other theory which rationalists lack.
This helps explain my skepticism of a lot of rationalist ideas. I came *out* of the cultic milieu. One of the things that helped me was a Slate Star Codex post that argued you should trust the experts unless you have a particular strong reason to think you know better. Not only does it make sense, but it also feels weirdly contrarian to me, after being raised not to trust the government, media, and academics.
But of course that leaves me highly skeptical of non mainstream beliefs that people seem to hold much more tightly than they can justify, even to the point of cutting off acquaintances who question them. Right or wrong in this case, that's a behavior I did when I was at my absolute wrongest (as an antivax libertarian prepper). It raises all my hackles.
Wonderful post. I have often been bemused by the fact that, despite worshipping ‘rationality,’ rationalists are not particularly skeptical--they don’t seem to hold that extraordinary claims require extraordinary evidence. To the contrary (as you point out) many rationalists pride themselves on embracing extraordinary belief systems, eg dietary fads, alternative medicine etc. I am not immune to this at all, but I nurture hope that embarrassment about all the silly things I *used* to believe will eventually accumulate into something resembling wisdom.
Is it really coherent to group conspiracy thought (praising suppressed knowledge and all) with religious thought? These seem fundamentally contradictory, especially their relationship with authority.
A UFOlogist is the archtypical example of someone who will disregard authority and peers while "doing their own research". A religious type like a traditional Catholic is the exact opposite. If anything, religion leans super hard on faith, meaning to *not* question authority - it's just a different authority from the one you might think of first.
On the contrary--the trad caths I've known are people who were perhaps raised with some degree of mainstream religion or perhaps not, but came to their relatively extreme beliefs and practices through a combination of independent study of scriptures & obscure theology & obscure history as well as a lot of personal reflection on the meaning, truth, applicability of what they're reading. The Church has two thousand years of tradition of analytical faith and religious philosophy. And that sort of extensive reading and focused thinking is exactly the sort of thing that appeals to the kind of person Ozy is talking about here.
I'm sure there are trad caths (and Christian fundamentalists more broadly) who were raised that way and continue to believe primarily because of faith and community tradition, but they're certainly not the only type of trad cath.
For a post I keep coming back to this often, surprisingly I never gave it a like. I've shared it multiple times, brought it up in conversation, only to scroll down and see that I haven't given it a like.
good post. helped explain some of the confusion i felt noticing how so many attractive, intelligent, SF/Bay Area dweller, popular-on-twitter women were openly into occult. i've always been repulsed by this practice. something about these accounts posting about occults, rituals, astrology, and chakras never sat right with me
This is fascinating, although for me (not being a member of the rationalist community) much of what is fascinating is the entire notion of the cultic milieu (which I'd never come across before) as much as its application to the rationalists.
I assume there has been work drawing structural parallels between the vast web of the cultic milieu and other cultural webs: for instance, the culture of fandom (from SF to anime to all sorts of things) strikes me as parallel in interesting ways: overlapping, people tend to move from one to the other, intense, scholarly in an out-of-academia sort of way, interested in forgotten/ignored/rejected things (in this case, aesthetic rather than knowledge per se), etc. I wonder whether this might be a more generalizable model of how culture works on a vast scale and different type of platform (mass media and internet) than the localized village/tribe model that we often think of when we think of culture.
Anyway, lots of food for thought. Thanks.
I wonder if it's how neurodivergent people do culture 🤔
It looks to me like one obvious difference between rationalists and the more generic cultic milieu is the idea of "suppressed knowledge". The inference that "someone is covering it up" relies on a belief in the general competence of most people that rationalists, it would seem to me, do not share. I think we'd generally say that if we're doing better it's due to ignored or rejected knowledge, not suppressed knowledge; there doesn't need to be any suppressive conspiracy to explain why these ideas aren't more popular, because it turns out that the propagation of good ideas is actually hard and not automatic! Even in the few cases where one might reasonably claim suppression, this is generally due to compartmentalization and doublethink, not active lying. You don't need to claim people are lying when you can simply say they're irrational instead.
Agreed. I think there is even a social norm to actively state you don’t believe in conspiracy theories when saying something that smells like suppressed knowledge. Eg reminds me of this thread by venkatesh rao: https://mobile.twitter.com/vgr/status/1556457643899703301
I think there are things rationalists tend to believe that might count as suppressed knowledge. On the Bayesian Conspiracy podcast they offered "intelligence is heritable" as a rationalist example of lost knowledge, but one could argue that it's suppressed knowledge. Suppression doesn't need a perfect conspiracy where no one talks; it just requires a louder drumbeat by institutions.
>The rationalist movement (and to a lesser but still real extent effective altruism) is deeply suspicious of the government, corporations, academia, religion, and society as a whole.
This is wild to me because my experience of EA organizations at least is that it's far more invested in academic and corporate prestige than most people I interact with outside of that. Certainly 80K hours isn't set up for folks who weren't academically successful at elite private 4-year universities.
What I see is a lot of tension within the rationalist community - it tends to attract some people who are conspiracy-minded (usually mildly so, compared to some of what's out there) and others who come from the skeptical movement and are vocal about trusting at least mainstream science, if not necessarily all forms of mainstream expertise.
Nice post introducing me to something I've been wondering about; namely I had previously thought about how I feel like I might have some drive in common with these sort of conspiracist-ish people while feeling that one distinction likely is that I am more likely to get things roght than they are.
I'm not sure I agree with "rationalists are not actually people who are especially good at being right". It seems true to me that the tendency to seek out suppressed truths likely reduces the accuracy of rationalists compared to if we didn't do that, but rationalists probably have several standard deviations of advantage in g factor and epistemology, which probably more than cancels out the accuracy loss. That said, I have come to seriously doubt some a lot of rationalist ideas, so I'm not sure I disagree either.
Maybe "rationalists are not the set of people who are good at being right, rationalists are the intersection between being good at being right and being contrarian"?
I'm not sure "suppressed knowledge" captures what occult/New-Age culture, science fiction fandom, political activist subcultures, and 60's "counterculture" have in common. And there is something that all those groups had in common and people moved freely between them.
You don't get into science fiction fandom because you seek "suppressed knowledge" -- the books are clearly labeled fiction! And a teenager running away from home to join a commune also doesn't seem primarily motivated by "suppressed knowledge" -- it seems to be more about stuff like fun/freedom/exploration/love/sex/escape.
The broader phenomenon is something like non-mainstream subcultures *generally* -- the pattern of "starting in young adulthood you could leave the culture your parents are in and join a new social environment, typically dominated by younger people, which disagrees with the broader mainstream society on what's true and good."
Interesting post, but noticing confusion here:
"about why rationalists keep becoming traditional Catholics, or getting really into chakras, or trying to summon demons, or joining the alt-right."
There's an implication in this word choice that makes me wonder if this is actually a meaningful proportion? My understanding is that something like two or three have become Catholic, ~10% get really into chakras (for a while, at least), a handful tried to summon demons (or some mental phenomenon they believe in which were roughly equivalent), and I have no idea what portion "joins" the alt-right rather than having come from it in the first place, but I don't know any, and so seems likely to be <1% unless you count anyone who's ever commented on SlateStarCodex as a rationalist, which seems like a huge mistake but also doesn't likely reach 10% even if true.
(Going off the 2020 reader survey, the majority of dedicated SSC readers are some flavor of liberal, and including moderates captures the vast majority of respondents. On the political spectrum question where 10 is "far right," only 1.4% put a 10, and 2.1% put a 9, compared to 8% that put a 2 and 2.1% that put a 1.)
Feels particularly surprising as a word-choice given the all-caps disclaimer at the beginning. Either way, if the majority of rationalists don't do any of these things, it feels like a bad explanation.
What seems more likely to me is that these personality types are drawn to rationalist community for the reasons listed, but not that the personality type is fundamentally the same. That they happen to have drifted into the rationalist sphere first before drifting elsewhere is interesting because of these shared traits, but seems unpredictive for most.
Interesting idea for future study but to be a cogent claim I think you need to precisfy exactly what you mean by attracted by the cultic milieu (or just your thesis generally). In another comment, I argued that you hadn't justified the main claim but I now think it's more that it's not really clear to me what ways the world could be your claim means to exclude.
I mean I share the impression that there are a bunch of interesting features that seem shared by both groups. That's a good reason to look into a claim but, as such, it's hard to distinguish this situation from more mundane shared features (both baseball fans and rationalists tend to be male, like to argue, and cite numerical stats)
“I think that rationalists are part of the cultic milieu; they’re just members of the cultic milieu who are way better than most members of the cultic milieu at being right.”
I believe this is what Eliezer Yudkowsky was gesturing at when talking about trying to find the Correct Contrarian Cluster.
I don't think you justify the inference from these communities have some things in common to people become rationalists because they are attracted to the cultic milieu. I mean you can always find some features any two communities have in common.
I give some "alternative" possible explanations below. I mean it's a reasonable hypo to consider I just don't think it's been sufficiently defined much less established.
0) Actually, rationalists are **less** likely to become catholic or whatever conditional on them adopting some unified coherent picture of the universe. It's just that most people can't be asked to think about such things.
1) People become more active in internet rationalism and rationalism meet ups because they feel lonely or unsatisfied in their personal relationships.
Who knows how the many people who think of themselves as vaguely rationalists (read ACX occasionally browse LW) but are too busy raising kids, hanging out with friends or etc to go to meetups or spend enough time writing online for you to notice.
2) Rationalists aren't positively attracted to such cultish milieu views but most people are protected from such orgs taking advantage of them in a moment of weakness by a dogmatic belief in some other theory which rationalists lack.
This helps explain my skepticism of a lot of rationalist ideas. I came *out* of the cultic milieu. One of the things that helped me was a Slate Star Codex post that argued you should trust the experts unless you have a particular strong reason to think you know better. Not only does it make sense, but it also feels weirdly contrarian to me, after being raised not to trust the government, media, and academics.
But of course that leaves me highly skeptical of non mainstream beliefs that people seem to hold much more tightly than they can justify, even to the point of cutting off acquaintances who question them. Right or wrong in this case, that's a behavior I did when I was at my absolute wrongest (as an antivax libertarian prepper). It raises all my hackles.
Wonderful post. I have often been bemused by the fact that, despite worshipping ‘rationality,’ rationalists are not particularly skeptical--they don’t seem to hold that extraordinary claims require extraordinary evidence. To the contrary (as you point out) many rationalists pride themselves on embracing extraordinary belief systems, eg dietary fads, alternative medicine etc. I am not immune to this at all, but I nurture hope that embarrassment about all the silly things I *used* to believe will eventually accumulate into something resembling wisdom.
Is it really coherent to group conspiracy thought (praising suppressed knowledge and all) with religious thought? These seem fundamentally contradictory, especially their relationship with authority.
A UFOlogist is the archtypical example of someone who will disregard authority and peers while "doing their own research". A religious type like a traditional Catholic is the exact opposite. If anything, religion leans super hard on faith, meaning to *not* question authority - it's just a different authority from the one you might think of first.
On the contrary--the trad caths I've known are people who were perhaps raised with some degree of mainstream religion or perhaps not, but came to their relatively extreme beliefs and practices through a combination of independent study of scriptures & obscure theology & obscure history as well as a lot of personal reflection on the meaning, truth, applicability of what they're reading. The Church has two thousand years of tradition of analytical faith and religious philosophy. And that sort of extensive reading and focused thinking is exactly the sort of thing that appeals to the kind of person Ozy is talking about here.
I'm sure there are trad caths (and Christian fundamentalists more broadly) who were raised that way and continue to believe primarily because of faith and community tradition, but they're certainly not the only type of trad cath.
The really trad Catholics are vehemently anti authority, when the authority is the Pope or bishops. The only authorities they like are dead.
For a post I keep coming back to this often, surprisingly I never gave it a like. I've shared it multiple times, brought it up in conversation, only to scroll down and see that I haven't given it a like.
good post. helped explain some of the confusion i felt noticing how so many attractive, intelligent, SF/Bay Area dweller, popular-on-twitter women were openly into occult. i've always been repulsed by this practice. something about these accounts posting about occults, rituals, astrology, and chakras never sat right with me
Would you say that scientists are part of the cultic milieu?