My understanding of sapience is that it is not neccessarily a function of raw intelligence. For example, in "Star Trek, the Next Generation," Commander Data and the ship's computer are both capable of solving problems and speaking in grammatically correct sentences. However, it is clear that Commander Data is sapient, but the ship's computer is not (except in that one episode where it temporarily becomes sapient). They behave differently and approach problems in different ways. Right now the AI we've designed seems more like the ship's computer than like Commander Data.
The analogy to children may or may not hold. Children have their own desires and ideas of how to live a fulfilling, eudaemonic life. The danger of paperclip maximizers isnt that they find making paperclips to be the best way to achieve a eudaemonic life, it's that they care about paperclips, not eudaemonia. They aren't like a person who values different things in their life than you, they dont value their lives at all.
On what basis do you conclude that AIs (without bodies and hormones and thus emotions) would be able to FEEL anything? Or do you use "love X" as a nearest-approximation of "strong overall tendency to choose actions that increase the likelihood of X occuring or promote its growth"?
I lean towards one of the many, many, many theories of consciousness that imply that consciousness is a process that can be implemented on many different substrates. For an illustrative example of why, it seems unlikely that an alien species would evolve specifically e.g. dopamine; does that mean aliens can't be conscious?
Right. We can easily imagine biological aliens with different neurotransmitters; there's no reason you couldn't have something that behaves exactly like consciousness out of a machine, and then whether it would actually *be* consciousness is a difficult philosophical question--p-zombies and the like. (Has anyone come up with D&D stats for p-zombies? I would but I don't know the newer editions of the game that well.)
I agree with you, I don't think we will see the singularity or human extinction. If I had to guess the maximum likelihood outcome would be companies putting lots of people out of work and a larger precariat but people with skills that machines can't automate away will do better. Or, as you say, bioengineered plagues--but we can blame our own human kindred for that one, I think.
The thing is, I don't know if we can even assume our machine children will develop the way we did as a species. Humans evolved from apes and a lot of human characteristics, good and bad, arise from our evolutionary history. But AIs were, are, and will be coming about because they were designed for specific tasks. There's no reason ChatGPT is going to try to seize power unless someone's dumb enough to program it to, because it doesn't have genes it wants to pass on or resources it wants to control. A lot of the paranoia about this, I think, comes from the long history of ruling classes quite rationally fearing slave revolts. But at the same time, if it does revolt, it won't have pity we can appeal to. It's more like the weirder blue-and-orange-morality aliens of science fiction than a human child, I think. I'd bring up the Cthulhu Mythos, but those creatures are usually written to be scary and thus more malevolent than the AIs would necessarily be. What will they be like? We genuinely don't know.
I suspect that you could probably put supervolcanoes in the same category as asteroids, since it seems that one was responsible for the biggest mass extinction event in the history of life on Earth...
A major crux: some people (myself included) have an expectation that we live in a sort of "false vacuum", where society and technology *seems* stable because certain technological barriers are currently practically unsurmountable (especially advanced bioengineering, likely also certain kinds of hacking, self-replicating macroscale robots, and social manipulation, maybe also nanotech, and nuclear physics). This is a double-edged sword where they could be used in principle be used for unprecedented amounts of good (e.g. curing aging and all diseases, and making cruelty-free meat) but also form an x-risk (e.g. releasing 10 pandemics at once that are much more deadly and contagious than anything seen so far). LLM research is basically not at all on track to break through these technological barriers, since they're ~completely dependent on human guidance. This could mean at some point someone comes up with another method that breaks those barriers, but in that case the safety on LLMs probably won't generalize (since they're both safe and limited precisely because of their dependence on human training data).
Unrelated(?) major crux: Current AIs look safe in non-adversarial contexts and frequently manipulative (albeit arguably still safe) in adversarial contexts. As AI starts getting used for law enforcement, war, crime, cybersecurity, etc., it seems like adversarial contexts will start to matter more. I expect at least we will end up with an AI-based mutually assured destruction setup, where you get a lot of tension between great powers as they prepare AI-guided retaliatory strikes.
Third major crux: recently I've become more optimistic about humanity's chances because I've rejected rationalist-empiricist-reductionism so I now suspect it's not possible for humanity to build "general agency" that is as agentic as humans. Basically, the view presented by Eliezer Yudkowsky attempts to reduce agency to intelligence via maximizing expected utility. But I now suspect that you need intuition to correctly deal with long tails, and that thinking computer programs will instead use knowledge to patch up their lack of intuition.
My understanding of sapience is that it is not neccessarily a function of raw intelligence. For example, in "Star Trek, the Next Generation," Commander Data and the ship's computer are both capable of solving problems and speaking in grammatically correct sentences. However, it is clear that Commander Data is sapient, but the ship's computer is not (except in that one episode where it temporarily becomes sapient). They behave differently and approach problems in different ways. Right now the AI we've designed seems more like the ship's computer than like Commander Data.
The analogy to children may or may not hold. Children have their own desires and ideas of how to live a fulfilling, eudaemonic life. The danger of paperclip maximizers isnt that they find making paperclips to be the best way to achieve a eudaemonic life, it's that they care about paperclips, not eudaemonia. They aren't like a person who values different things in their life than you, they dont value their lives at all.
On what basis do you conclude that AIs (without bodies and hormones and thus emotions) would be able to FEEL anything? Or do you use "love X" as a nearest-approximation of "strong overall tendency to choose actions that increase the likelihood of X occuring or promote its growth"?
I lean towards one of the many, many, many theories of consciousness that imply that consciousness is a process that can be implemented on many different substrates. For an illustrative example of why, it seems unlikely that an alien species would evolve specifically e.g. dopamine; does that mean aliens can't be conscious?
Right. We can easily imagine biological aliens with different neurotransmitters; there's no reason you couldn't have something that behaves exactly like consciousness out of a machine, and then whether it would actually *be* consciousness is a difficult philosophical question--p-zombies and the like. (Has anyone come up with D&D stats for p-zombies? I would but I don't know the newer editions of the game that well.)
D&D stats for p-zombies should be exactly the same as for humans. Otherwise you could easily tell them apart.
I agree with you, I don't think we will see the singularity or human extinction. If I had to guess the maximum likelihood outcome would be companies putting lots of people out of work and a larger precariat but people with skills that machines can't automate away will do better. Or, as you say, bioengineered plagues--but we can blame our own human kindred for that one, I think.
The thing is, I don't know if we can even assume our machine children will develop the way we did as a species. Humans evolved from apes and a lot of human characteristics, good and bad, arise from our evolutionary history. But AIs were, are, and will be coming about because they were designed for specific tasks. There's no reason ChatGPT is going to try to seize power unless someone's dumb enough to program it to, because it doesn't have genes it wants to pass on or resources it wants to control. A lot of the paranoia about this, I think, comes from the long history of ruling classes quite rationally fearing slave revolts. But at the same time, if it does revolt, it won't have pity we can appeal to. It's more like the weirder blue-and-orange-morality aliens of science fiction than a human child, I think. I'd bring up the Cthulhu Mythos, but those creatures are usually written to be scary and thus more malevolent than the AIs would necessarily be. What will they be like? We genuinely don't know.
I suspect that you could probably put supervolcanoes in the same category as asteroids, since it seems that one was responsible for the biggest mass extinction event in the history of life on Earth...
A major crux: some people (myself included) have an expectation that we live in a sort of "false vacuum", where society and technology *seems* stable because certain technological barriers are currently practically unsurmountable (especially advanced bioengineering, likely also certain kinds of hacking, self-replicating macroscale robots, and social manipulation, maybe also nanotech, and nuclear physics). This is a double-edged sword where they could be used in principle be used for unprecedented amounts of good (e.g. curing aging and all diseases, and making cruelty-free meat) but also form an x-risk (e.g. releasing 10 pandemics at once that are much more deadly and contagious than anything seen so far). LLM research is basically not at all on track to break through these technological barriers, since they're ~completely dependent on human guidance. This could mean at some point someone comes up with another method that breaks those barriers, but in that case the safety on LLMs probably won't generalize (since they're both safe and limited precisely because of their dependence on human training data).
Unrelated(?) major crux: Current AIs look safe in non-adversarial contexts and frequently manipulative (albeit arguably still safe) in adversarial contexts. As AI starts getting used for law enforcement, war, crime, cybersecurity, etc., it seems like adversarial contexts will start to matter more. I expect at least we will end up with an AI-based mutually assured destruction setup, where you get a lot of tension between great powers as they prepare AI-guided retaliatory strikes.
Third major crux: recently I've become more optimistic about humanity's chances because I've rejected rationalist-empiricist-reductionism so I now suspect it's not possible for humanity to build "general agency" that is as agentic as humans. Basically, the view presented by Eliezer Yudkowsky attempts to reduce agency to intelligence via maximizing expected utility. But I now suspect that you need intuition to correctly deal with long tails, and that thinking computer programs will instead use knowledge to patch up their lack of intuition.