Dave5

Let’s start looking at this as the speed run for first-contact… our assumptions and values are never going to be universal and here is the perfect practice run.

AI safety, originally “friendly AI,” was conceived as the problem of how to create an agent that would be benevolent towards humanity in general, on the assumption that that agent would have godlike superpowers. This whole line of thinking is rife with faulty assumptions—not even necessarily about the technology, which has yet to come into being, but about humans, agency, values, and benevolence. 

“How do we make AI benevolent?” is a badly formulated problem. In its very asking, it ascribes agency to the AI that we don’t have to give it, that it may or may not even be able to acquire, and that is naturally ours in the first place. It implicitly ascribes all future moral agency to the AI. “How can we align AI with human values?” is also a badly formulated problem. It assumes that “human values” are or even should be universal and invariant, that they can be “figured out,” and that they need to be figured out to generate good outcomes from AI in the first place. 

https://www.palladiummag.com/2023/11/10/benevolent-ai-is-a-bad-idea/