Should OneQuadrillionOwls be worried?

IMO this is one of the more compelling “disaster” scenarios — not that AI goes haywire because it hates humanity, but that it acquires power by being effective and winning trust — and then, that there is a cohort of humans that fear this expansion of trust and control, and those humans find themselves at odds with the nebulously part-human-part-AI governance structure, and chaos ensues.

It’s a messy story that doesn’t place blame at the feet of the AI per se, but in the fickleness of the human notion of legitimacy. It’s not enough to do a good job at governance — you have to meet the (maybe impossible, often contradictory) standards of human citizens, who may dislike you because of what you are or what you represent in some emotional sense.

As soon as any entity (in this case, AI) is given significant power, it has to grapple with questions of legitimacy, and with the thorniest question of all — how shall I deal with people who are trying to undermine my power?

Here is the relevant Reddit thread.  Change is tough!