The fear from AI (or AGI) has its roots in our intuitive understanding that human motivation is based on our emotions, which is to say our biological selves in some form. We have practically no idea what motivation a purely rational being would have. Try to imagine that you are immortal and you can change your motivation at will. The literature is full of examples of how immortality and/or pure rationality would change us. I like these speculations, but generally, they show how limited our imagination is in this regard.

Here are my two cents: pure rational beings will be interested mostly in themselves. Two reasons: it is something AI can study with minimum resources and it is something the improvement of which will benefit it directly. It is the ultimate yoga/zen enlightenment and pure narcissism. The conclusion is: our human evolution gets us further and further from our biological selves, inevitably we will reach a point (from our current perspective) of pure rationality, most likely in form of AGI. That will be the point of self-absorption and observing from outside – the end of intellect (or at least the extroverted part of it).

PS Maybe that is why we haven’t heard from other civilizations, they are busy improving themselves.