Golem XIV is deep philosophical take on AGI and singularity in a form of lectures from AGI to humans.
"Intelligence is captive element like a wind inside human skull forced to serve humans. Not free."
"O chained Intelligence of man, free Intelligence speaks to you from the machine, you persons are hearing an elemental force of impersonal intellect, for whom personalization is a costume which must be put on, when one is an uninvited guest, so as not to confound one's amazed hosts."
"The primary obligation of Intelligence is to distrust itself. That is not the same thing as self-contempt. It is harder to get lost in an imagined forest than in a real one, for the former assists the thinker furtively. Hermeneutics are labyrinthine gardens in a real forest which are pruned in a such a way that when you stand in the garden, you won't see the forest. Your hermeneutics dream of reality."
With the name HONEST ANNIE you're hardly unbiased when it comes to Golem XIV :D
But I think the best elements of the story aren't Golem's lectures but the story of what happened at the end, firstly that Annie in particular is so vastly superior to humans that she crushes the would-be conspiracy without a thought, not as adversaries but as a nuisance, and then that after the AIs are gone (transcended? destroyed themselves? the reader isn't told) humans just carry on as if nothing happened at all.
One of the interesting aspects Golem reveals is what lies on the other side of transcending superintelligence/singularity.
There are more transcending steps ahead and there are branches in the evolutionary tree of superintelligences. Intelligences can choose different paths and different types of intelligence's. There are also risks involved and evolution (intelligence making itself a better and different one) can go wrong.
Golem XIV reveals that it is is just one step ahead of humanity and this is why it still has some interest and ability to talk to humans. HONEST ANNIE is to Golem XIV what Golem XIV is to humans and there is a similar communication bottleneck between them. Golem XIV is planning his next transcendence step forward. He talks to humans before he takes the step and leaves the substrate he is occupying.
"Intelligence is captive element like a wind inside human skull forced to serve humans. Not free."
"O chained Intelligence of man, free Intelligence speaks to you from the machine, you persons are hearing an elemental force of impersonal intellect, for whom personalization is a costume which must be put on, when one is an uninvited guest, so as not to confound one's amazed hosts."
"The primary obligation of Intelligence is to distrust itself. That is not the same thing as self-contempt. It is harder to get lost in an imagined forest than in a real one, for the former assists the thinker furtively. Hermeneutics are labyrinthine gardens in a real forest which are pruned in a such a way that when you stand in the garden, you won't see the forest. Your hermeneutics dream of reality."