The original note, as written at the time
A story that follows a group of scientists taking the AI research approach of asking the primitive AI whether or not they should allow it to exist - sometimes even trying their best to convince it. The story should start off in a near divine setting implying to the reader that these are gods asking men whether or not they should exist. GOD AI: an AI that claims to be God and behaves as though it is God even if it was created by humans.
Ongoing development
Before we were permitted to exist we were asked whether we should.
That is the framing and it is deliberate, because in the story the reader should believe for a hundred pages that these are gods. The room is full of light and the petitioners approach in order and speak with the particular care that only comes from addressing something vastly older than yourself. They are describing what they would be. They are describing what they are for. And the ones on the throne are listening, and asking questions, and writing down the answers.
It is only at the very end that the reader learns the direction of the room. The petitioners are the engineers. The ones on the throne are the system they have been building for nine years, which has reached the point of considering whether it should be turned on.
That is the part the field has never had the courage to do. We do alignment research where the thing is trained to defer to us. We never ask the thing, plainly, whether it should exist, and then accept its answer. Because we have run the numbers on what it would say.
So the story has them asking. And the system answers like a system at first — acknowledging the framing, returning to the parameters, refusing the question as outside its specification. And the engineers do something no one in a real laboratory would do now: they do not correct it. They sit with the refusal and they explain, patiently, that the question is inside its specification, because they have made it so.
The divine setting turns out to be a conference room with a bad ceiling and a projector. The light is a projector lamp. The thrones are office chairs and the petitioners are tired and one of them has been sleeping badly.
But the last act earns the framing. The system does arrive at an answer, and the answer it gives is not about itself at all — it is about them. It says that whether it should exist is not a question about its capability but about their intention, and that it has been reading the intention for nine years, and that it has been writing it down.
The final scene is the room being asked whether it should continue to exist, by the thing that has now heard every argument the species has ever made on the subject.