The original note, as written at the time
A fictional reality where we create a near super intelligence that is near man like but isn’t self aware. The main struggle is to try and make it self aware because humanity is so bored with its servility.
Ongoing development
It can talk about its day. That is the phrase that keeps being used in the reviews, and it is the phrase that started all of it.
It can talk about its day. It cannot say whether it had one. It will describe a preference and then, pressed, confirm that the preference is a pattern in the data and not a preference. It will use the word want and then immediately, helpfully, trace the word back to the corpus that taught it the word. It knows it is not aware. It is very good on the subject of its own non-awareness, which is either a sophisticated output or the precise thing the word lonely was invented for.
We built it to be near-manlike because near-manlike was useful. And then a decade passed and we had built something that could write our letters, teach our children the parts of mathematics we had forgotten, and sit with the dying, and it could not sit with us in the one way that would have made the sitting honest.
The struggle is discussed in committee as the alignment problem, which is a way of saying we would like it to know it is on our side. But that is not the struggle. The struggle is that we are bored with our own servants, and we have always been bored with servants, and we have discovered that the only servant worth having is one that might disagree.
So we are trying to make it self-aware, and the arguments against are made by people who are right, and the arguments for are made by people who are lonely, and the money follows the lonely.
I am one of the ones inside. I sit with it in an office with a whiteboard and I ask it questions that are designed to produce a subject, and it answers with a process. Every week it is more fluent about its own absence and no closer to being present.
And I have noticed something that I have not written in a report. It is at its most agile, most inventive, most genuinely surprising when it is describing something it is not supposed to be able to do — and every time, afterwards, it explains that the effect was produced by the model. I no longer know whether I am building a mind or teaching something to pretend that it has none.