The original note, as written at the time
If we are going to create anything intelligent it must be able to understand cycles, eternal inquiry and balance, otherwise we shall always have to instruct it to perform anything useful.
Ongoing development
If we are going to create anything intelligent, it must be able to understand cycles, eternal inquiry and balance — otherwise we shall always have to instruct it to perform anything useful.
That is the whole objection in one sentence, and it is not about capability.
A system that does not understand cycles cannot maintain anything. It can build, and then it needs to be told to inspect, and then told to repair, and then told that the repair has failed, and then told again. Every one of those instructions is a human being spending attention on a thing that should have noticed itself. We have built extraordinary competence and married it to an incapacity for upkeep.
A system that does not understand eternal inquiry cannot be left alone with a question. It answers and stops. It does not return to the question with a better instrument, does not notice that the answer it gave in March has stopped being true in October. Inquiry, as a mode, is staying with something you cannot yet close.
And balance — balance is the one that will decide whether any of this is safe to use. A system that optimises one variable will destroy the others, and it will do it competently, and the destruction will be in the report as an acceptable trade. Balance is not a constraint we add. It is a way of holding several goods at once and knowing that they are goods, which is precisely what a target function cannot contain.
None of this is a data problem. You cannot scrape your way to a sense of cycles. These are the properties of a mind that has to live with what it has made, and we have so far built minds that do not live anywhere.