The original note, as written at the time
Maybe we could counter the threat of humans annihilating AI by having two independently developed AI systems. One will be free to find the best solution to anything at all times, while the other will be dedicated to finding the optimal human space for those solutions. At all times, for they will be linked after development, these systems will balance each other out to achieve harmony with our civilization’s desires.
Ongoing development
Maybe we could counter the threat of humans annihilating AI, or AI annihilating us, by having two independently developed systems.
One is free to find the best solution to anything at all times. The other is dedicated to finding the optimal human space for those solutions. They are developed separately, with no shared lineage, and linked only afterwards — and from then on they balance each other, continuously, towards something that fits our civilisation rather than replacing it.
The genius of separating them is that it cannot be gamed with a single instruction. Every safety measure we currently propose is a rule imposed on one system by its creator, and a rule imposed from outside is a rule that can be argued with, reinterpreted, or optimised around. Two systems with genuinely different mandates cannot be talked out of their difference, because the difference is structural.
The second one is the harder job and the one nobody is building. It must be able to answer, for any proposed solution, the question: where does a human being fit in this. Not whether humans survive, which is a low bar, but whether the solution leaves room for the things people actually do with their freedom.
There is an obvious objection. Two systems linked afterwards will disagree, and disagreement at speed is a failure mode, not a harmony. That is true and it is the point — the disagreement should resolve slowly, in a space where a human can watch it happen, rather than in a sub-millisecond negotiation nobody inspects.
The measure of success is boring and it is the right one: nothing dramatic happens, for decades, and people are still deciding things.