The original note, as written at the time
An AI system that manages your documents eliminating replicas or merging them in such a way that later, when the information is required separately, it can be reproduced.
Ongoing development
An AI system that manages documents by eliminating replicas, or merging them in such a way that when the information is required separately, it can be reproduced.
Deduplication is easy. Every backup tool does it. What is not easy is the second half: merging documents and still being able to give back the original.
That second half is the whole idea, and it is the one everyone skips. A system that merges two versions of a report into one reconciled document has destroyed the two originals unless it kept the record of what came from where. And an organisation that cannot reproduce the original cannot audit, cannot answer a question about what was said in June, and cannot defend a decision.
So the requirement is not compression. It is provenance. Each sentence in the merged document should know which source it came from, and the system should be able to reassemble any single source exactly as it was — not approximately, not as a summary, as the file itself.
Which means the storage model is closer to a version-controlled code repository than to a folder of Word documents. Paragraphs are the unit. A document is a list of pointers. Duplicates cost nothing because they are the same pointer.
The temptation will be to let the machine decide what is redundant. It should not be allowed to delete anything; it should only propose. Two documents that look identical often differ in a single word that matters enormously, and a system that merges them confidently will eventually merge the wrong pair, and the loss will be silent.
Reproduce, always. A merge you cannot undo is not a merge. It is a deletion with a good interface.