imagicthecat / bitsea-dds Goto Github PK

For more powerful collaborative data structures.

Examples:

bitsea:1CMn1nKMPqjFA2PxcA2vihGyhTZ9vPjDhGu7hHZVj464H/...
bitsea:topic:1CMn1nKMPqjFA2PxcA2vihGyhTZ9vPjDhGu7hHZVj464H
bitsea:fJx4T3yQnDbUsNRkTzoH9YJz9hAqSsA9JyFu3PA5f7o4P/...
bitsea:fJx4T3yQnDbUsNRkTzoH9YJz9hAqSsA9JyFu3PA5f7o4P.1CMn1nKMPqjFA2PxcA2vihGyhTZ9vPjDhGu7hHZVj464H/...
bitsea:org/wikipedia/...

Mechanims could be used to filter content, especially discovered by topics, based on identities.

Examples:

• Local rules, e.g. a blacklist system.
• An authority, e.g. a common directory built on external verification mechanisms (like emails) to limit spam.
• Web-of-trust: each node trust other nodes, forming a graph of trust.

The system should be powerful enough to reasonably implement all of them and more.

To maximize data deduplication, the way a file data structure is generated from a file should follow a stable consensus.

Content-defined chunking may improve content-level deduplication, but is probably not worth it:

• In a world with compression algorithms, a lot of data (if not most) will not benefit from it.
• Multiple algorithms and parameters may yield better results on specific datasets, finding a general method is probably not reasonable.
• It increases complexity and costs more than fixed chunking.

Instead, file-level deduplication is a more reasonable goal:

• A simple fixed chunking consensus should be adopted to at least deduplicate the whole file at the leaf level.
• If a deterministic way of constructing the data structure is used, all data blocks will be deduplicated.

For other kind of files which could greatly benefit from more advanced deduplication, the file data structure should be specialized. E.g. files combining other files could reference them directly instead of doing content-defined chunking in the hope they will match.

Edit: combined files may prevent optimized structures for a seldom use case.

Allow directory entries to pointers, or create a different data structure for dynamic content to keep the filesystem semantic for the directory ? (use case: "domain" name directory)

Create a general purpose key-value store data structure with B-trees and specialize it to implement these two kind of directories ?

Constraints to simplify the system. The URI resource path after the root segment is about traversing an Hash DAG, thus only the root segment may represent a mutable resource, called the root or origin.

Instead of the alias idea, which is somewhat opposed to the principles behind an URI, a global name system should be used. A namespaces path like a domain name is now well understood, thus BitSea should reuse that; but it should not be restricted to the regular DNS architecture, it may be extended with new top-level domains implementing other kinds of consensus, e.g. a distributed system like a blockchain. Examples are the Ethereum's .eth and Namecoin's .bit.

In any case, a domain name/path must establish a consensus about what resource that origin represents (formatted as a BitSea URI without non-root segments representing a block or pointer ?).

For regular DNS, a _bitsea subdomain with a TXT record could hold that URI.