Providers are not removing files from the disk that they lose to downtime. #52 attempts to address this, which seems inefficient but works right now to mitigate the issue.

Currently, providers' RAM usage is way too high. Some providers are reporting their daemons are being OOM'ed which is forcing us to apply to temporary fix of auto-restart services. Although auto-restart is nice to have, it should not be required to keep your provider online.

• #46 Is our first attempt at reducing ram usage by swapping encoding/json for easyjson which has already been shown to improve performance on smaller machines.

Going forward we need to diagnose the underlying issue & watch the provider status as the network scales, there is a good chance there is a memory leak somewhere, but finding it has not been easy.

• Find the source of the leak assuming there is one.
• Patch the leak.
• Deploy the patch to some smaller providers to gauge performance.
• Deploy the patch to every provider.

Currently, prooving a file depends on one thread. Other actions of the provider such as claiming strays have their own threads, but that still leaves us with the possibility to speed up the proof system on multi-core machines to really utilize all their available power.

Our first attempt at this, #25, now outdated and I don't believe that it is worth getting into the weeds of multi-threading our provider's proof system until there is 100% stability on a single thread.

v2

Bidirectionally reliant on JackalLabs/canine-chain#365

For the Jackal Protocol v4 upgrade, the logic surrounding uploading files & posting proofs has changed enormously. Thus making existing storage providers incompatible with the chain. Just before the upgrade takes place, all providers will need to run a migration script to ensure that once the upgrade passes, all file deals will still be intact. During the chain upgrade, all providers will need to upgrade to v2 to continue posting proofs and accepting deals to/from the chain.

For new providers interested in starting fresh after the upgrade, another lighter-weight binary is being tested that will be compatible with the v4 upgrade immediately: https://github.com/JackalLabs/sequoia

Provider hash a ton of stuff. Every proof is made up of tons of hashes, we save these hashed values to disk to save compute time but that eats away at storage space, something very valuable to the Jackal Protocol.

• Hook in CUDA library.
• Replace the Merkle-tree hashing operations with CUDA-accelerated hashing.
• Benchmark performance to see if it is as fast or faster than CPU hashing under normal load (obviously synthetic tests will show improvement).
• Remove saving the hash trees & rebuild on the fly using GPU.
• Benchmark this disk-space saving technique.

We would need two binaries, or maybe a CLI flag if possible to handle systems without a very powerful GPU to accelerate the system.