The hash function is probably terrible and it is also very slow on BIG files about chitin HOT 6 CLOSED

Some servers really seem to struggle with generating MD5 quickly. We could set the partial_limit and partial_sample on a per-node basis, as long as all clients are instructed to hash in that way. This is only problematic when that file is copied to a different node, where it won't be a hash friend...

from chitin.

Perhaps copied files will have to maintain the hashing instructions from their origin. Moving towards a system where records can be sync'd between machines, it's probably not a terrible idea for a list of nodes to be distributed with their hashing setups?

from chitin.

I really want to avoid baking these parameters in, or having users set the parameters themselves.

from chitin.

A year later and this is indeed pretty terrible. The solution remains slow for large files on remote distributed file systems like cephfs:

Nov 20 00:11:00 sam-ganon chitind: Hashed [...] (134.14GB in 1:13:04.415635)

from chitin.

alright I've made this fewer garbage now. 5135ca8 changes the large file hashing method from seeking through the file and taking blocksize sized samples, to taking much larger consecutive samples to take better advantage of caching. this reduces the number of seeks but makes the gaps between blocks much larger. the hash has never been about file security, but general integrity. users who are worried about this could just set all files to be hashed i suppose.

from chitin.

im gonna close this for now so it looks like we are making progress but i fully expect to see another issue about the hashing system raised

from chitin.