gmbecker / fastdigest Goto Github PK

Background: richfitz/storr#82, eddelbuettel/digest#59 (comment), eddelbuettel/digest#59 (comment).

In storr, serialization and hashing happen separately, which makes sense because storr actually needs that serialized object to put in storage. So when I plug fastdigest straight into storr, the same object gets serialized twice. Benchmarks: ropensci/drake#498 (comment).

For the sake of richfitz/storr#82 and ropensci/drake#498, would you be open to separating these steps in fastdigest? I think this may also potentially make #1 easier if you choose to pursue it.

python's spooky, which uses the Hash128 function of the original SpookyHash source, shows a different value to fastdigest when I hexlify the output for a given string input. Should these match given you're using the streaming version of SpookyHash? Is there another reason why they don't match? I am trying to incorporate this into digest and I'm guessing the author will insist that the default output matches something canonical, if such a thing exists, for a few strings.

I have changed the fastdigest seeds to 0 (which is what the python package uses by default) and rebuilt.

You convert the two final 64-bit integer output to a string using:

    SEXP ans;
    PROTECT(ans = allocVector(RAWSXP, 16));
    unsigned char *tmp;
    tmp  = (unsigned char *) &h1;
    for(int i = 0; i < 8; i++)
	RAW(ans)[i] = tmp[i];
    tmp = (unsigned char *) &h2;
    for(int j = 0; j < 8; j++)
	RAW(ans)[j + 8] = tmp[j];
    UNPROTECT(1);
    return ans;

and the python package converts the output of SpookyHash's Hash128 function to a 128-bit integer using:

PyObject *retval = _PyLong_FromByteArray((unsigned char *)seed, 16, 1, 0);
    return retval;

and I convert that value to a string using:

hexlify(hash128("abc").to_bytes(16, 'little'))

The digest API is already plugged into storr, and by extension, drake. Integration into digest could automatically speed up these tools. Related: eddelbuettel/digest#59 (comment), eddelbuettel/digest#59 (comment).

The hashes in fastdigest need to skip the first 14 bytes of the serialized output to avoid hashing information about the R version used for serialization.

R 3.6.0 also changes the default serialization to version 3, so fastdigest also needs to set the version to 2 to stay consistent across R versions.

[kendon.bell@mahuika02 ~]$ module load R/3.5.1-gimkl-2017a
[kendon.bell@mahuika02 ~]$ R

R version 3.5.1 (2018-07-02) -- "Feather Spray"
...
Type 'q()' to quit R.

> fastdigest::fastdigest("abc")
[1] "226bf6584240d7991746ea2e3b3f3e48"
> q()
[kendon.bell@mahuika02 ~]$ module load R/3.5.3-gimkl-2018b

[kendon.bell@mahuika02 ~]$ R

R version 3.5.3 (2019-03-11) -- "Great Truth"
...
Type 'q()' to quit R.

> fastdigest::fastdigest("abc")
[1] "99fbc3aaa10bbc6c19b3745a18e1bc00"

FYI, I have now completed incorporating SpookyHash, mostly using your code, into digest: eddelbuettel/digest#100. It should be released soon. Many thanks for this contribution! I have implemented header skipping and version fixing.