Comments (4)
I guess 1b82edf96afa4 is relevant, but I'm still sort of confused. Something with "128" in the name returning 4 bytes is... unexpected.
from go-multihash.
Yeah, this seems wrong. I'm not sure what's going on here.
from go-multihash.
I believe we should be using Sum64 (which is also not 128 but it's what go-unixfs uses). Honestly, we should never use this hash function for anything.
from go-multihash.
I did a bit of poking around and here's what I've got.
murmur3
- There are 3 variants - 32 bit, 128 bit optimized for 32bit machines, 128 bit optimized for 64bit machines
- We have 2 codes - murmur3-32 and murmur3-128. @Stebalien's comment multiformats/multihash#85 (comment) indicates that the UnixFS version is the 64bit one (and that seems to match my poking at the test vectors below, but perhaps with some endianness issues).
- Finding reliable test vectors is a bit hard because test vectors aren't really visible, but there are some here aappleby/smhasher#6.
- I added some of the test vectors to IPFS for future reference, as I don't want the test vectors to only be stored on pastebin
- x86-32: https://ipfs.io/ipfs/bafkreicowio3fgyeqz6ns73vzppbkbnfhfedpkemch5iuz2fc6sjnzmix4
- x64-128: https://ipfs.io/ipfs/bafkreihqwk2t3vhfmaga55d2tkhzmwol7vt5xh22xclw6icdmwhegalina
- Note for posterity: the hashes inside the CIDs are the SHA-256 of the raw file bytes
- For some reason when murmer3 was initially implemented in this library in #42 for some reason it was implemented as murmur3-32 (not what was already in use by UnixFS). Also, it was encoded as LittleEndian which happens to match the test vectors I found but does not match what the underlying Go library we're using does (which uses BigEndian).
- When we redefined/clarified that the murmur3 code was actually the murmur3-(x64)-128 code we didn't change the murmer3 implementation used here
UnixFS Sharding
- @Stebalien switched from "64bit" (i.e. first half of 128 bit) to full 128bit in ipfs/go-unixfs#53
- Less than a month later it seems to have been reverted accidentally by ipfs/go-unixfs#42 🤦♂️
Plan I think we should go with:
- Figure out what we're actually using in UnixFS
- If we can fix or throw some definitions into UnixFS that make the hash function actually match mumur3-x64-128
- If not then we have a tough decision to make about if we need to redefine the code
- In the meanwhile in order to merge #136, I think we should just remove the murmur3 hash entirely
- I really really hope no one is using this murmur3 implementation, but we can try and do some spelunking to warn them. This implementation isn't really valid since UnixFS was AFAICT the first user and so the definition is supposed to match that one. It definitely hasn't been valid since we clarified murmer3 -> murmer3-(x64)-128.
Note: I'll put up some code on the endianness + test vectors when I get a chance. Need another pair of eyes to make sure I'm doing it correctly, and also need some context on UnixFS usage to make sure I'm understanding how the murmur3 hashes are being used.
from go-multihash.
Related Issues (20)
- Possibly erroneous tag pushed: v1.0.2
- Possibly erroneous tag pushed: v1.0.2
- Possibly erroneous tag pushed: v1.0.2
- Version tags gone missing? HOT 3
- Doesn't work on Go playground HOT 2
- Convert hash to hex hash and vice versa HOT 2
- Please choose between identity / id and stick to it across all projects HOT 1
- Harden varint decoding
- Some algorithms don't implement Sum HOT 2
- [enhancement] adding BLAKE3 support HOT 21
- Split core types into a core subpackage HOT 1
- Go get multihash fails compilation due to sha3.NewLegacyKeccak512
- Generated go constants for multicodecs HOT 16
- approach to constants HOT 6
- Decode allocates
- Create a new tag to allow using recent master HOT 3
- hasher.Write error is not handled
- Interview partners for research about communication in GitHub projects wanted
- Faster blake3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from go-multihash.