chrisjmccormick / minhash Goto Github PK

How can we integrate this code with GPU?

Chis,

Thank you for the tutorial and code. I found yours to be one of the more understandable explanations. MinHashing seems extremely clever.

In your runHashMinExample, you commented that you used the direct calculation for the similarities, which I have no problems understanding. However, I have been searching for this "MinHash approach approach" to creating similarities. I was wondering if you had written this for if you can point me in the right direction. I have a huge dataset and I would like to use MinHash approach.

I am assuming that this would be faster. I am also assuming that the storage code (for the triangle matrix) would remain the same.

Thank you,
Ben

As a and b have a maximum value of your chosen mersenne prime and hv can be any 32-bit integer, (a * hv + b) can yield integers larger than 2^64. Isn't that a problem? Is Python capable of dealing with this?

https://github.com/ekzhu/datasketch/blob/260b80063b7d3599f93d7179a7749d626f010973/datasketch/minhash.py#L134

Hi Chris. Nice article on MinHash! The code is throwing the type error below (in Python 3.7.3). Not sure what to do about that as I am new to Python but if you have recommendations or can point me in the right direction it would be appreciated.

$ python runMinHashExample.py
runMinHashExample.py:63: DeprecationWarning: 'U' mode is deprecated
  f = open(truthFile, "rU")
Shingling articles...
runMinHashExample.py:94: DeprecationWarning: 'U' mode is deprecated
  f = open(dataFile, "rU")
Traceback (most recent call last):
  File "runMinHashExample.py", line 127, in <module>
    crc = binascii.crc32(shingle) & 0xffffffff
TypeError: a bytes-like object is required, not 'str'

Hello, how are you?

I read your code and it made me finally understand the fundamentals of the MinHash algorithm. The reason I'm creating this issue is because I wanted your permission to port part of your code to C# for educational purposes. In case you allow this I also wanted to ask you to provide a license since this repository don't have one. I plan to publish the ported code in my account as a public repository.

Thanks so much for putting this post and code together @chrisjmccormick, this is great stuff.

I just have a quick query on the training data--where does this come from? I ask because I'm trying to tune an algorithm on this data but it looks like the *.truth files are missing some true positive matches (e.g. [u't8061', u't8071'] should be a match, but that pair isn't identified in articles_10000.truth). I've got many pairs with very high similarity scores that aren't identified in the truth files, which makes it tricky to measure the precision of the algorithm I'm tuning.

If you recall where the data came from, I'd be game to hunt down the origin and see if there are more true matches there. In any event, thanks again for this work!