Out of memory under OS X when adding items to index about annoy HOT 8 CLOSED

That's very strange; could you share the code you're using to generate and to use the indexes? When does it run out of memory -- when creating the index or after saving it and using it for ANN lookups?

How much memory does it use in your small test? It shouldn't be very much if it's only 500 items, even if they're 128 dimensional items.

Annoy definitely works under Linux (Debian or Ubuntu is what we use) and I thought it worked fine on OS X as well.

from annoy.

While stripping down the code to get a minimal working example, I found the actual cause of the problem.

I was generating quite long integers that I have used as identifiers (see https://gist.github.com/herrbuerger/5f22ee159d0df800c844). Once I had changed them back to smaller numbers, everything went smoothly.

Here is a minimal working script to reproduce the issue: https://gist.github.com/herrbuerger/e4452d55ffb03a53d423

from annoy.

I have found another issue regarding the identifiers, not sure this is related. If not, a separate issue would probably make sense:

https://gist.github.com/herrbuerger/6cbeada92901a282c957

If you check the output of the script, you will see that annoy shows identifiers which have never been used. Took me a while to catch this :)

from annoy.

We should probably couple Annoy with a hashtable so that you can put in arbitrary stuff. But yeah, until then, you will see those issues :(

from annoy.

I'm not sure if there is a fix to this – what do you suggest?

from annoy.

A hashtable would be great. This would definitely solve my problems.

Still, I have a working example with the integers now as well and the results look pretty good. However, I will have to create a lookup table, which will probably be a challenge itself depending on the number of items.

from annoy.

There's a lot of file-based fast hashtables, including Spotify's own: https://github.com/spotify/sparkey – other ones I'm aware of are bsddb and Tokyo Cabinet.

from annoy.

Closing this issue since it's not a "bug" per se – maybe the documentation should be more clear

from annoy.