Anserini utilities for working with word embeddings. Currently, these tools are held in a separate repository because these are experimental features that depend on deeplearning4j, and direct inclusion of all dependent artifacts in Anserini would blow up the size of the Anserini fatjar.
Here's a sample invocation of taking GloVe embeddings and creating a Lucene index for lookup. This is treating Lucene as a simple key-value store.
$ target/appassembler/bin/IndexWordEmbeddings -index glove -input glove.840B.300d.txt
Simple lookup example:
$ target/appassembler/bin/LookupWordEmbeddings -index glove -word "happy"
Index dimensionality reduced word embeddings as Lucene FloatPoints
.
Number of dimensions allowed is between 1 and 8.
$ target/appassembler/bin/IndexReducedWordEmbeddings -index glove -input glove.840B.300d.txt -dimensions 8
Simple nearest neighbour example:
$ target/appassembler/bin/NearestNeighbour -index glove -word "thomas"
thomas
vincent
lewis
fred
williams