This is an implementation of the LexVec word embedding model (similar to word2vec and GloVe) that achieves state of the art results in multiple NLP tasks, as described in this paper and this one.

• Common Crawl - 58B tokens - 2,000,000 words - 300 dimensions

  • Word Vectors (2.2GB)
  • Word + Context Vectors (2.3GB)

• English Wikipedia 2015 + NewsCrawl - 7B tokens - 368,999 words - 300 dimensions

  • Word Vectors (398MB)
  • Word + Context Vectors (426MB)

• All three models were trained using the same English Wikipedia 2015 + NewsCrawl corpus.

• GSem, GSyn, and MSR analogies were solved using 3CosMul.

• LexVec was trained using the default parameters, expanded here for comparison:

  $ ./lexvec -corpus enwiki+newscrawl.txt -output lexvecvectors -dim 300 -window 2 \
  -subsample 1e-5 -negative 5 -iterations 5 -minfreq 100 -matrix ppmi -model 0
  

• word2vec Skip-gram was trained using:

  $ ./word2vec -train enwiki+newscrawl.txt -output sgnsvectors -size 300 -window 10 \
  -sample 1e-5 -negative 5 -hs 0 -binary 0 -cbow 0 -iter 5 -min-count 100
  

• All models use vectors with 300 dimensions.

• GSem, GSyn, and MSR analogies were solved using 3CosMul.

• LexVec was trained using this release of Common Crawl which contains 58B tokens, restricting the vocabulary to the 2 million most frequent words, using the following command:

  $ OUTPUTDIR=output ./external_memory_lexvec.sh -corpus common_crawl.txt -negative 3 \
  -model 0 -maxvocab 2000000 -minfreq 0 -window 2                                             
  

• The pre-trained word2vec vectors were trained using the unreleased Google News corpus containing 100B tokens, restricting the vocabulary to the 3 million most frequent words.

• The pre-trained GloVe vectors were trained using Common Crawl (release unknown) containing 42B tokens, restricting the vocabulary to the 1.9 million most frequent words.

The easiest way to get started with LexVec is to download the binary release. We only distribute amd64 binaries for Linux.

Download binary

If you are using Windows, OS X, 32-bit Linux, or any other OS, follow the instructions below to build from source.

1. Install the Go compiler

2. Make sure your $GOPATH is set

3. Execute the following commands in your terminal:

   $ go get github.com/alexandres/lexvec
   $ cd $GOPATH/src/github.com/alexandres/lexvec
   $ go build

To get started, run $ ./demo.sh which trains a model using the small text8 corpus (100MB from Wikipedia).

Basic usage of LexVec is:

$ ./lexvec -corpus somecorpus -output someoutputdirectory/vectors

Run $ ./lexvec -h for a full list of options.

Additionally, we provide a word2vec script which implements the exact same interface as the word2vec package should you want to test LexVec using existing scripts.

By default, LexVec stores the sparse matrix being factorized in-memory. This can be a problem if your training corpus is large and your system memory limited. We suggest you first try using the in-memory implementation, which achieves higher scores in evaluations. If you run into Out-Of-Memory issues, try this External Memory approximation (not as accurate as in-memory; read paper for details).

$ env OUTPUTDIR=output ./external_memory_lexvec.sh -corpus somecorpus -dim 300 ...exactsameoptionsasinmemory

Pre-processing can be accelerated by installing nsort and pypy and editing pairs_to_counts.sh.

Salle, Alexandre, Marco Idiart, and Aline Villavicencio. Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations. The 54th Annual Meeting of the Association for Computational Linguistics. 2016.

Salle, A., Idiart, M., & Villavicencio, A. (2016). Enhancing the LexVec Distributed Word Representation Model Using Positional Contexts and External Memory. arXiv preprint arXiv:1606.01283.

Copyright (c) 2016 Salle, Alexandre [email protected]. All work in this package is distributed under the MIT License.