Package for Statistically Significant Language Change.

• Free software: BSD license
• Documentation: https://langchangetrack.readthedocs.org.

• This package provides tools to detect linguistic change in temporal corpora.

• The meta algorithm works in 2 main steps

  1. Time series construction:Given a word, we construct a time series that tracks the displacement of a word through time. We track the displacement of a word using either Frequency, Part of Speech Distribution or Co-occurrences.
  2. Change point detection: We then use change point detection methods to detect if the time series contains a change point and if so what the change point is.

The details of the above steps are outlined in : http://arxiv.org/abs/1411.3315

Please see this for a cool visualization of words moving through time: http://tinyurl.com/wordvis

We assume a temporal corpus of text files (appropriately tokenized) to be present in a directory. In addition we assume list of words in a single text file that one is interested in tracking. This could just be the set of words in the common vocabulary of the temporal corpus.

The output consists of the pvalues for each word indicating the significance of the changepoint detected.

$ngrams_pipeline.py --corpus-dir data/temporal_corpus/ --file-extension "ngrams" --working-dir ./working --output-dir ./output --context-size 5 --epochs 3 --start-time-point 1900 --end-time-point 2000 --step-size 5 --vocabulary-file data/temporal_corpus/common_vocab.txt --workers 16

$pos_pipeline.py --corpus-dir data/temporal_corpus/ --file-extension "ngrams" --working-dir ./working --output-dir ./output --start-time-point 1900 --end-time-point 1930 --step-size 5 --vocabulary-file data/temporal_corpus/common_vocab.txt --workers 16

$freq_pipeline.py --corpus-dir data/temporal_corpus/ --file-extension "ngrams" --working-dir ./working --output-dir ./output --start-time-point 1900 --end-time-point 2000 --step-size 5 --vocabulary-file data/temporal_corpus/common_vocab.txt --workers 16

You might need to tune the hyper parameters as per your specific need.

Usage: ngrams_pipeline.py

optional arguments:

-h, --help	show this help message and exit
--corpus-dir CORPUS_DIR
	Corpus directory
--file-extension EXT
	Corpus file extension
--working-dir WORKING_DIR
	Working directory
--output-dir OUTPUT_DIR
	Output directory
--context-size WINDOW
	Context size to use for training embeddings
--epochs EPOCHS
	Number of epochs to training embeddings
--start-time-point START
	Start time point
--end-time-point END
	End time point
--step-size STEP
	Step size for timepoints
--model-family MODEL_FAMILY
	Model family default (locallinear)
--number-nearest-neighbors KNN
	Number of nearest neighbors to use for mapping to joint space (default:1000) --vocabulary-file VOCAB_FILE Common vocabulary file
--threshold THRESHOLD
	Threshold for mean shift model for change point detection (default: 1.75)
--bootstrap-samples BOOTSTRAP
	Number of bootstrap samples to draw (default: 1000)
--workers WORKERS
	Maximum number of workers (default: 1)
-l LOG, --log LOG
	log verbosity level

Usage: pos_pipeline.py

optional arguments:

-h, --help	show this help message and exit
--corpus-dir CORPUS_DIR
	Corpus directory
--file-extension EXT
	Corpus file extension
--working-dir WORKING_DIR
	Working directory
--output-dir OUTPUT_DIR
	Output directory
--start-time-point START
	Start time point
--end-time-point END
	End time point
--step-size STEP
	Step size for timepoints
--vocabulary-file VOCAB_FILE
	Common vocabulary file
--threshold THRESHOLD
	Threshold for mean shift model for change point detection
--bootstrap-samples BOOTSTRAP
	Number of bootstrap samples to draw
--workers WORKERS
	Maximum number of workers
-l LOG, --log LOG
	log verbosity level

usage: freq_pipeline.py

optional arguments:

-h, --help	show this help message and exit
--corpus-dir CORPUS_DIR
	Corpus directory
--file-extension EXT
	Corpus file extension
--working-dir WORKING_DIR
	Working directory
--output-dir OUTPUT_DIR
	Output directory
--start-time-point START
	Start time point
--end-time-point END
	End time point
--step-size STEP
	Step size for timepoints
--vocabulary-file VOCAB_FILE
	Common vocabulary file
--threshold THRESHOLD
	Threshold for mean shift model for change point detection
--bootstrap-samples BOOTSTRAP
	Number of bootstrap samples to draw
--workers WORKERS
	Maximum number of workers
-l LOG, --log LOG
	log verbosity level

• wheel==0.23.0
• argparse>=1.2.1
• numpy>=0.9.1
• scipy>=0.15.1
• more_itertools>=2.2
• joblib>=0.8.3-r1
• gensim==0.10.3
• statsmodels>=0.5.0
• changepoint>=0.1.0
• nltk>=3.0.0
• textblob>=0.9.0
• textblob-aptagger>=0.2.0
• psutil>=2.2.0
• GNU Parallel
• R (good to have)
• rpy2 (good to have)

1. Install GNU Parallel from here: www.gnu.org/software/software.html
2. cd langchangetrack
3. pip install -r requirements.txt
4. python setup.py install