Date: 05-05-2019 Authors: Sudhir Singh

Demystify languages and varieties similarity: the identification of similar languages and varieties.

Following modules are required in python 3 environment:

1) sklearn
2) imblearn
3) matplotlib

The requirements.txt file contains all dependant libraries to be installed. To install all dependant libraries, please run the following command:

pip install -r requirements.txt

1) Using Jupyter notebook
2) Using Python code

If using Jupyter notebook, make sure you have a self created environment or Anaconda created environment and Jupyter notebook installed.

a) start Jupyter notebook:

sudhirsingh$ jupyter notebook

b) In the Jupyter notebook, browse to the folder where the code is located:
c) Click on the "Term Project - language identification.ipynb" file. It will open in new tab. d) In the new tab, click on the "Cell" and the select "Run All".
e) Enter train, dev test, and test data set full file path as asked by program.
f) Now program will execute and output results.

a) By providing command line arguments:

python3 TP-code.py train.txt devel.txt test-gold.txt

b) By executing the code and then providing the input file names:

python3 TP-code.py

1) Model accuracy: with two classifier (Multinomial Naive Bayes & Linear SVM classifier)
2) Confusion matrix - without normalization
3) Classification report