This repository holds a solution for Assignment 2 in the course Netural Language Processing held during semester 1, 2016, by Professor [Michael Elhadad] (https://www.cs.bgu.ac.il/~elhadad/)

This assignment covers the topic of statistical distributions, regression and classification. The objective is:

1. Experiment and evaluate classifiers for the tasks of named entity recognition and document classification.
2. Explore the task of Document Classification, comparing email spam detection, SMS spam detection and news categorization.
3. Explore the problem of domain adaptation by comparing the performance of classifiers trained in one domain when tested in another.
4. Explore the task of Named Entity Recognition (NER), which features work for this task, and which classifier algorithms help: ..* -logistic regression, Naive Bayes and HMM. ..* -Use pre-trained word embeddings and measure whether they help for the task of NER.

The solution for the Assignment is in the form of an iPython (Jupyter) notebook file (with extension ipynb). The main noteboos solution file is p1.ipynb

Most important code is also located in a .py file under/code/assignment2.py

The Spam Data set should be extracted to the folder data, as it is left empty.

An organized submitted solution is available under release

The course is an introduction to Natural Language Processing. The main objective of the course is to learn how to develop practical computer systems capable of performing intelligent tasks on natural language: analyze, understand and generate written text. This task requires learning material from several fields: linguistics, machine learning and statistical analysis, and core natural language techniques. [course web site] (https://www.cs.bgu.ac.il/~elhadad/nlp16.html)