document-classification's Introduction

Document Classification

Classificate documents on categories

Requirements

Please see requirements.txt.
To install these packages, use

$ pip install -r requirements.txt

Training data

Based on Reuters-21578 files.
Available in sgm format on

classification/data/

Trained data's topics can be found in

classification/data/all-topics-strings.lc.txt

To train/test, run the following scripts from classification/

Train

$ python train_and_classify_reuters_data.py

Classify

$ python test_classifier.py

Will predict the topic for the testing articles from

classification/data/reuters_test_json/reuters_test1.json

If all the json testing articles are provided with the specific topics field mentioned,
it will show the Hit-rate score and the Confusion matrix .

Example

$ python train_and_classify_reuters_data 
$ python test_classifier.py

document-classification's People

Contributors

Watchers

document-classification's Issues

Error in running train_and_classify_reuters_data.py

after running this script get this error

python2.7 document-classification-master/classification/train_and_classify_reuters_data.py
Parsing training data...
/usr/local/lib/python2.7/dist-packages/sklearn/svm/base.py:85: DeprecationWarning: gamma=0.0 has been deprecated in favor of gamma='auto' as of 0.17. Backward compatibility for gamma=0.0 will be removed in 0.18
Training...
DeprecationWarning)
Training time 59.95 seconds.
Traceback (most recent call last):
Saving data from training...
File "document-classification-master/classification/train_and_classify_reuters_data.py", line 130, in
main()
File "document-classification-master/classification/train_and_classify_reuters_data.py", line 108, in main
joblib.dump(svm, "saved_data/saved_trained_data.pkl")
File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 402, in dump
cache_size=cache_size, protocol=protocol)
File "/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/numpy_pickle.py", line 209, in init
self.file = open(filename, 'wb')
IOError: [Errno 2] No such file or directory: 'saved_data/saved_trained_data.pkl'