hironsan / asari Goto Github PK

View Code? Open in Web Editor NEW

141.0 141.0 20.0 51.49 MB

Japanese sentiment analyzer implemented in Python.

License: MIT License

Python 97.00% Makefile 3.00%

machine-learning natural-language-processing sentiment-analysis

asari's Introduction

Hi there, I'm Hiroki 👋

I'm interested in creating software related to machine learning and natural language processing.

Also, I like writing and translating books in Japanese. Here are some examples:

Author

機械学習・深層学習による自然言語処理入門

Translator

If you want to support me, see GitHub Sponsors❤️

asari's People

Contributors

Stargazers

Watchers

asari's Issues

TypeError: object of type 'generator' has no len()

$ python sentiment_evaluate.py
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\utils\deprecation.py:
143: FutureWarning: The sklearn.svm.classes module is  deprecated in version 0.2
2 and will be removed in version 0.24. The corresponding classes / functions sho
uld instead be imported from sklearn.svm. Anything that cannot be imported from
sklearn.svm is now part of the private API.
  warnings.warn(message, FutureWarning)
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator LinearSVC from version 0.20.2 when using versi
on 0.23.2. This might lead to breaking code or invalid results. Use at your own
risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\utils\deprecation.py:
143: FutureWarning: The sklearn.preprocessing.label module is  deprecated in ver
sion 0.22 and will be removed in version 0.24. The corresponding classes / funct
ions should instead be imported from sklearn.preprocessing. Anything that cannot
 be imported from sklearn.preprocessing is now part of the private API.
  warnings.warn(message, FutureWarning)
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator LabelEncoder from version 0.20.2 when using ve
rsion 0.23.2. This might lead to breaking code or invalid results. Use at your o
wn risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator _SigmoidCalibration from version 0.20.2 when u
sing version 0.23.2. This might lead to breaking code or invalid results. Use at
 your own risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator CalibratedClassifierCV from version 0.20.2 whe
n using version 0.23.2. This might lead to breaking code or invalid results. Use
 at your own risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator TfidfTransformer from version 0.20.2 when usin
g version 0.23.2. This might lead to breaking code or invalid results. Use at yo
ur own risk.
  warnings.warn(
C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\base.py:329: UserWarn
ing: Trying to unpickle estimator TfidfVectorizer from version 0.20.2 when using
 version 0.23.2. This might lead to breaking code or invalid results. Use at you
r own risk.
  warnings.warn(
2020-08-27 14:38:47 [INFO] Sentiment: Started.
Traceback (most recent call last):
  File "sentiment_evaluate.py", line 40, in <module>
    Sentiment().evaluate()
  File "sentiment_evaluate.py", line 36, in evaluate
    print(self._asari_sa(example))
  File "sentiment_evaluate.py", line 31, in _asari_sa
    res = self.asari.ping(text=sent)['top_class']
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\asari\api.py", line 2
7, in ping
    vector = self.preprocessor.transform([text])
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 1880, in transform
    X = super().transform(raw_documents)
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 1250, in transform
    _, X = self._count_vocab(raw_documents, fixed_vocab=True)
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 1110, in _count_vocab
    for feature in analyze(doc):
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 111, in _analyze
    doc = ngrams(doc)
  File "C:\Users\Merli\anaconda3\envs\tm\lib\site-packages\sklearn\feature_extra
ction\text.py", line 242, in _word_ngrams
    n_original_tokens = len(original_tokens)
TypeError: object of type 'generator' has no len()

Let me know the environment in which asari works properly.

Thanks for the great project!

I am currently trying to use asari, but when I resolve one dependency, it breaks another package dependency or causes other errors.
I have tried running various python versions, joblib, Janome, scikit-learn and asari, but so far it is still not resolved.

If you could please tell me the python version and the requirements.txt output by pip freeze > requirements.txt that the code in the README works with.

Thank you in advance.

joblib import error

importing joblib from sklearn.externals is deprecated

Error: While importing "app", an ImportError was raised:

Traceback (most recent call last):
  File "/Users/.virnev/lib/python3.7/site-packages/flask/cli.py", line 240, in locate_app
    __import__(module_name)
  File "/Users/app/app.py", line 4, in 
    from asari.api import Sonar
  File "/Users/.virnev/lib/python3.7/site-packages/asari/api.py", line 12, in 
    from sklearn.externals import joblib
ImportError: cannot import name 'joblib' from 'sklearn.externals'

Python37\lib\site-packages\sklearn\externals\joblib_init_.py:15: DeprecationWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.

should be fixed by directly importing joblib

TypeError: object of type 'generator' has no len()

When I tried to run it, I got the following error.

>>> sonar.ping(text="広告多すぎる")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/asari/api.py", line 27, in ping
    vector = self.preprocessor.transform([text])
  File "/usr/local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 1880, in transform
    X = super().transform(raw_documents)
  File "/usr/local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 1250, in transform
    _, X = self._count_vocab(raw_documents, fixed_vocab=True)
  File "/usr/local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 1110, in _count_vocab
    for feature in analyze(doc):
  File "/usr/local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 111, in _analyze
    doc = ngrams(doc)
  File "/usr/local/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 242, in _word_ngrams
    n_original_tokens = len(original_tokens)
TypeError: object of type 'generator' has no len()

Where does the training data come from?

What does the model here use as training data? Didn't see an explanation anywhere...

can't use sklearn 0.24.2

It seems depends on old scikit-learn, so I can't use it with 0.24.2.

~/anaconda3/lib/python3.7/pickle.py in load_global(self)
   1374         module = self.readline()[:-1].decode("utf-8")
   1375         name = self.readline()[:-1].decode("utf-8")
-> 1376         klass = self.find_class(module, name)
   1377         self.append(klass)
   1378     dispatch[GLOBAL[0]] = load_global

~/anaconda3/lib/python3.7/pickle.py in find_class(self, module, name)
   1424             elif module in _compat_pickle.IMPORT_MAPPING:
   1425                 module = _compat_pickle.IMPORT_MAPPING[module]
-> 1426         __import__(module, level=0)
   1427         if self.proto >= 4:
   1428             return _getattribute(sys.modules[module], name)[0]

ModuleNotFoundError: No module named 'sklearn.svm.classes'

Do you have plan to update the model?
I think it is relevant with #6 .

can't install with python3.10 on raspberry pi zero

I am using python 3.10.9 and pip 22.3.1 on raspberry pi zero.
I tried to install asari but got the following error and could not install it.

pi@raspberrypi:~/asari $ python3 -m pip install . --log ./pip_log.txt
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /home/pi/asari
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting scikit-learn<2.0.0,>=1.1.1
  Using cached scikit-learn-1.2.0.tar.gz (7.2 MB)
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.       
  Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I have tried the following.

Installing with python3.7
Downgrading pip and installing

Is there any possibility that asari is not compatible with python3.10?

joblib エラー

pip install git+https://github.com/Hironsan/asari
最新コードですと、下記部分でエラーになるようです。

from asari.api import Sonar
sonar = Sonar()
/usr/local/lib/python3.8/site-packages/sklearn/utils/deprecation.py:143: FutureWarning: The sklearn.svm.classes module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.svm. Anything that cannot be imported from sklearn.svm is now part of the private API.
warnings.warn(message, FutureWarning)
/usr/local/lib/python3.8/site-packages/sklearn/base.py:329: UserWarning: Trying to unpickle estimator LinearSVC from version 0.20.2 when using version 0.23.1. This might lead to breaking code or invalid results. Use at your own risk.
warnings.warn(
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.8/site-packages/asari/api.py", line 21, in init
self.estimator = joblib.load(model_file)
File "/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 585, in load
obj = _unpickle(fobj, filename, mmap_mode)
File "/usr/local/lib/python3.8/site-packages/joblib/numpy_pickle.py", line 504, in _unpickle
obj = unpickler.load()
File "/usr/local/lib/python3.8/pickle.py", line 1210, in load
dispatchkey[0]
File "/usr/local/lib/python3.8/pickle.py", line 1526, in load_global
klass = self.find_class(module, name)
File "/usr/local/lib/python3.8/pickle.py", line 1577, in find_class
import(module, level=0)
ModuleNotFoundError: No module named 'sklearn.externals.joblib'

hironsan / asari Goto Github PK

asari's Introduction

Hi there, I'm Hiroki 👋

asari's People

Contributors

Stargazers

Watchers

Forkers

asari's Issues

TypeError: object of type 'generator' has no len()

Let me know the environment in which asari works properly.

joblib import error

TypeError: object of type 'generator' has no len()

Where does the training data come from?

can't use sklearn 0.24.2

can't install with python3.10 on raspberry pi zero

joblib エラー

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent