When the downloader is trigger because the data is missing, the logger is not properly flushed.

Hello,

i use melt_tagger in a class. The sentence to analyze is sent with socketio

import spacy
import sys
import random
from spacy_lefff import LefffLemmatizer, POSTagger
import socketio

class SomeClass():
    def __init__(self):
        self.nlp = spacy.load('fr')
        self.pos = POSTagger()  # comments in console
        self.french_lemmatizer = LefffLemmatizer(
            after_melt=True, default=False)
        self.nlp.add_pipe(self.pos, name='pos', after='parser')
        self.nlp.add_pipe(self.french_lemmatizer, name='lefff', after='pos')

    def analyze(self, param1):
        self.doc = self.nlp(param1)
        for d in self.doc:
            print(d.text, d.pos_, d._.melt_tagger)

# --- Socket

sio = socketio.Client()

@sio.on('connect', namespace='/test')
def on_connect():
    print('--> connection established')


@sio.on('disconnect', namespace='/test')
def on_disconnect():
    print('--> disconnected from server')


@sio.on('myevent', namespace='/test')
def on_message(data):
    print(' Received message : ',  data['spoken'])
    obj1.analyze(data['spoken'])

sio.connect('http://localhost:7000', namespaces=['/test'])

obj1 = SomeClass()

sio.wait()

On the first try with data[''spoken"] equals to 'les carottes et les radis sont des légumes',
the console prints :

 Received message : {'spoken': ' les carottes et les radis sont des légumes'}
2019-09-04 12:43:21,766 - spacy_lefff.melt_tagger - INFO -   TAGGER: POS Tagging...
les DET DET
carottes NOUN NC
et CCONJ CC
les DET DET
radis NOUN NC
sont AUX V
des DET DET
légumes NOUN NC

The next tries print in console :

Received message : {'spoken': ' les carottes et les radis sont des légumes'}
2019-09-04 12:43:27,007 - spacy_lefff.melt_tagger - INFO -   TAGGER: POS Tagging...
les DET None
carottes NOUN None
et CCONJ None
les DET None
radis NOUN None
sont AUX None
des DET None
légumes NOUN None

d.text, d.pos_ work on every request, melt_taggerdoesn't.

When the token property token._.malt parser is missing lefff throws an exception at line 64 of lefff.py

This can be fixed by changing the line to
t = token._.melt_tagger.lower() if self.after_melt and hasattr(token._, 'melt_tagger') else token.pos_

Thanks for this. Managed to get to work on Python 3 with spacy > 2.0.9 (2.0.11) -- at least I think it works.

1. 2to3 all files
2. in melt_tagger.py, change win /= 2 to win = win // 2
3. in melt_tagger.py, add , encoding='latin1' as an argument to np.load

Not sure if still works on Python 2 with these modifications.

Should mark it as Python 2 somewhere until Python 3 support added.

With newer version of spaCy (cf explosion/spaCy@62b4b52#diff-f9114e62ea7b18cbc72d24ccb00c61a5R32), an error appear when setting a new extension to a Token.
Build failed : https://travis-ci.org/sammous/spacy-lefff/builds/357411570#L620

E   ValueError: [E090] Extension 'lefff_lemma' already exists on Token. To overwrite the existing extension, set `force=True` on `Token.set_extension`.

Hi,

I have an problem of compatibility between spacy, fr-core-news-sm and spacy-lefff

spacy-transformers 1.0.6 requires spacy<4.0.0,>=3.1.0, but you have spacy 3.0.4 which is incompatible.
fr-dep-news-trf 3.1.0 requires spacy<3.2.0,>=3.1.0, but you have spacy 3.0.4 which is incompatible.

spacy-lefff 0.4.0 requires spacy<3.0.5,>=3.0.0, but you have spacy 3.2.1 which is incompatible.

It's possible to have an update of requirements.txt of spacy-lefff to, at least, spacy 3.1 ?

Hello,

i'm using Spacy-Lefff with Mac 10.11.6 running python 3.7.

I can't find a way to avoid printing these informations :

2019-08-29 13:33:03,196 - spacy_lefff.downloader - INFO - data already set up
2019-08-29 13:33:03,197 - spacy_lefff.melt_tagger - INFO -   TAGGER: Loading lexicon...
2019-08-29 13:33:04,339 - spacy_lefff.melt_tagger - INFO -   TAGGER: Loading tags...
2019-08-29 13:33:04,398 - spacy_lefff.melt_tagger - INFO -   TAGGER: Loading model from /usr/local/lib/python3.7/site-packages/spacy_lefff/data/tagger/models/fr...
2019-08-29 13:33:05,311 - spacy_lefff.melt_tagger - INFO -   TAGGER: Loading model from /usr/local/lib/python3.7/site-packages/spacy_lefff/data/tagger/models/fr: done
2019-08-29 13:33:05,311 - spacy_lefff.lefff - INFO - New LefffLemmatizer instantiated.
2019-08-29 13:33:05,312 - spacy_lefff.lefff - INFO - Reading lefff data...
2019-08-29 13:33:06,414 - spacy_lefff.lefff - INFO - Successfully loaded lefff lemmatizer
2019-08-29 13:33:06,439 - spacy_lefff.melt_tagger - INFO -   TAGGER: POS Tagging...

Is there a way to do this ?

Add a table to the README for the benchmark on common datasets for POS tagging in French, and compare it to built-in spaCy.

Lefff and MElt use different mapping for the part of speech tagging.
For instance, one say prep and the other p to qualify a preposition.
Though, the lemmatization uses the part of speech tagging of MElt when it is added to a pipe in spaCy.

So in the example in the README, we can see that the lemma for de or dollard are not found, yet they do exist in Lefff (unlike Apple).

Code

• update the file mappings.py
• make sure that when MElt was used, it uses the correct mapping to get the lemma, at line .

Hello Hello,

I wanted to use your library for a NLP project but when I try to use the POSTagger, I get the following error:
FileNotFoundError: [Errno 2] No such file or directory: '*/.user_conda/envs/conda_env/lib/python3.7/site-packages/spacy_lefff/data/tagger/models/fr/lexicon.json'

The line that make the error is this one:

pos = POSTagger()

There is some more files to download or i messing something?

Thank you and have a great day.

Woud be nice to have a preview of the package (to engage people) with the actual prints from the code snippet

With Spacy version 2.1.3. on Windows 10, Python 3.7.2
With the fr_core_news_md model language
There is a bad analysis of a construction such like this one (a verb followed by an adjective)

"Une personne fait une déclaration qu'elle sait fausse ou trompeuse."

The verb "savoir" is analysed as an auxiliary which is wrong because this is a full verb. As well, the token "fausse" is wrongly analyzed as a verb whereas it's an adjective (at least trompeuse is correctly analysed as an adjective).

This analysis has not been tested with spacy-Leff (I am attempting to install it but installation problems/warnings have prevented me to do so until now - still trying to resolve the issue), but this improvement should definitely be implemented in a future version of spacy-Leff.

Thanks!

Hello,

thanks for the greate work, just got an issue trying to custumize dir where to download model :

melt_tagger.py init :

    def __init__(
            self,
            data_dir=DATA_DIR,
            lexicon_file_name=LEXICON_FILE,
            tag_file_name=TAG_DICT,
            print_probas=False):
        super(
            POSTagger,
            self).__init__(
            PACKAGE,
            url=URL_MODEL,
            download_dir=DATA_DIR)

data_dir=DATA_DIR is a parameter that is not used as it calls it super with download_dir=DATA_DIR.
Shouldn't it be download_dir=data_dir ?

I created a PR #28

Regards.

Hello ! just starting to explore spacy-leff : I have been trying to run the examples given here and on spacy.io
but I stumble on the same error. See below.
I assume it's a simple one, but I did not find help for this in the available documentation (I may not be good at looking at the right place though...).

I intend to use specy-leff, to replace MElt perl utilities, for our bamanan-french parallel corpus. Bamanan or bambara is a language spoken in west africa. See http://cormand.huma-num.fr/

Thanks for help with this...
Jean-Jacques

python3 testleff2.py
Traceback (most recent call last):
File "testleff2.py", line 9, in
nlp = spacy.load('fr_core_news_sm')
File "/usr/local/lib/python3.8/dist-packages/spacy/init.py", line 47, in load
return util.load_model(name, disable=disable, exclude=exclude, config=config)
File "/usr/local/lib/python3.8/dist-packages/spacy/util.py", line 329, in load_model
raise IOError(Errors.E050.format(name=name))
OSError: [E050] Can't find model 'fr_core_news_sm'. It doesn't seem to be a Python package or a valid path to a data directory.

Problem

Sometimes, a word can be at the same time an adj and a nc and it is missing.
An example in Lefff is costaricain, which can be both at the same time but it is not taken into account in Lefff.

Problem

Current problem is to be able to take advantage of Lefff exhaustivity by having a good French POS in the Spacy pipeline before the lemmatization process.

Solution

A possible solution is to take advantage of the POS developed by the same persons of Lefff called MElt available here

seems the word "DEV" triggers an error.

import spacy
from spacy_lefff import LefffLemmatizer, POSTagger
nlp = spacy.load('fr_core_news_md')
pos = POSTagger()
french_lemmatizer = LefffLemmatizer(after_melt=True)
nlp.add_pipe(pos, name='pos', after='parser')
nlp.add_pipe(french_lemmatizer, name='lefff', after='pos')
doc = nlp(u"Paris est une ville très DEV ce jour.")
for d in doc:
    print(d.text, d.pos_, d._.melt_tagger, d._.lefff_lemma, d.tag_, d.lemma_)

Traceback (most recent call last): File "test-lemm.py", line 11, in <module> doc = nlp(u"Paris est une ville très DEV ce jour.") File "/usr/local/lib/python3.6/dist-packages/spacy/language.py", line 402, in __call__ doc = proc(doc, **component_cfg.get(name, {})) File "/usr/local/lib/python3.6/dist-packages/spacy_lefff/melt_tagger.py", line 242, in __call__ beam_size=beam_size) File "/usr/local/lib/python3.6/dist-packages/spacy_lefff/melt_tagger.py", line 194, in tag_token_sequence best_sequence = sequences[-1][0] IndexError: list index out of range