mikahama / natas Goto Github PK

Hey, would use is_correctly_spelled for OCR errors but it seems overly generous.

I guess the pages that it finds are https://en.wiktionary.org/wiki/HATO and https://en.wiktionary.org/wiki/Teri, it nicely failed at 'itsejf'.

I wonder if it could make a more conservative match, and only offer lowercase options as real words? Perhaps optionally?

Thanks!

First off, this is a great library. I've been quite impressed with the results of its normalization functionality. One thing I'd be keen to see, however, is some kind of metric with which to associate the topn candidate words produced by the normalization process (say, for instance, vector similarity or the NMT model's prediction score). Is Natas already capable of doing this (in which case I'm missing something), or are there plans to implement such functionality?

Running test examples, it seems to work very well, except there seems to be a problem using a set here. I'm probably just using it wrong, so advice is helpful.

seed_words = set(["logic", "logical"]) #list of correctly spelled words you want to find matching OCR errors for
dictionary = wiktionary #Lemmas of the English Wiktionary, you will need to change this if working with any other language
lemmatize = True #Uses Spacy with English model, use natas.set_spacy(nlp) for other models and languages

results = ocr_builder.extract_parallel(seed_words, model, dictionary=dictionary, lemmatize=lemmatize)

I get the error, TypeError: 'set' object is not subscriptable

Any idea what might be going on? Thanks!