crazyfrogspb / redditscore Goto Github PK

Steps

from redditscore.tokenizer import CrazyTokenizer

tokenizer = CrazyTokenizer(hashtags='split')
tokenizer.tokenize("#20yearsago")

Actual Result

['2', '0', 'y', 'e', 'a', 'r', 's', 'a', 'g', 'o']

Expected Result

['20', 'years', 'ago']

With the CrazyTokenizer (excellent results, btw, thanks!) I am running into an issue with a maximum character length for SpaCY: "[E088] Text of length 3029371 exceeds maximum of 1000000."
You can change nlp.max_length , but for that you have to load spacy itself.
Is there a way that nlp.max_length can be set. when loading the CrazyTokenizer?
(I know I could simply cut the file in 3, but I'd rather avoid that as I'd have to manually stitch the resulting token set back together again and I'll have to do this for various files).

There used to be no problems when importing CrazyTokenizer in Colab until February, but now it throws an error of "Protocol not found".

To reproduce errors, Type:

!pip install git+https://github.com/crazyfrogspb/RedditScore.git
from redditscore.tokenizer import CrazyTokenizer

Maybe it's because of some updates made? Is there any way to downgrade the version when installing?

Installation

pip install git+https://github.com/crazyfrogspb/RedditScore.git

Steps to reproduce

from redditscore.tokenizer import CrazyTokenizer

tokenizer = CrazyTokenizer(hashtags=False)
text = "Let's #makeamericagreatagain#americafirst"
print(tokenizer.tokenize(text))

Expected output

["let's", "#makeamericagreatagain", "#americafirst"]

Actual output

['let', "'s", '#makeamericagreatagain', '#americafirst']

Environment

Kaggle's Notebook