Minimum text length violated despite preprocessing about declutr HOT 2 CLOSED

Hi, I have a similar issue: I get ValueError: len(tokenizer(text)) should be at least <number here> (num_anchors * max_span_len * 2), got 0

I have filtered my input data to have both minimum length beyond parameter min_length and greater than 0, of course! I also tried to print out the text to be tokenized, it was empty!? How can that be?

I use a pretrained model from Sentence Transformers, sentence-transformers/paraphrase-multilingual-mpnet-base-v2, so the tokenizer comes from that model - do you have any idea, why the text is empty?

EDIT: I found the problem, I was missing out some carriage returns (\r) when filtering the input which obviously led to an empty document somehow...

from declutr.

This check uses the tokenizer passed to dataset_reader.py IF it is a PretrainedTransformerTokenizer. Otherwise is uses .split().

Did you use the same tokenizer to filter your dataset as the tokenizer you passed to dataset_reader.py?

from declutr.