[BUG] clx.analytics.tokenizer gives error to sentence length > 500 about clx HOT 3 CLOSED

We recently moved the subword tokenizer from CLX to cuDF (rapidsai/cudf#5511), where it's now a cuDF function that can be called on a cuDF Series. In doing so, we patched a number of things and improved the performance. We're also removing the 500 character limit (rapidsai/cudf#5919). Using the new cuDF version of the tokenizer is very similar, and here is a notebook where we use it: https://github.com/rapidsai-community/notebooks-contrib/blob/branch-0.14/conference_notebooks/KDD_2020/notebooks/cybert/cyBERT_training_inference.ipynb. There is a slight change in what it returns (API ref).

from clx.

@colpark Just curious if you were able to test the tokenizer now that it's in cuDF and if you're still running into issues.

from clx.

@colpark We're going to close this issue. If you're still having trouble or there is something else happening, please feel free to open a new issue and we're happy to take a look.

from clx.