Adding StopWordsRemover about spark-nlp HOT 4 CLOSED

How do I add the pyspark.ml.feature StopWordsRemover to the spark-nlp_2.11-1.2.3.jar file?

from spark-nlp.

@clayms As I see you have a line self._java_obj = self._new_java_obj("com.johnsnowlabs.nlp.annotators.StopWordsRemover", self.uid)
But there is not StopWordsRemover in package com.johnsnowlabs.nlp.annotators. I think that is the reason why you got this Exception.

I suggest to add pyspark StopWordsRemover as a first stage of your pipeline.

from spark-nlp.

Thank you @aleksei-ai . I see that now. I got the Spark ML StopWordsRemover to work previously together with the Spark ML RegexTokenizer, but was having trouble getting them to work in the same pipeline as the John Snow Labs annotators. I ended up putting those Spark ML stages at the end of the John Snow Labs annotator stages. The end results are what I am after. Thank you.

from spark-nlp.

In the upcoming release of Spark NLP 2.3.0, we will have a native StopWordsCleaner annotator.

from spark-nlp.