Access to filter method in handyspark API about handyspark HOT 2 CLOSED

Hi,

Thanks! :-)
So, regarding the filtering, you can use Spark's own expr function to easily come up with expressions that will return the filter criteria. For instance, if you would like to get only rows where Age is greater than 25:

from pyspark.sql import functions as F
sdf.filter(F.expr('Age > 25'))

There is no need to create a column, Spark will evaluate the expression and filter it accordingly.

But if you want to do something fancier, you can leverage the pandas object of HandySpark as well. Let's say you want to filter only for a list of specific values:

hdf.filter(hdf.pandas['Age'].isin(values=[25, 26]))

HandySpark implements several column methods available in Pandas, like isin.

from handyspark.

Great Thanks, that worked, good to know I can filter within Handy.

from handyspark.