Filter with `UserDefined` predicates? about parquet4s HOT 5 CLOSED

Sounds great. I prefer the 1st option, too, so I'll get started on that.

I'll probably start with the SQL-like in function. That, together with the existing operators, would cover 90% of the use cases I can think of. Let me know if you think otherwise.

from parquet4s.

Hi Sean!

I see two ways that you can go.

Take a look at Col case class here https://github.com/mjakubowski84/parquet4s/blob/master/core/src/main/scala/com/github/mjakubowski84/parquet4s/Filter.scala#L133

1. You can create a PR that introduces your desired functionality in Col. It can be SQL-like in function, it can be userDefined - I will be very happy with any of them. Just extend API by adding new functions. Take example from existing operators.

2. You can introduce a static function in your own code. Your function would be very similar to operators in Col. For example, you can create implicit class that wraps Col.

Of course, I prefer the first option, so other people could benefit from your solution!

from parquet4s.

@SeanU in filtering operator has been release in 0.11.0. Do you want to continue and create generic userDefined operators, too?

from parquet4s.

@mjakubowski84 Thanks!

I would be glad to do generic userDefined operators, too I may have to do it on a somewhat more relaxed schedule. Shall we leave this issue open until I do? Or close it for now, and I can open a new one when I start work?

from parquet4s.

from parquet4s.