Comments (4)
After a bit of hacking, I got this working:
In [1]: from ibis.interactive import *
In [2]: @ibis.udf.scalar.python
...: def add_one(x: int) -> int:
...: return x + 1
...:
In [3]: import cloudpickle
In [4]: cloudpickle.loads(cloudpickle.dumps(add_one))
Out[4]: <function __main__.add_one(x: int) -> int>
from ibis.
From looking into this further, I think this is related to the fact that cloudpickle doesn't serialize functions with type annotations e.g. cloudpipe/cloudpickle#541.
I see the same error with the following
@ibis.udf.scalar.python
def add_one(x: int) -> int:
return x + 1
with open("func.pkl", "wb") as f:
cloudpickle.dump(add_one, f)
with open("func.pkl", "rb") as f:
loaded_model = cloudpickle.load(f)
from ibis.
cloudpickle
does seem to preserve annotations (I think cloudpipe/cloudpickle#541 is about annotating the cloudpickle
library itself, not about supporting them in pickling/unpickling).
Perhaps there's an issue on our side.
In [1]: import cloudpickle
In [2]: def add_one(x: int) -> int:
...: return x + 1
...:
In [3]: func = cloudpickle.loads(cloudpickle.dumps(add_one))
In [4]: func
Out[4]: <function __main__.add_one(x: int) -> int>
In [5]: func.__annotations__
Out[5]: {'x': int, 'return': int}
from ibis.
To add to this issue, I've been investigating using pickle
instead, but have hit a blocker with a case
statement
import pickle
from ibis import _
import ibis
c = ibis.case().when(_.s == 1, "EXACT").when(True, "DIFFERENT").end()
pickle.loads(pickle.dumps(c))
raises
PicklingError: Can't pickle <function _finish_searched_case at 0x78fdf8061900>: it's not the same object as ibis.expr.builders._finish_searched_case
I am able to pickle/unpickle this using cloudpickle
however
from ibis.
Related Issues (20)
- docs(ops): operation matrix for streaming ops
- bug: Error Reading Delta Lake Tables With New delta_scan Call HOT 3
- feat: Windowing/aggregating over literals HOT 2
- feat: Support `QUALIFY` syntax to filter on analytic expression HOT 1
- feat: `to_interval` for Snowflake backend
- refactor(api): rename `to_*` methods for type coercion to something else
- bug(pkl): some case expressions cannot be pickled HOT 2
- feat: ArrayCollect aggregation for Flink backend HOT 2
- feat: pandas backend cannot use sql, can you add this? HOT 2
- feat: add Excel backend is necessary HOT 2
- bug: repr for count now includes np.int64 or whatever type HOT 5
- [meta]: Expression IR performance
- feat: Connecting snowflake using Key Pair Authentication HOT 2
- feat: MonetDB as backend HOT 3
- bug: mssql pandas data can't be converted by correct type
- feat: MariaDB as new Backend HOT 3
- bug: error with create_materialized_view with Risingwave backend HOT 5
- refactor(api): remove casting from timestamp to integer and integer to timestamp HOT 5
- feat(tables): show whether abstract table is table or view (or materialized, etc...) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ibis.