Comments (4)
I think having separate methods makes sense for this, while keyword arguments clutter the API.
How about load_*
equivalents that unconditionally persist data and mirror the read_*
APIs?
from ibis.
I think best distinction we can do between any approach here is:
- Things persist after the session ends
- Things do not persist after the session ends
The former almost certainly requires physical storage (ruling out views), while the latter can be more less or whatever's well-suited to backend.
If people need something specific, they have to call create_{view,table}
from ibis.
I think having separate methods makes sense for this, while keyword arguments clutter the API.
If these APIs are identical except for how the backend loads the data (a binary option), I'd argue that a keyword argument makes more sense than duplicate methods which do the same thing for most backends.
How about
# cache=False is whatever the backend currently does, cache=True always uses a table
t = con.read_parquet(..., cache=True)
How about
load_*
equivalents that unconditionally persist data and mirror theread_*
APIs?
I hadn't considered the temporary/persistent choice, but I suppose that is another axis. Since our read_*
methods generate a unique name by default (and anything that persists permanently should probably have an explicit name), I'm not sure if a binary modality here would make sense, and agree a new load_*
method might make more sense (so we could better indicate and enforce that table_name
is required).
In my case though, I don't really want a persistent table, but do want the load to happen only once for the lifetime of the session. In this case a simple kwarg (that really only needs to be implemented by the few backends that use views by default) seems like the simpler option.
from ibis.
After talking this through, I think this issue would be better resolved through fixing #6195, and documenting this pattern for backends where it's relevant (mostly just duckdb). Closing.
from ibis.
Related Issues (20)
- refactor(api): remove casting from timestamp to integer and integer to timestamp HOT 5
- feat(tables): show whether abstract table is table or view (or materialized, etc...) HOT 1
- feat: Datafusion add support for using a list of paths when reading csv parquet etc HOT 3
- bug(clickhouse): query clickhouse table with uint256 type columns
- bug(bigquery): Syntax error when applying an empty window spec HOT 3
- bug: AttributeError: 'StringColumn' object has no attribute 'as_date' HOT 3
- feat: make it possible to opt-in to query caching for clickhouse (and perhaps others) backends too HOT 13
- bug[duckdb-geospatial]: read_parquet defaults silently to pyarrow reading geometry as binary HOT 4
- bug(flink): nested struct raises SQL validation error
- feat: `to_dask` (or more generally `to_batches`?) HOT 5
- ci(oracle): flaky test assumes tables exist HOT 1
- bug: methods like `get_name()` and `type()` do not work with underscore API HOT 10
- feat: simplify generated select distinct expressions HOT 1
- ERROR When Using MSSQL XML Functions thru Connection.sql HOT 1
- refactor: a couple selectors rough edges
- feat(consider): Make `isinstance(ibis.Deferred, ibis.Value)` return true HOT 1
- feat(udf): support both ibis and original types HOT 2
- bug(duckdb, sqlite): setting a file-backed backend doesn't write to file HOT 2
- feat: add `name` param to `.value_counts()` and `topk()`
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ibis.