Comments (2)
alxmrs
commented on May 8, 2024
I'm looking into this now.
I have a design question, though: What is the best PCollection interface? tuple[xbeam.Key, tuple[xarray.Dataset, xarray.Dataset]] or tuple[xbeam.Key, xarray.Dataset, xarray.Dataset]? I have a slight preference for the latter (and, this implementation would not be so bad). This version also seems fairly natural for operations like beam.MapTuple() (can handle n-ary tuples) and beam.GroupByKey() (will just use the first value in the tuple). It feels more zen to me, too. :)
WDYT?
from xarray-beam.
alxmrs
commented on May 8, 2024
A small update -- the former does seem to offer a better typing story (python/typing#180), I now am leaning that way.
from xarray-beam.
Related Issues (20)
- Figure out how to combine efforts with xpartition HOT 7
- Support for striding / rolling windows HOT 1
- Consider adding ZarrToChunks() and/or an open_zarr() helper function
- Support pangeo_forge_recipes.patterns.MergeDim in `FilePatternToChunks`
- Some notes on alignment with pangeo-forge-recipes around "keys" HOT 1
- Better documentation of behaviour with irregular source chunks HOT 5
- Support opening datasets with file-like objects in a Beam pipeline
- Add `split_vars` to FilePatternToChunks transform.
- Document how to add `weather-dl` clients & manifests in the Contributing guide. HOT 1
- Running pipelines on AWS? HOT 1
- Consider omitting unchunked dimensions from Key objects created with DatasetToChunks HOT 1
- Adding source distribution for xarray-beam on PyPI HOT 2
- Help with opening netcdf4 files via HTTP HOT 19
- Support missing chunks in ConsolidateChunks
- Require using make_template() if providing a template to ChunksToZarr? HOT 2
- Add CI to check that code has been formatted with pyink formatter.
- Error in ChunksToZarr appears on docs page
- Register Beam coder(s) to avoid "Using fallback deterministic coder" warnings
- Race condition in ChunksToZarr when template is not supplied explicitly as an xarray.Dataset
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xarray-beam.