mpiannucci / gribberish Goto Github PK

A clean example for this is NWPS grib files, which contain all timesteps and all swell components in one big grib message.

This comes up with the dense hrrr grid

When there are multiple time dimensions in a dataset, the kerchunk mapping will conflate all of the time readings and mash them together, when instead there should be different time dimensions for each time horizon.

This is handled in cfgrib by the filter rules and not letting multiple time horizons, but I do not want that solution

Use pyi files

Currently not being used but need to be

This looks good:

This does not....

https://github.com/wmo-im/GRIB2

Use from_vec instead of from_slice so the rust memory is used instead of allocating another memory block on the heap

Performance is currently much worse than cfgrib:

FIGURE IT OUT

Running this code after loading in the multizarr will give a different result over and over again. When the same code is run against the single zarr, there is no such instability.

EDIT: Nevermind happens with single zarr too :(

There is probably about a 50% hit rate with getting garbage or correct data

1. Set the preferred_chunks attribute on the variable
2. Read only the necessary chunks when user indexes into backend array

Latitude and longitude direction increments should be switched

https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_temp5-41.shtml

Currently have to hack to get unique names with variable levels. Need to clean this up appropriately

Each message comes with its grid template information, which has all the information to generate a proj string to enable reprojection

See GRIBCodec

http://gradsusr.org/pipermail/gradsusr/2008-July/007358.html

https://github.com/j-m-adams/GrADS/blob/master/src/gribmap.c

EDIT: I think the formatting comes from wgrib2:

Vertical Levels

https://github.com/NOAA-EMC/NCEPLIBS-grib_util/blob/558fad4ae6121b5e1754177839cf7c8179abcb26/src/wgrib/wgrib.c#L2004

https://github.com/NOAA-EMC/NCEPLIBS-grib_util/blob/558fad4ae6121b5e1754177839cf7c8179abcb26/src/degrib2/prlevel.F90#L57

Currently they are ignored so there becomes duplicate variables for multilevel vars

https://docs.xarray.dev/en/stable/internals/how-to-add-new-backend.html#rst-lazy-loading
https://github.com/ecmwf/cfgrib/blob/master/cfgrib/xarray_plugin.py

https://xarray.pydata.org/en/stable/internals/how-to-add-new-backend.html

Have the templates conform to trait that returns projection generator for given indices

Support the inverse of drop_variables specifically, like only_variables