What packages should we use to store data? about plasmapy HOT 17 CLOSED

What about finding a way to be agnostic to some of these other packages? One way I get around subclassing things like numpy and pandas is doing the following.

    class Plasma(object):
        __init__(self, data):
            self.set_data(data)
        @property
        def data(self):
            return self._data
        def set_data(self, new):
             assert isinstance(new, (pd.DataFrame, xr.DataArray, ....))
             self._data = new
        
        def do_physics(self, *args, *kwargs):
              data = self.data
              out = physics(data)
              return out

If we do something like this, there are huge piles of code I can contribute to the project (10,000s of lines, across many files). There are also benefits of doing things like not assuming a specific particle species. For example, some solar wind people include ions of oxygen and nitrogen, among others, along with the typical protons. But some people really only care about protons or protons & electrons. Being agnostic to these by developing the code in appropriate ways would make the PlasmaPy project accessible to a much larger field of users.

from plasmapy.

And with regards to the name I'd say we still want to keep everything within PlasmaPy and have experiment and simulation (or whatever) as top-level submodules.

from plasmapy.

What about HDF5? We can use PyTables, Pandas, and h5py to handle HDF5 files in different ways (with different learning curves). There is already archiving and compression functionality built-in. It's cross-platform, highly-highly-optimized (fast read/write for massive simulation/experimental data files), and language agnostic. This latter point would be important for converting to other formats like CDF.

from plasmapy.

It's also worth noting that with something like Pandas we get to leverage all of the Cython level code that the package has completed.

from plasmapy.

Hey all,

I dig into yt (on their site ). They seem to have everything we need !!

• units and dimensions label
• can load lots of common data type (see this list )
• grid and particle representations as well as data-frame like Pandas
• conda's supported
• has a large community
• seems fast and easy from the examples, I did not check it myself.

Moreover :

• They use Cython and can be used in parallel (see here)
• There are lots of implemented functions, from field analysis to particle to cell
• a lot of work on visualization as been done

On my opinion, yt seems to be the best solution. On the other hand, I never tried it myself, so I am not confident in building a whole package over it.

I don't know how would you like to make the decision (by a vote, by example ?). I think we can fix a "decision date" like September the 1rst, let everyone think about it and then make the decision.

PS : @bla1089 in your example : assert isinstance(new, (pd.DataFrame, xr.DataArray, ....)) do you expect new to be either a pd or a xr data, and PlasmaPy to be able to deal with both of them or did you used the two only for the sake of the example ?

from plasmapy.

from plasmapy.

@bla1089's abstraction idea seems reasonable. I'm not 100% sure how it'd work in practice just yet, but... I'm pretty sure I'd need pointers on implementing this kind of thing.

As for data structures, at a glance yt does seem to be the better fit at least for huge simulation etc data, given that it also seems to make grid visualization easier. But it seems like there's nobody among us but Cadair who's used it yet! I've started learning it, for what it's worth.

from plasmapy.

@SolarDrew has used it a bit as well.

from plasmapy.

Indeed. I've found it really good for storing and plotting simulation data, but I don't know what kind of support it has for timeseries-type data of the kind @bla1089 has been talking about.

That said, I think it's great for pretty much everything else though, so in practice I think the real solution is going to be to use both as appropriate and try to provide consistent wrappers so the user doesn't need to worry what's underneath.

from plasmapy.

from plasmapy.

Sure, that sounds pretty reasonable. I wouldn't mind this being the semi-final consensus we reach, as we'll probably eventually need to split simulation and experiment up anyway.

from plasmapy.

from plasmapy.

Yeah that's pretty reasonable, I had considered we might have to do that. If possible I would like to keep a single mostly-consistent API, or at least have a single contructor that returns an appropriate subclass of Plasma, a la SunPy Maps. But yes, if necessary having two would be a decent compromise.

from plasmapy.

Also, just for the record here, someone is working on a pure-python implementation of a CDF package: https://github.com/MAVENSDC/cdflib

I believe it is currently read only.

from plasmapy.

from plasmapy.

Do you have links to the others? Perhaps we should get them talking? This one is being written by someone I know at GSFC.

from plasmapy.

from plasmapy.