Make RegressionResults serializable about gneiss HOT 7 OPEN

I like pickle files, easy with Pandas.

from gneiss.

Pickle is easy but python specific. TSV may be nice as it allows for operability easily in R or Python

from gneiss.

Good point.

Note that there are advantages of saving the raw Python object. There are functions like predict that cannot be serialized, since it operates directly on the RegressionResults object. Right now, it is possible to serialize individual attributes such as residuals, so this functionality is somewhat supported.

from gneiss.

@wasade @antgonza it looks like pickle is not going to work, particularly with large trees. I get the following error with a tree with 14000 tips

RecursionError: maximum recursion depth exceeded while pickling an object

I guess we'll want to invent a file format to store all of this information ...

from gneiss.

store the tree as a newick string?

from gneiss.

Question @wasade when it comes to converting objects to json, there are any limitations?

Right now, part of the problem is that there will need to be Python objects serialized such as OLSResults. If so, then it may doable to have a json parser on top of a pickle parser.

from gneiss.

This is going to be increasingly necessary as we string up more tree visualizations to the CLI.

Right now, I'm thinking about having two main types of file formats, namely. Model and Results formats. The Model file formats are only responsible for storing the barebones model parameters. For example, the coefficients in a regression model, and the tree used to generate the balances. And these models can be used for making predictions.

The Results model format will be much larger, storing all of the model statistics and residuals. Maybe using something like pytables + hdf5 will be more suitable for handling these large tables.

from gneiss.