Retrieve loaded data as pandas.DataFrame about pyorc HOT 9 OPEN

You're probably right, separate classes for reading and writing are overkill. Maybe two simple functions defined in a submodule could be sufficient enough.

I don't want to add pandas as a default requirement to the module.

from pyorc.

Honestly, I'm not that familiar with pandas. I have to look into more, but the simplest solution to add pandas as an extra to the module, and inherit a new Reader/Writer from the existing ones with methods that can read and write pandas dataframes. There's probably a need for special converter functions for pandas' special types as well.

As you've already noticed, my goal with this library is to be a simple ORC reader and writer with the smallest overhead as possible. A few smaller tasks are on my todo list now, but I'm not against the idea. The best would be, if someone with a better knowledge of pandas could contribute. 😉

from pyorc.

Therefore, what do you suggest to solve this issue? These are the options I guess:

1. PandasReader & PandasWriter (I don't like this one)
2. "as_pandas" as method or argument on reader (but this adds pandas as dependency)
3. Just an example of how to get a Pandas DF from loaded data.
4. Nothing to do (just add a comment to let users understand the scope of pyorc)

Let me know what is your concern on this and then I could make a PR related to that.

from pyorc.

OK so you prefer to add functions/methods for that, but not to require pandas to use pyorc. I can add this behavior without adding Panda as a requirement, and raise an error if Pandas is not installed, but that will be a very bad practice. Therefore, the best option will be to just add an example with Pandas for start, and if the example is not enough you could consider expanding the scope of pyorc to be more integrated with Pandas (which is my suggestion, since it's a pretty common requirement in most of data processing libraries). Let me know your thoughts and I can make a PR from that (either adding an example or adding methods with Pandas)

from pyorc.

I think an example would be great as a start. A PR about that would be much appreciated. Thank you.

from pyorc.

Can someone please include a short example of how to use converters in the Reader? I tried really hard to figure this out but I couldn't. I'm reading a file like this where orc_bytes is of class bytes:
orc = pyorc.Reader(fileo=io.BytesIO(orc_bytes))
This works fine and I can convert the resulting Reader object to a pandas dataframe. Now I am trying to add a converter to Reader to convert decimal to float upon reading. I know it needs a dictionary with keys being TypeKind but I can't figure out how to pass the dictionary values. So I am stuck at:
orc = pyorc.Reader(fileo=io.BytesIO(orc_bytes), converters={pyorc.TypeKind.DECIMAL: ???})
I found an example in test_reader.py file (https://github.com/noirello/pyorc/blob/master/tests/test_reader.py#L325) that uses ORCConverter to define a class and a from_orc method for TypeKind.TIMESTAMP but I have no idea how this should be defined to convert decimal to float. Any help please?

from pyorc.

This is what I have so far but it is not doing any conversion:

import numpy as np
import pyorc
from pyorc.converters import ORCConverter

class TypeConverter(ORCConverter):
    @staticmethod
    def from_orc(decimal_input):
        return np.array(decimal_input, dtype=float)

orc = pyorc.Reader(fileo=io.BytesIO(orc_bytes), converters={pyorc.TypeKind.DECIMAL: TypeConverter})

Any suggestions?!

from pyorc.

I updated the docs about ORCConverter

Your converter above should return a numpy array with a float in it, for every item in a decimal ORC column.

from pyorc.

@noirello Thanks a lot! Highly appreciated.

from pyorc.