Integer datatypes with missing values changes to object in python using pyreadr package, after importing data from RData file about pyreadr HOT 5 CLOSED

yes, this is currently the correct expected behavior. Why? because in older version of pandas it only used numpy arrays in the back, if you have a numpy array of type integer, it would not let you put a np.nan, which is of a float type, therefore to mix integers and nans, you needed an object type series.

In more recent version of pandas, they have introduced a nullable integer pandas array (IntegerArray. I can try to use that one, since using the integer numpy array is still impossible. Would that help?

from pyreadr.

I think it would be great to have nullable integer pandas array, because it would help to maintain the data types integrity when sending data.frames back an forth from Python to R.

from pyreadr.

Could you please elaborate on why you need the type of the column to be nullable integer instead of object?

Right now the python object column with integer values should be correctly translated into an R Integer type column, so if the main use case is going back and forth between python and R, there is currently no issue and no change needed.

Changing the datatype to nullable integer is not so trivial, and in addition can potentially break people's code (somebody in her code is expecting an object column and now suddenly gets a nullable integer type), so I am reluctant to do it unless there is a strong reason.

from pyreadr.

Another thing to take into consideration and will not be solved by the nullable type by itself (this is described in the Readme ) but you can solve manually already is the following:

R integer type is a 32 bit integer. Python has 64, 32, 16 and 8 bit integers also with unsigned versions. When you read a RData file with an integer type it will be translated to a numpy 32 bit integer. If you immediately write it back to R, it will be translated to an Integer column.

However if you create your own integer columns in pandas, by default they are 64 bit integer. This cannot be converted to a 32 bit integer, because there is the risk of an overflow. Therefore it has to be converted to a R numeric type (float64) to avoid any overflow. That means, if you have integer columns in your data, you have to make sure they are or convert them manually to an 32 bit integer (you can use the nullable 32 bit version) and then it will be converted to an R integer column type.

from pyreadr.

closed since there is no activity.

from pyreadr.