Error in unpickly root file with python3 about rootpy HOT 8 OPEN

Hi,

I have tried to pickle and load some data, but can not reproduce your problem. However, these type of errors are common in python3 imported code and I would not be surprised if pickle module has some problems with python3.
If you could share your input file, I might have a look at this, should not be hard to fix.

Cheers!

from rootpy.

First thing tomorrow I will share the file!

from rootpy.

Hi!

geofile_full.conical.Pythia8-TGeant4.root.zip

Here is the file.

Let me know how I can help investigate the issue!

from rootpy.

Hi,

I can not load this file even in python 2.7.15. Generally, pickles are not meant as long term storage. If you use some other python version for loading, different from that which was used for pickling, problems are expected. I think if you will pickle and unpickle in the same python3 version, this approach should work.

from rootpy.

I can re-run the simulation but I am pretty sure that the file was generated (pickled) by the exact same python version that is trying to unpickle it.

It is the pipeline here: https://github.com/ShipSoft/FairShip#run-instructions

run_Script.py generate the file and then ShipReco.py try to read it back.

Do you have any idea of what could it be or where to look? I spend quite some time trying to debug this so any hint is very appreciated!

from rootpy.

Your output:

from rootpy.io.pickler import load

In [23]: l = load('geofile_full.conical.Pythia8-TGeant4.root', key='ShipGeo')                                           
INFO:rootpy.io.pickler] Making dummy module copy_reg
INFO:rootpy.io.pickler] Making dummy class copy_reg._reconstructor
INFO:rootpy.io.pickler] Making dummy module __builtin__
INFO:rootpy.io.pickler] Making dummy class __builtin__.dict

shows that rootpy is trying to pickle from copy_reg._reconstructor and __builtin__.dict classes. To me it is surprising why this classes are present in a pickle made by python3. In python3 we do not have such modules, we have copyreg and builtin instead. So python3 can not import them and rootpy creates Dummy classes.

Could you please confirm again, that you are trying to read pickle, which is made by python3?

Looking at your file, I see besides that classes there are two classes used from ShipGeoConfig module, which probably contain main information. This is probably defined and read from FairShip and problem should no be there (there are no complaints in your logs).

TypeError: object() takes no parameters

This error is caused, because the class which rootpy tries to pickle from does not have a __init__() method. Here is described this problem in more details, and why it works in python2:
https://stackoverflow.com/a/34503234/11455115

The workaround for your problem would be to modify this file:
~/.local/lib/python3.6/site-packages/rootpy/io/pickler.py
Line 323:

class Dummy(object):
  __init__(self, *args):
    pass
  __setstate__(self, state):
    return self.__setstate__

If copy_reg and __builtin__ classes are not important for your pickle, that should work.

from rootpy.

Hi there!

yes I confirm that the pickle is created by Python3.

[FairShip/latest] ~/ship $> python
Python 3.6.8 (default, Jul 23 2019, 09:35:39) 
[GCC 6.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
[FairShip/latest] ~/ship $> 
[FairShip/latest] ~/ship $> python $FAIRSHIP/macro/run_simScript.py --test
FairShip setup for Pythia8 to produce 50 events
[INFO] FairRootManager::OpenOutFile("./ship.conical.Pythia8-TGeant4.root")
[INFO] Media file used: /root/ship/sw/slc7_x86-64/FairShip/master-1/geometry/media.geo
...
--------------  input/output  --------------------------------------------------
first input: none
second input: none
output:
OBJ: FairParRootFile    ./ship.params.conical.Pythia8-TGeant4.root       : 0 at: 0x7a349290
Root file I/O ./ship.params.conical.Pythia8-TGeant4.root is open
detector I/Os:  FairGenericParIo
 
Macro finished succesfully.
number of retries, events without HNL  436
Output file is  ./ship.conical.Pythia8-TGeant4.root
Parameter file is  ./ship.params.conical.Pythia8-TGeant4.root
Real time  81.75536012649536  s, CPU time  81.71 s
make suicid, until better solution found to ROOT/genfit interference
Terminated
[FairShip/latest] ~/ship $> ls
alidist  FairShip  geofile_full.conical.Pythia8-TGeant4.root  pythia8_conf.txt  ship.conical.Pythia8-TGeant4.root  shipdist  ship.params.conical.Pythia8-TGeant4.root  sw
[FairShip/latest] ~/ship $> python $FAIRSHIP/macro/ShipReco.py -f ship.conical.Pythia8-TGeant4.root -g geofile_full.conical.Pythia8-TGeant4.root
configured to process  999999  events from  ship.conical.Pythia8-TGeant4.root  starting with event  0  with option Yheight =  None  with vertexing True  and real pattern reco  No
Traceback (most recent call last):
  File "/root/ship/sw/slc7_x86-64/FairShip/master-1/macro/ShipReco.py", line 120, in <module>
    ShipGeo = upkl.load('ShipGeo')
  File "/root/ship/sw/slc7_x86-64/FairShip/master-1/python/rootpyPickler.py", line 286, in load
    obj = super(Unpickler, self).load()
TypeError: object() takes no parameters

is this file: geofile_full.conical.Pythia8-TGeant4.root

I will try to follow your suggestion and see how it goes.

Also we don't use the rootpy code, but something that we just copied from the main codebase a while ago...

https://github.com/ShipSoft/FairShip/blob/master/python/rootpyPickler.py

from rootpy.

I come back working on this but without success :(

I enable the logs in the code:

INFO: Making dummy module copy_reg
INFO: Making dummy class copy_reg._reconstructor
INFO: Making dummy module __builtin__
INFO: Making dummy class __builtin__.dict
Traceback (most recent call last):
  File "/root/ship-py3/sw/slc7_x86-64/FairShip/master-1/macro/ShipReco.py", line 121, in <module>
    ecalGeoFile = ShipGeo.ecal.File
AttributeError: 'Dummy' object has no attribute 'ecal'

Now, copy_reg is the PY2 version of copyreg and I have zero idea how it ends up there. Similarly builtin seems to be PY2 stuff.

I run the whole thing inside strace and then grep for python2 without any result, so I am quite baffled...

Any ideas? Did you run in something similar?

from rootpy.