Hi,

Today a college of mine encountered an interesting problem. He tried to convert a TProfile to a rootpy hist to a python list. The error shows itself in an infinite amount of the following lines:

"Error in TArrayD::operator[]: index 102 out of bounds (size: 102, this: 0x2bb9dd8)"

with increasing index.

Example code (taken from ROOT TProfile):

from rootpy.utils import asrootpy
from ROOT import TProfile
from ROOT import gRandom
import ROOT

hprof = TProfile("hprof","Profile of pz versus px",100,-4,4,0,20)

px, py, pz = 0,0,0

px = ROOT.Double(0)
py = ROOT.Double(0)
pz = ROOT.Double(0)
for i in range (25000):
gRandom.Rannor(px,py)
pz = px_px + py_py
hprof.Fill(px, pz, 1)

h_rootpy = asrootpy(hprof)
print list(h_rootpy)

This code, if output is redirected into a file, produces error messages worth a hundred MB every few seconds. In general the prompt becomes non-responsive.

This is a minor peeve. The name of the game is to make import rootpy no slower than import ROOT.

When you access a ROOT.* symbol for the first time (other than a few special ones), this can cause a lot of stuff to get searched for loaded, including dynamic libraries. This can be slow, especially on networked filesystems.

It's super annoying when "myrootapp --help" takes tens of seconds to show, and this issue aims to avoid that.

It's possible to wrap finalSetup so that defaults such as ROOT.TH1.SetDefaultSumw2() can be set without triggering it. We could provide a function to allow users to do attach additional setup functions just before finalSetup() time.

#47 (comment) is also motivated by this.

I would propose to collect all external code (i.e. code that we don't maintain ourselves) in one place: rootpy/rootpy/extern to simplify maintenance and make it easier for newcomers to understand the codebase.

If we have external C or Cython code we should make a separate folder cextern like e.g. the astropy project, but at the moment we don't have C or Cython code we don't maintain, right?

@ndawe If you agree, I can change move the files and change all imports accordingly.

segfault in test_dict_load in rootpy/tests/test_dicts.py with ROOT 5.30.05:

https://gist.github.com/4164952

I cannot reproduce this with ROOT 5.32 or 5.34.

The reason we didn't see this before in 5.30 seems to be due to an issue with opening a TFile before loading the dictionaries (which of course is always the case when loading a Tree from a File). types.py used to load all the dicts at once and this happened to be before any file was opened in any of my code.

gdb --args python test_segfault.py

from rootpy.io import open
f = open('test_dicts.root')
f.data

Hello,

I am trying to convert a root ttree in a csv table and I get a segmentation violation. Here is the code I use (with root version 5.32.02 and rootpy on branch master, revision number 2d21206) :

import rootpy.io
from rootpy.tree import Tree

f = rootpy.io.open('ttree.root')
tree = f.get('ParTree_Postselect', ignore_unsupported=True)
tree.csv(stream=open('test.csv', 'w'))

I get the following segmentation violation : http://dl.dropbox.com/u/79664529/csv_crash.txt
Here is the root file I used : http://dl.dropbox.com/u/79664529/ttree.root

rootpy.io.File.get returns a ROOT.TTree, I think it should return a rootpy.tree.Tree.

I think this was not the case before, because rootpy/bugs/csv/test.py used to work, but now I get (with v5-34-00-patches on Mac):

$ python test.py 
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    tree.csv()
AttributeError: 'TTree' object has no attribute 'csv'

If I add this line at the top

from rootpy.tree import Tree

I get another error:

$ python test.py 
LatEvent,CoreYGround,EventNr,dEoverE,AzSystem,AltSystem,ShowerChisquared,MSCWO,BunchNr,NtelInShower,MSCL,AzEvent,LonSystem,DecSystem,AltEvent,PsiSq,TangentialXEvent,CoreXGround,EventTimeSec,DecEvent,LatSystem,EventTimeNanoSec,CorrEnergy,RaEvent,CoreXTilted,Hmax,MSCW,GlobalEventNr,CameraYEvent,CameraXEvent,TangentialYEvent,CoreDist,TriggerPattern,Energy,ShowerNdf,RunNr,LonEvent,MSCLO,CoreYTilted,RaSystem,NtelWData
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    tree.csv()
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/rootpy-0.7.0_a0-py2.7-macosx-10.8-x86_64.egg/rootpy/tree/tree.py", line 334, in csv
    for i in len(range(self)):
TypeError: range() integer end argument expected, got Tree.

It seems that what is returned by rootpy.io.File.get depends on the local imports of the user!?
Would it be possible to consistently return rootpy.tree.Tree objects and mention that in the documentation?

@ndawe Do you have an idea what the problem is or should I investigate?
(I actually need to convert TTrees to FITS (via numpy arrays) for work right now.)

Instead of using print, use an appropriate log function. In order of increasing severity: log.debug, log.note, log.warning, log.error, log.critical. If you're inside an exception handler (i.e, sys.exc_info() returns things which are non-null, you can also use log.exception(msg).

If the module you're in doesn't have a log variable, top level modules (i.e, __init__.py) can obtain it with:

from .. import log; log = log[__name__]

or other modules can get one with:

from . import log; log = log[__name__]

The ; log = log[__name__] can be omitted it it doesn't add any useful information to the logger origin at the expense of space.

This is another feature request, similar to the one mentioned in issue #24.

Would it be possible to implement a function to fill histograms from numpy arrays, something like this fill_hist function, but in a memory and speed efficient wat?

In [1]: def fill_hist(hist, array):
   ...:         [hist.Fill(_) for _ in array]                                                                                                                 
   ...:                                                                                                                                                       
In [2]: import numpy as np                                                                                                                                    
In [3]: import ROOT                                                                                                                                           
In [4]: hist = ROOT.TH1F('hist', 'hist', 100, 0, 1)                                                                                                           
In [5]: array = np.random.rand(1e6)                                                                                                                           
In [6]: %timeit fill_hist(hist, array)                                                                                                                        
1 loops, best of 3: 4.04 s per loop

Check this out:

https://github.com/piti118/root_numpy

The root2array module contains functions to convert ROOT TTrees to numpy arrays.

Would it be possible to add similar methods to convert ROOT THs to numpy arrays?

I wrote my own little function which works for my special case of a 2D float histogram.

def hist_to_ndarray(hist, include_weight=False):
    """Convert ROOT hist to numpy array"""
    # @note: Numpy array index order is (y, x), whereas ROOT TH2 has (x, y)
    nx, ny = h.GetNbinsX(), h.GetNbinsY()
    # @todo This doesn't work properly:
    # dtype = type(h.GetBinContent(0))
    dtype = 'float32'
    array = np.empty((ny, nx), dtype=dtype)
    for ix in range(nx):
        for iy in range(ny):
            array[iy, ix] = h.GetBinContent(ix, iy)
    return array

Having something that works for all dimensions / data types in rootpy and preferably is fast would be nice.

Implementing this. Just making a note of it here. Currently there is no way to draw ROOT histograms with a hollow fill in matplotlib. matplotlib needs to see facecolor=None. If SetFillColor(None) has been called then GetFillColor('mpl') should return None but GetFillColor('root') should return white.

edit: fill=False works...

All scripts should have a short usage message what they do. @ndawe Can you do that? It's not obvious to me with a few what their purpose is.

And they should use from rootpy.extern import argparse instead of import optparse or import argparse. I'd be happy to do that part.

It would probably be straightforward to implement a LineWidth class in plotting.styles which would be used for hists to keep changes in sync between root and matplotlib, but I have not done this yet. Leaving a note to pick this up later.

Looks like we might be able to use this: http://dispy.sourceforge.net/

rootpy is currently failing to import due to dictionary generation failure. I have a feeling this might be due to my having imported it in incompatible environments. Backtrace below. One way I handled this failure in the past was to retry the build once, deleting the bad libraries, but unfortunately this seems to be crashing when considering a different library to the one which is actually bad. So obviously my solution is to delete all of my dicts and try again, but it is better if this can be avoided somehow.

150e42c9ab24483586058034be7b9d9d.so definitely does exist.

Message: (file "/user1/pwaller/.rootpy/dicts/335872/4bbfff3dcd114711929316862a96aafe.so", line -1) dlopen error: 150e42c9ab24483586058034be7b9d9d.so: cannot open shared object file: No such file or directory
Library loading failure. Current dynamic path:
['/user1/pwaller/.rootpy/dicts/335872', '/batchsoft/root/root532-x86_64-slc5-43/lib', '/user1/pwaller/local/lib', '', '.', '/batchsoft/root/root532-x86_64-slc5-43/lib', '', '/batchsoft/root/root532-x86_64-slc5-43/cint/cint/stl']
Traceback (most recent call last):
  File "./plot_smbkg_vs_data.py", line 4, in <module>
    from rootpy.io import File
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/io/__init__.py", line 1, in <module>
    from .file import *
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/io/file.py", line 7, in <module>
    from ..utils import asrootpy
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/utils.py", line 3, in <module>
    from .plotting.core import Plottable
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/plotting/__init__.py", line 1, in <module>
    from .hist import Hist, Hist2D, Hist3D, HistStack
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/plotting/hist.py", line 4, in <module>
    from ..objectproxy import ObjectProxy
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/objectproxy.py", line 1, in <module>
    import types
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/types.py", line 432, in <module>
    generate('vector<vector<int> >', '<vector>')
  File "/user1/pwaller/env-slc6/src/rootpy/rootpy/classfactory.py", line 49, in generate
    loaded = ROOT.gSystem.Load('%s.so' % __LOOKUP_TABLE[unique_name])
RuntimeError: (file "/user1/pwaller/.rootpy/dicts/335872/4bbfff3dcd114711929316862a96aafe.so", line -1) Failed to load Dynamic link library /user1/pwaller/.rootpy/dicts/335872/4bbfff3dcd114711929316862a96aafe.so

At the moment everyone uses python 2, but it would be nice to support python 3 at some point.

matplotlib 1.2 will be released soon, and then the full scientific python stack will support python 3 and as I understand it rootpy already does, so it would be nice to have python 3 support in rootpy in the mid-term future.

I'll see if I can get http://travis-ci.org to run rootpy tests on each commit.
They basically let you run whatever you want, you get root rights in your VM image, it's awesome.

The only issue is that we probably can't build root (or e.g. pytables or matplotlib) without running into the build timeout. I'll try if installing root (and the others if possible) via the package manager works.

Eventually it would be nice to get a small test matrix for different ROOT / python / numpy versions, but let's try to get a ROOT install and rootpy build working first.

To test it we can simply run this as a test:

python -c 'import rootpy.tables'

ROOT 5.34, SLC 6.3, rootpy master @ f60d92b184.

This means I can't seem to do all that much, since lots of things import rootpy.types.

In [1]: import rootpy.types
(Bool_t)1
generating dictionary for vector<vector<bool> >...
/user1/pwaller/.rootpy/dicts/64/336384/842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.o: In function `G__setup_memfuncvectorlEvectorlEboolcOallocatorlEboolgRsPgRcOallocatorlEvectorlEboolcOallocatorlEboolgRsPgRsPgRsPgR()':
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x20): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x27): undefined reference to `G__tag_memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x33): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x44): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xad): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xb9): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xc3): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x133): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x142): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x1ae): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x1b6): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x222): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x22e): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x241): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x2ac): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x2b8): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x2c4): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x32f): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x33b): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x3a7): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x3b3): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x41f): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x492): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x505): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x511): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x57d): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x5e9): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x5f5): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x600): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x669): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x675): undefined reference to `G__defined_typename'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x680): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x6f0): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x6f8): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x764): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x76c): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x7df): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x7e7): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x853): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x85b): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x8ce): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x8d6): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x942): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x9ae): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0x9b6): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xa22): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xa2a): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xa96): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xb09): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xb7c): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xb84): undefined reference to `G__get_linked_tagnum'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xbf7): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xc6a): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xcdd): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xd49): undefined reference to `G__memfunc_setup'
842da8bfa78a43818b0945957ea4ca36_ACLiC_dict.cxx:(.text+0xdbc): undefined reference to `G__memfunc_setup'

Hello,

I tried importing rootpy.io (python -c 'import rootpy.io') while using root version 5.28, rootpy on branch master revision 2d21206 and got the following errors :
http://dl.dropbox.com/u/79664529/import_crash.txt

scikit-hep/root_numpy#3

See #71

rootpy has been released once as version 0.6:

http://pypi.python.org/pypi/rootpy/0.6

We should start thinking about when we would like to have a 0.7 release out and what deliverables could be ready. Also this might be a good time to consider various versioning schemes. My original plan was to have 0.7, 0.8, 0.9 and then 1.0 would be the first "production-ready" release. Anyone oppose this plan? I'm open to other suggestions. Ubuntu's versioning is convenient (as @pwaller has pointed out) since the version is the date and therefore you know how old a version is without looking it up.

Once we have a concrete plan we can outline the release schedule and deliverables on our github wiki.

rootpy.io.tests.test_file.test_file fails for me on Mac and Linux:
https://gist.github.com/3685144

We already have included root_numpy in rootpy, see issue #13 . @piti118 has made another piece of ROOT / python goodness, https://github.com/piti118/RTMinuit , a python-friendly ROOT.TMinuit wrapper.

@piti118 Are we allowed to include / modify it in rootpy?

There's also https://github.com/piti118/dist_fit , but I don't know if it is considered too specialized for rootpy. @ndawe and others: what do you think?

Hello,

I deleted an old release area (and it's associated libs). I had previously used roosh etc to browse the CMSSW files, and classfactory cached the libraries. Now if I open any old file in roosh I get:

Message: (file "/afs/hep.wisc.edu/home/efriis/.rootpy/dicts/334598/4c6661edd304453ea4e66a622924e0b9.so", line -1) dlopen error: /afs/hep.wisc.edu/user/efriis/cmssw/Analysis/lib/slc5_amd64_gcc434/libDataFormatsPatCandidates.so: cannot open shared object file: No such file or directory

which indeed doesn't exist. The whole traceback is below.

I fixed the crash by nuking my ~/.rootpy directory. I'm not sure what the best behavior is.

Cheers
Evan

Message: (file "/afs/hep.wisc.edu/home/efriis/.rootpy/dicts/334598/4c6661edd304453ea4e66a622924e0b9.so", line -1) dlopen error: /afs/hep.wisc.edu/user/efriis/cmssw/Analysis/lib/slc5
_amd64_gcc434/libDataFormatsPatCandidates.so: cannot open shared object file: No such file or directory
Traceback (most recent call last): File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/bin/roosh", line 5, in
pkg_resources.run_script('rootpy==0.6', 'roosh')
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/pkg_resources.py", line 499, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/pkg_resources.py", line 1235, in run_script
execfile(script_filename, namespace, namespace)
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/EGG-INFO/scripts/roosh", line 1
8, in
from rootpy.io import open as ropen, utils
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/io/init.py", line 1,
in
from .file import *
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/io/file.py", line 7, in

from ..utils import asrootpy
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/utils.py", line 3, in
from .plotting.core import Plottable
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/plotting/init.py", l
ine 1, in
from .hist import Hist, Hist2D, Hist3D, HistStack
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/plotting/hist.py", line
4, in
from ..objectproxy import ObjectProxy
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/objectproxy.py", line 1,
in
import types
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/types.py", line 430, in

generate('vector<vector >', '')
File "/afs/hep.wisc.edu/user/efriis/cmssw/Analysis2/src/FinalStateAnalysis/recipe/external/vpython/lib/python2.6/site-packages/rootpy-0.6-py2.6.egg/rootpy/classfactory.py", line 4
5, in generate
if ROOT.gSystem.Load("%s.so"% __lookup_table[unique_name]) in (0, 1):
RuntimeError: (file "/afs/hep.wisc.edu/home/efriis/.rootpy/dicts/334598/4c6661edd304453ea4e66a622924e0b9.so", line -1) Failed to load Dynamic link library /afs/hep.wisc.edu/home/efr
iis/.rootpy/dicts/334598/4c6661edd304453ea4e66a622924e0b9.so

@ndawe commented

This common module contains fragments from older frameworks I had. It remains to be thoroughly cleaned up. Some of these functions should go into rootpy.plotting and others could probably be removed altogether.

We should find a good solution for ROOT-based extension module handling in rootpy, i.e. decide when to generate shared libraries and where to put them.

Current situation

Currently python setup.py install generates these files and puts them in site-packages:

rootpy/interactive/_pydispatcher_processed_event.so
rootpy/root2array/_libnumpyhist.so
rootpy/root2array/root_numpy/_librootnumpy.so

Using another ROOT version at import time than install time is not possible, it segfaults.

import rootpy.types generates dictionaries for commonly used types not included in ROOT (e.g. vector<double> ...). This is done per user and per ROOT version and for 32-bit and 64-bit separately and the files are stored in ~/.rootpy/dicts:

/Users/deil/.rootpy/dicts//64/336386/01ce40b7f1f644a89b7fedf96fe0c726.C
/Users/deil/.rootpy/dicts//64/336386/01ce40b7f1f644a89b7fedf96fe0c726.so
/Users/deil/.rootpy/dicts//64/336386/01ce40b7f1f644a89b7fedf96fe0c726_C.d

In this example 64 means 64 bit and 336386 is the ROOT version:

In [3]: ROOT.gROOT.GetVersionCode() 
Out[3]: 336387
In [4]: ROOT.gROOT.GetVersion()
Out[4]: '5.34/03'

Have a look here to see how rootpy.types works in detail:
https://github.com/rootpy/rootpy/blob/master/rootpy/types.py
https://github.com/rootpy/rootpy/blob/master/rootpy/classfactory.py

Possible improvements

For rootpy.interactive and rootpy.root2array we should either add a ROOT version check and raise an ImportError instead of crashing if there is a ROOT version mismatch, something like:

You built and installed rootpy with ROOT version 5.32, but at the moment are using version 5.34.
This can't work, please use matching ROOT versions.

Or we put these extension modules in ~/.rootpy/extensions/ and generate them on import if they don't exist already, similar to how rootpy.types works now. The advantage would be that one rootpy installation can then be used with many different ROOT versions. I as a rootpy developer would find this useful and also I maintain a python site-packages for out group on our cluster at work and I can't control which ROOT version other users use with that. The disadvantage is that every user duplicates the extension modules, which is more error-prone because things can go wrong when building the extension.

We could also give make it flexible when (install or import time) and where (site-packages or home or user-chosen folder) to install the rootpy dicts and extensions for expert users and cluster admins or when rootpy is installed via a package manager, with simple default behaviour for new users.

I'm not sure what the best solution is, the only thing that seems clear to me is that we should add the import check to avoid crashes and document clearly in the installation instructions how it works.

@ndawe What do you think?

Add an option to Tree and TreeChain to ignore branches where no dictionary is available.

With rootpy e6afde1 I see some problems on Mac OS X 10.8:
https://gist.github.com/4121872

Gamma-ray astronomers [1, 2] and I think also [3, 4, 5] and others internally store event lists in ROOT TTrees and images and spectra and instrument responses in ROOT histograms (1D, 2D, 3D).

Often internally ROOT is used, but then because it not used by other astronomers, data is exported to FITS format [6].
A FITS file is a collection of header-data units (HDUs), each one being either an n-dimensional array or a table, so it is possible to to convert simple TTrees to FITS tables and ROOT histograms to FITS arrays.

There once was the astroroot project [7], but it has been dead for many years. I think rootpy would be the right place for easy and efficient ROOT <-> FITS conversion. Basically ROOT TTree <-> numpy array is in place already (although I didn't check if it is efficient yet) and ROOT histogram <-> numpy array could probably easily be added (make a separate ticket?). So what I plan to do is add wrappers around these ROOT <-> numpy conversion functions that do FITS I/O via the AstroPy package [8], which would become an optional dependency of rootpy, just as PyTables is an optional dependency for HDF5 I/O at the moment.

[1] http://www.mpi-hd.mpg.de/hfm/HESS/
[2] http://www.cta-observatory.org/
[3] http://fermi.gsfc.nasa.gov/
[4] http://veritas.sao.arizona.edu/
[5] http://magic.mppmu.mpg.de/
[6] http://en.wikipedia.org/wiki/FITS
[7] http://www.isdc.unige.ch/astroroot/index
[8] http://www.astropy.org/

It would be very nice if one could run rootpy unit test to find problems with the ROOT / rootpy installation directly and not later when using certain parts.
(e.g. via python setup.py test or python -c "import rootpy; rootpy.test() or nosetests rootpy or ...)

I would think that basic things (e.g. simply running all examples) would be enough to expose > 90% of all installation problems.

Also I think the first time rootpy is imported it generates 5 MB of ROOT dicts for stl containers in ~/.rootpy?
Wouldn't it be better to do this when installing rootpy and include a check that the rootpy software version and files in ~/.rootpy are compatible. I think this might have been the origin of my problem in issue #18, but I'm not sure.

I get a few -Wunused-variable warnings when compiling librootnumpy with the most recent clang on Mac OS X:
https://gist.github.com/3687558#file_clang_librootnumpy_compile_warnings.log

With gcc 4.4. on Linux I get a few complaints about mixing signed and unsigned ints:
https://gist.github.com/3687558#file_gcc_linux_compile_warnings

Not a big deal at all, but I like a clean build.

This one does not appear to be fatal to the continued execution, just annoying.

Error: class,struct,union or type UInt not defined  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/memory:37:
Error: class,struct,union or type UInt not defined  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/memory:38:
Error: class,struct,union or type UInt not defined  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/memory:43:
Error: class,struct,union or type UInt not defined  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/memory:44:
Error: class,struct,union or type UInt not defined  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/memory:45:
Error: class,struct,union or type UInt not defined  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/vector:51:
Error: Function UInt() is not defined in current scope  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/vector:444:
Error: Function UInt() is not defined in current scope  /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/vector:444:
Error: Syntax error /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/vector:525:
Error: void type variable can not be declared /batchsoft/root/root53403-x86_64-slc6-46/cint/cint/lib/prec_stl/vector:525:
Warning: Error occurred during reading source files
Note: Link requested for undefined class vector<UInt,allocator<UInt> >::value_type (ignore this message) :0:
Warning: Error occurred during dictionary source generation
!!!Removing /user1/pwaller/thesis/studies/photonreco/AutoDict_vector_UInt__cxx_ACLiC_dict.cxx /user1/pwaller/thesis/studies/photonreco/AutoDict_vector_UInt__cxx_ACLiC_dict.h !!!
Error: /batchsoft/root/root53403-x86_64-slc6-46/bin/rootcint: error loading headers...
Error in <ACLiC>: Dictionary generation failed!

Building the docs fails with rootpy b339488 on Mac OS X Lion.

I think in docs/conf.py you need to change __VERSION__ to __version__, at least that's what I did to make sphinx run.

Then there are plenty of warnings and finally a segfault:
http://dl.dropbox.com/u/4923986/bug_reports/rootpy_docs.log

This works though: import rootpy and python examples/tree/tree.py

rootpy currently has submodules for interfacing with matplotlib, numpy, and pytables. The numpy interface is basically a copy of root_numpy: https://github.com/piti118/root_numpy.

Instead, what if rootpy only held the core essential stuff but we had the optional packages:

root_matplotlib
root_tables
root_numpy

and root_X for interfacing with some other technology X, etc.

root_numpy could continue to stand on its own, owned by @piti118 and forked under rootpy or moved to the rootpy collaboration and forked by @piti118. I have no preference either way. That would be up to @piti118.

root_matplotlib would depend on rootpy since it requires rootpy's Hist API.
root_tables would depend on root_numpy and rootpy.
root_numpy stands on its own.

Some of the numpy-related code in rootpy could be moved to root_numpy (filling hists with numpy arrays, etc).

Then I think the various capabilities of rootpy are made more explicit and the user can choose to install whatever "plugins" they choose.

@piti118, @cdeil, @pwaller, @ekfriis, @jklukas what do you think?

You use check_output which is a new function to python 2.7, whilst your setup() call seems to suggest you're targeting >=2.6.

When building the docs on my Mac, I get an error:

Error in TDatabasePDG::ReadPDGTable: Could not open PDG particle file ${ROOTSYS}/etc/pdg_table.txt

I have root via the "fixed location" method in /usr/local/, so there is no need for a ROOTSYS variable, and indeed I believe that the pdg_table.txt file is not even installed on my system by this method. I would propose that we store the file within rootpy itself, but I don't know offhand how to access such a file other than through an import statement.

@ndawe @piti118 @ekfriis @jklukas @timelike @pwaller @jhegeman @kreczko @mverzett

You have contributed to rootpy via pull requests or issue reports, so we wanted to let you know that we are doing a coding sprint on rootpy:
https://groups.google.com/d/topic/rootpy-dev/wxrS_L-gQRc/discussion

(Sorry, I didn't know how else to contact you, since not all github users have a public email address. Please sign up for the rootpy-users or rootpy-dev google group.)

At the top level of rootpy, types, io, and math are hidden. These modules in rootpy should probably be renamed...

Alternative solution here:

http://stackoverflow.com/questions/6031584/python-importing-from-builtin-library-when-module-with-same-name-exists

Here is a new logo design idea. Criticism welcome!

@ndawe commented

PyX is http://pyx.sourceforge.net/, used to draw the decision tree graphs, but I think we should consider moving to http://www.graphviz.org/ which can also handle drawing the trees nicely instead of doing this ourselves in Node.draw.

I haven't tried either PyX or graphviz and it's not clear to me what rootpy/tree/categories.py does.
I guess categories.py is a somewhat special use case that most rootpy users don't need?
Maybe you could add some documentation an example anyways?

For reference: there are several graphviz bindings for python, PyGraphviz and PyDot seem to be the most popular: PyGraphviz, PyDot

I see that the theme directory links to a different repository. Cool. This is not transferred when you pull from this repository, though. Can you add something to the README in docs that explains how to pull the theme?

Hi all,

When trying to install rootpy against a 5.27 version of ROOT (I know, FML), I get a compile error:

rootpy/interactive/src/_pydispatcher.cxx:30:28: error: TFileMergeInfo.h: No such file or directory

error: command 'gcc' failed with exit status 1

it seems that the generated(?) file

https://github.com/rootpy/rootpy/blob/master/rootpy/interactive/src/_pydispatcher.cxx#L30

includes a non-existent header file. It compiles okay if I comment that line out, so perhaps it isn't even needed?

Evan

.. is this expected? I was expecting a rootpy.tree.Tree or something. What's the best way to get a rootpy...Tree?

(my opening the file is separate from the .my_ttree, just put that in the subject for terseness)

For users the two main pages of interest are https://github.com/rootpy/rootpy and http://ndawe.github.com/rootpy/ .

Would it be possible to move http://ndawe.github.com/rootpy/ to http://rootpy.github.com/rootpy/ and have the "Fork me on github" link point to https://github.com/rootpy/rootpy instead of https://github.com/ndawe/rootpy as it does now.

Also I think it would be nice to have very prominent front and center links from https://github.com/rootpy/rootpy to http://rootpy.github.com/rootpy/ and vice versa. At the moment I always use google to switch between the two.
See e.g. https://github.com/statsmodels/statsmodels/ where the link to the docs is in the one-line description on github on top and then there's a link back to github from the docs in the sidebar.

I think it would be useful to point users to a google group or stackoverflow or the ROOT forums or whatever you think would be best for simple usage questions like "How can I do X with rootpy?" Most projects have this, here's an example:
http://statsmodels.sourceforge.net/documentation.html

I have some time this week, should I give it a try and set up a rootpy webpage?
I haven't looked in detail how it works, but I believe we could simply use the structure from ipython or statsmodels or ..., they have a separate github repo for their webpage which is then auto-build on commit.

I'm playing around with the docs at the moment, and I'm seeing that the classes, such as rootpy.plotting.Hist are appearing with lower-case names. This was confusing to me, and I expect it would be to users, so we should try to figure out if there's a less-greedy way of performing the conversion.

I've been playing around with rootpy lately and noticed that the automatic setting of the y-axis is working except for histograms with large errors.
The calculation of the upper limit seems to exclude the errors on the values and therefore leads to results like this one:
https://raw.github.com/kreczko/DailyPythonScripts/master/examples/plots/Hist_with_dataMCRatio.png

EDIT:
The code for the creation of this plot can be found here:
https://github.com/kreczko/DailyPythonScripts/blob/master/examples/hist_with_dataMCRatio_example.py