Hi,

I know one should not give a file to dirlist, but it would be great if it could report an error instead of doing a segfault.
Here is a snipet that reproduces the problem

from XRootD import client
from XRootD.client.flags import DirListFlags

myclient = client.FileSystem('root://castorlhcb.cern.ch')

This works

myclient.dirlist("/castor/cern.ch/grid/lhcb/data/2011/RAW/FULL/LHCb/COLLISION11/88033/", DirListFlags.NONE)

This segfaults

myclient.dirlist("/castor/cern.ch/grid/lhcb/data/2011/RAW/FULL/LHCb/COLLISION11/88033/088033_0000000001.raw", DirListFlags.NONE)

This sefgfaults as well

myclient.dirlist("/castor/cern.ch/grid/lhcb/data/2011/RAW/FULL/LHCb/COLLISION11/88033/088033_0000000001.raw?svcClass=lhcbtape", DirListFlags.NONE)

Thanks,
Chris

Is it reasonable to consider adding glob functionality to the python bindings? I'd be happy to try and implement this myself, but I wanted to check how it would be received and if there were suggested approaches. I imagine this being a new function within the FileSystem module that replicates the standard glob package. Given a path containing wildcards (potentially more than one), a list of files that match this (as per fnmatch) should be returned.

Just some context on why I'd like to do this: I've previously added such functionality into the rootpy project (here), but I now wanted to get such functionality without requiring ROOT to be installed rootpy.

(This is a duplicate of this issue xrootd/xrootd#832, but this seems like the more appropriate repository and I haven't had a response there. I'm happy to close one of these issues if one is more appropriately located than the other)

It appears that it's incredibly difficult to use the asynchronous callback mechanism due to lock ordering issues:

• The callback thread pool holds various Xrootd locks when it invokes python, which needs the python GIL.
• Python-owned threads have the python GIL and needs various Xrootd locks when invoking operations in the Xrootd module.

Hence, it's quite straightforward to deadlock.

In order to actually use the asynchronous callbacks, you must be able to guarantee there is no call to the Xrootd module from any python threads as long as there are pending callbacks. It's a pretty high barrier.

Would it be possible to invoke callbacks from a separate thread pool which takes the python GIL first?

It would be really useful to have a class that behaves as much as possible like python's File. This will allow for interfacing with e.g. gzip.GzipFile, pickle.load, etc. This can be a simple wrapper of XRootD.client.File, which makes synchronous requests and throws exceptions when they are not successful.

There appears to be no OpenFlags.APPEND and it seems that it was removed. The documentation still lists APPEND as a valid flag. Can it be brought back?

Hi,

When you use the xrdcp command line, you can give a directory as destination, in which case the file will be named as the original one.
With the python api, using a directory as a destination results in an error stating :
"[ERROR] OS Error: Is a directory"
First of all it would be useful to write which of the argument (source/target) is a folder.
Also, for sake of consistency with the xrdcp command line, could you apply the same policy?

Many thanks
Chris

I found the following 'minor' bug with pyxoortd:

python
>>> from XRootD import client
>>> fs = client.FileSystem("root://...")
>>> print fs.dirlist("./invalid/path")
Segmentation fault (core dumped)

It happens with invalid paths or with valid remote paths that do not point to a directory. I would expect a failed XRootDStatus instead...

I am running with XRootD v3.3.4 at SLC6 with pyxrootd compiled locally with sources from Github.

I solved the issue on my program by introducing a stat call and checking if the path is valid and the entry is a directory.

Cheers,
George

Hi,
Shouldn't src/PyXRootDFile.cc:428 be 1024 * 1024 *2 ?

The tag v1.1.2 is misleading and looking at the tag point probably should have been called v0.1.2.

Hi,
STFC in the UK are moving to using XRoot to access their Castor system, replacing RFIO. I have a service which interfaces with Castor for deposition of experimental data into an archive. The service is driven by a multi-threaded TCPServer written in Python, so I was pleased to find your bindings.
The service receives large numbers of small files and concatenates them into large files before sending them to Castor for storage on tape. On retrieval from tape, we perform a stager_get and then we run multiple retrieval jobs copying the large files back to disk, and we serve files to the client from there. We do this to attempt to avoid long delays waiting for files to be staged from tape, and it works well for us.
I modified the backend to use these bindings for copying the large files to Castor (previously it was shelling out to rfcp), but I found that when I called FileSystem.copy() on a large file it would hang the whole process until the copy finished. I assume this is because the copy() is implemented in C and therefore is not subject to the timeslicing done by the Python interpreter (2.6.6)?
I'm aware of the CopyProcess() functionality you provide, but would that also pause until all the jobs are complete?
If I were to perform a FileSystem.copy() asynchronously with a callback, would that allow other threads of execution to carry on in the meantime?
I could use a loop and File.write() but since we're not concerned with writing or reading portions of the files it would seem preferable to just deal with a put/get style of operation.
I have sorted this for the time being by just shelling out to xrdcp for the copy to Castor, but I would really like to use these bindings. Is there any strategy I can adopt that would circumvent the locking I think I see?

Thanks in advance,

Roger Downing

Hello,

I'm trying to use pyxrootd on mac os x sierra.
I have installed xrootd 4.5.0 and I'm able to copy a file to a remote location:

 xrdcp foo.txt root://eospublic.cern.ch//eos/fcc/users/c/cbern/foo.txt

the source package xrootd-4.5.0.tar.gz comes with pyxrootd so at first I tried to use this version. It didn't work for the reasons explained below, and I then tried v0.3.0 of this repository.

Following the instructions given in the documentation, I wrote the following tests:

import unittest

from XRootD import client
from XRootD.client.flags import DirListFlags, OpenFlags, MkDirFlags, QueryCode

myclient = client.FileSystem('root://eospublic.cern.ch/:1094')

remote_dir = '/eos/fcc/users/c/cbern/Test/'
remote_file = remote_dir + '/foo'

class TestXROOTD(unittest.TestCase):
    
    def test_listdir(self):
        status, listing = myclient.dirlist(remote_dir, DirListFlags.STAT)
        self.assertTrue(status.ok)
        self.assertEqual(listing.parent, remote_dir)
        

    def test_1_copy_in(self):
        status = myclient.copy(remote_file, 'foo', force=True)
        self.assertTrue(status.ok)
        
        
if __name__ == '__main__':
    unittest.main()

The first test works and the second fails because, in filesystem.py, this call fails:

result = self.__fs.copy(source=source, target=target, force=force)[0]

First of all, self.__fs.copy returns a dictionary, and this dictionary has strings as keys. It does not have integer 0 as a key which leads to a KeyError in my case. Looking further, here is the dictionary:

{'status': 1, 'code': 9, 'ok': False, 'errno': 0, 'error': True, 'message': '[ERROR] Invalid arguments', 'fatal': False, 'shellcode': 50}

Why is the code trying to access element 0?
What am I doing wrong? the message '[ERROR] Invalid arguments' is not enough for me to understand what's going on, and the debugger does not help. As far as I can see, I just copied the lines in the documentation, http://xrootd.org/doc/python/xrootd-python-0.1.0/examples/filesystem.html

Finally, I tried to run py.test, but it gets stuck here:

[xrootd-python]$ py.test 
============================== test session starts ==============================
platform darwin -- Python 2.7.11, pytest-3.0.4, py-1.4.31, pluggy-0.4.0
rootdir: /Users/cbernet/Software/xrootd-python, inifile: 
collected 48 items 

tests/test_copy.py  

Could you please help?

Thanks,
Colin

Hi,

I wanted to use the python API against the castor storage at CERN. With the nsls command, you can know whether a file has been replicated with the "m" flag. Is there a way to get the same information from pyxrootd? The Offline flag from the StatInfo tells you whether the file is not on disk, but is irrelevant if it is on disk and tape. And I am not quite sure, but I think that PoscPending means that we are waiting for the replication to happen. So how can I know whether the file is on disk AND on tape?

Thanks a lot!

moved from https://savannah.cern.ch/bugs/?101932

Dear Sir,
I got the error when I was trying to download files by CopyProcess,
do you know how to fix it?
My OS: Scientific Linux CERN SLC release 6.4 (Carbon)
My Python version:2.6.6
My xrootd version:3.3.2-1

Traceback (most recent call last): 
File "/usr/lib64/python2.6/site-packages/XRootD/client/copyprocess.py", line 30, in begin 
self.handler.begin(id, total, URL(source), URL(target)) 
AttributeError: 'NoneType' object has no attribute 'begin' 
Traceback (most recent call last): 
File "/usr/lib64/python2.6/site-packages/XRootD/client/copyprocess.py", line 36, in update 
self.handler.update(processed, total) 
AttributeError: 'NoneType' object has no attribute 'update' 
Traceback (most recent call last): 
File "/usr/lib64/python2.6/site-packages/XRootD/client/copyprocess.py", line 33, in end 
self.handler.end(XRootDStatus(status)) 
AttributeError: 'NoneType' object has no attribute 'end' 

Thanks for your help.
Best Regards,
W.J. Chang

Hi,

the MkDirFlags.MAKEPATH does not seem effective to create the full tree structure:

from XRootD import client
from XRootD.client.flags import DirListFlags, MkDirFlags

server = "root://eoslhcb.cern.ch"
baseDir = "/eos/lhcb/xroot"

myclient = client.FileSystem(server)
status, listing = myclient.dirlist(baseDir, DirListFlags.STAT)

print "Content of %s"%baseDir
for entry in listing:
print "\t{0} {1:>10} {2}".format(entry.statinfo.modtimestr, entry.statinfo.size, entry.name)

newSubDirs="/".join( [ baseDir, "newDir", "newSubDir" ])
print "Attempting to create %s"%newSubDirs

status = myclient.mkdir(newSubDirs, MkDirFlags.MAKEPATH)
print status

This results in :
$ python createSubDir.py
Content of /eos/lhcb/xroot
2013-11-19 14:44:41 2 lhcb
2013-11-20 15:50:15 0 testDir
Attempting to create /eos/lhcb/xroot/newDir/newSubDir
(<status: 1, code: 400, ok: False, errno: 3011, error: True, message: '[ERROR] Server responded with an error: [3011] Unable to mkdir /eos/lhcb/xroot/newDir/newSubDir; No such file or directory\n', fatal: False, shellcode: 54>, None)

However, I can create the sub directories recursively myself

Cheers,
Chris

Hi,

With a fresh installation, I am unable to have the copy example work.

$ cat examples/copy.py
"""

Copy a file

See :mod:XRootD.client.CopyProcess if you need multiple/more configurable
copy jobs.
"""
from XRootD import client

myclient = client.FileSystem('root://localhost')
status = myclient.copy(source="/tmp/spam", target="/tmp/eggs", force=True)
assert status.ok

$ python examples/copy.py
Traceback (most recent call last):
File "examples/copy.py", line 11, in
status = myclient.copy(source="/tmp/spam", target="/tmp/eggs", force=True)
File "/home/chaen/dirac/Linux_x86_64_glibc-2.12/lib/python2.6/site-packages/XRootD/client/filesystem.py", line 61, in copy
force=force)), []
SystemError: Python/getargs.c:1389: bad argument to internal function

This remains true whatever xrootd server I use, whatever source or destination file, and whatever the force flag.

Here are some information on the running system :
$ uname -a
Linux slc64 2.6.32-358.23.2.el6.x86_64 #1 SMP Thu Oct 17 10:27:22 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -qa | grep xrootd
xrootd-client-devel-3.3.3-1.el6.x86_64
xrootd-python-0.1.2-1.slc6.x86_64
xrootd-libs-3.3.3-1.el6.x86_64
xrootd-devel-3.3.3-1.el6.x86_64
xrootd-client-libs-3.3.3-1.el6.x86_64
xrootd-client-3.3.3-1.el6.x86_64

Thanks,

Chris

Files are accessible from xrdcp but times out with the python open that is from xrootd-python package.
As a result, the SAM tests are failing from time to time.
Why is the python open so unpredictably difficult to open files?