diracgrid / diracos Goto Github PK
View Code? Open in Web Editor NEWProject for creating bundle of DIRAC dependencies.
License: GNU General Public License v3.0
Project for creating bundle of DIRAC dependencies.
License: GNU General Public License v3.0
Sorry to come late on this, just a comment. pytest by default looks for files named "test_*.py", and inside it for test items in functions named test_*. Here you go against the practice.
I think this can be merged. @andresailer would you take care of creating a readthedocs for DIRACOS, and link it from the main DIRAC docs? like done for VMDIRAC.
Originally posted by @fstagni in #129 (comment)
Hi all,
Are Rucio Python libs compatible with a current version of DIRACOS ? I remember @chrisburr mentioned this issue during our BiLD meeting, not sure what DIRACOS version he meant.
In order to know which version of diracos and which version of certain package (like Tornado) is used, would be nice to have a file with the content of diracos.
Found duplicates for libpango-1.0.so.0: ['/tmp/diracos/usr/lib64/libpango-1.0.so.0.2800.1', '/tmp/diracos/usr/lib64/firefox/bundled/lib64/libpango-1.0.so.0.4000.4']
Traceback (most recent call last):
File "/tmp/set_RPATH.py", line 176, in <module>
main(sys.argv[1])
File "/tmp/set_RPATH.py", line 171, in main
check_duplicates(so_names)
File "/tmp/set_RPATH.py", line 98, in check_duplicates
raise NotImplementedError(v, hashes)
NotImplementedError: (['/tmp/diracos/usr/lib64/libpango-1.0.so.0.2800.1', '/tmp/diracos/usr/lib64/firefox/bundled/lib64/libpango-1.0.so.0.4000.4'], {'536e9298b19719476b4ed03d150734da16fc57e17c067f6624b1d751b57ee904', '7b7caade480e9b2379686a2cceee98d8d2a5959d1b388ddc8f5ec2e7a492b017'})
Traceback (most recent call last):
File "/tmp/bundlelib.py", line 251, in <module>
main()
File "/tmp/bundlelib.py", line 245, in main
manualDependencies=manualDependencies)
File "/tmp/bundlelib.py", line 224, in _doBundleDIRACOS
subprocess.check_call(bundleCmd)
File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/bundleDiracOS.sh']' returned non-zero exit status 1
Finish: shell
Traceback (most recent call last):
File "/usr/bin/dos-bundle", line 9, in <module>
load_entry_point('diracos==1.0.0', 'console_scripts', 'dos-bundle')()
File "/usr/lib/python2.6/site-packages/diracos/scripts/bundleDiracOS.py", line 17, in main
diracoslib.bundleDIRACOS(cfg)
File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 605, in bundleDIRACOS
subprocess.check_call(bundleCmd)
File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mock', '-r', u'/builds/CLICdp/iLCDirac/DIRACOS/mockConfigs/mock-install-diracos.cfg', '--shell', 'python /tmp/bundlelib.py']' returned non-zero exit status 1
I already tried various compression level, but we are at the maximum for that.
By default, we also get ride of all the docs and debug info in the cfg
"excludePatterns": [
".*-doc-.*",
".*-debuginfo-.*"
],
So we will have to trim things down one way or another.
I do not want any dirty hack in any case.
As agreed during today's BILD meeting, as of v7r0, runit will not be distributed with DIRAC anymore and should thus be removed from DIRACOS
I get this error when trying to build the OS
docker run --privileged --cap-add=SYS_ADMIN --oom-kill-disable=true -it cern/slc6-base bash
yum install -y mock rpm-build fedora-packager createrepo python-pip
export DIRACOS_REPO=/cvmfs/clicdp.cern.ch/DIRAC/OS
mkdir -p $DIRACOS_REPO/i386 $DIRACOS_REPO/i686 $DIRACOS_REPO/src $DIRACOS_REPO/noarch $DIRACOS_REPO/x86_64 $DIRACOS_REPO/buildOnly
createrepo $DIRACOS_REPO
pip install git+https://github.com/DIRACGrid/DIRACOS.git
git clone https://github.com/DIRACGrid/DIRACOS.git
dos-build-all-rpms DIRACOS/config/diracos.json >& log.txt
As there is now a docker image for CentOS 8 I thought I'd quickly try it and unsurprisingly everything is horribly broken. More surprisingly it's trivial to fix enough that the client tests work.
The error when installing is:
2019-10-02 11:16:38 UTC dirac-install [NOTICE] Defaults written to defaults-DIRAC.cfg
2019-10-02 11:16:38 UTC dirac-install [NOTICE] Executing /admin/runit-2.1.2/scripts/dirac-externals-requirements...
/usr/bin/coreutils: symbol lookup error: /usr/bin/coreutils: undefined symbol: mode_to_security_class
It is due to LD_LIBRARY_PATH
is causing coreutils to take libselinux.so.1
from DIRAC OS. Ideally LD_LIBRARY_PATH
shouldn't be used at all but as that isn't an option I think it can be taken from the host instead of being included in DIRACOS.
$ docker run --rm -it centos:8 bash
dnf -y groupinstall "Development Tools"
dnf -y config-manager --set-enabled PowerTools
dnf -y install glibc-static wget freetype fontconfig pixman libXrender psmisc
wget http://smarden.org/runit/runit-2.1.2.tar.gz
tar xvf runit-2.1.2.tar.gz
cd admin/runit-2.1.2/
package/install
curl -O -L https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py
chmod +x dirac-install.py
dnf -y install python2
# Must set python to be python2 for dirac-install.py to work
alternatives --set python /usr/bin/python2
./dirac-install.py -r v7r0 -t client
[root@41a16d20ab28 /]# dos-build-package /tmp/DIRACOS/config/diracos.json rpm
INFO:root:Building http://ftp.redhat.com/pub/redhat/linux/enterprise/6Client/en/os/SRPMS/rpm-4.8.0-59.el6.src.rpm with mockConfig /tmp/DIRACOS/mockConfigs/mock-build-diracos.cfg
DEBUG:root:SRPM file is an URL, download it first
DEBUG:root:Beginning file download: http://ftp.redhat.com/pub/redhat/linux/enterprise/6Client/en/os/SRPMS/rpm-4.8.0-59.el6.src.rpm
DEBUG:root:Writing to /tmp/rpm-4.8.0-59.el6.src.rpm
DEBUG:root:Downloading with curl -o /tmp/rpm-4.8.0-59.el6.src.rpm -L http://ftp.redhat.com/pub/redhat/linux/enterprise/6Client/en/os/SRPMS/rpm-4.8.0-59.el6.src.rpm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3734k 100 3734k 0 0 786k 0 0:00:04 0:00:04 --:--:-- 877k
INFO:root:The repo already contains a build, not rebuilding: [u'/diracos_repo/buildOnly/rpm-4.8.0-55.el6.py27.usc4.src.rpm']
In v7r0, there is only gfal2 and the new fts3 rest interface, so maybe we can remove those. To be tested
/opt/dirac/diracos/usr/lib64/python2.7/distutils/__init__.py:14: UserWarning: The virtualenv distutils package at %s appears to be in the same location as the system distutils?
"The virtualenv distutils package at %s appears to be in the same location as the system distutils?")
can be solved with the following line in the diracosrc
PYTHONWARNINGS="ignore"
Or not? give a good reason.
It's used by, e.g., for querying the BDII.
maybe adding this to the cfg?
config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/pigz /usr/bin/lbzip2'
config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'
Initial issue is that the shebang of virtualenv is
DIRAC-PROD>virtualenv /tmp/toto
/usr/bin/env: python2.7 -s: No such file or directory
The problem is this line. Only one argument is passed to the command in the shebang (https://www.in-ulm.de/~mascheck/various/shebang/)
#!/usr/bin/env python2.7 -s
This is done by the diracos bundle script, originally it is #!/usr/bin/python2.7 -s
The -s
confuses things
But even when this is fixed, it does not work
DIRAC-PROD>virtualenv /tmp/toto
Using real prefix '/usr'
New python executable in /tmp/toto/bin/python2.7
Also creating executable in /tmp/toto/bin/python
Installing setuptools, pip, wheel...
Complete output from command /tmp/toto/bin/python2.7 - setuptools pip wheel:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
ImportError: No module named pkgutil
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
File "/opt/dirac/diracos/usr/bin/virtualenv", line 3, in <module>
virtualenv.main()
File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 711, in main
symlink=options.symlink)
File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 944, in create_environment
download=download,
File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 900, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 795, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /tmp/toto/bin/python2.7 - setuptools pip wheel failed with error code 1
If you follow the instructions for installation you will currently get an error when executing
$ yum install -y mock rpm-build fedora-packager createrepo python-pip
---> Package policycoreutils.x86_64 0:2.0.83-30.1.el6_8 will be installed
--> Finished Dependency Resolution
Beginning Kernel Module Plugin
Finished Kernel Module Plugin
Error: Package: fedpkg-1.34-3.el6.noarch (epel)
Requires: python2-openidc-client >= 0.6.0
Installing: python2-openidc-client-0-3.20170523git77cb3ee.el6.noarch (epel)
python2-openidc-client = 0-3.20170523git77cb3ee.el6
Error: Package: fedpkg-1.34-3.el6.noarch (epel)
Requires: python2-openidc-client >= 0.6.0
Available: python2-openidc-client-0-3.20170523git77cb3ee.el6.noarch (epel)
python2-openidc-client = 0-3.20170523git77cb3ee.el6
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
This is due to the update of fedpkg
from 1.34-1
to 1.34-3
. The dependent library python2-openidc-client.noarch 0:0.6.0-1.20180605gitcd8d91c.el6
is still in epel-testing
and should be moved moved to epel
soon, but according to discussion with IT they don't know when. In the meantime you have to enable epel-testing
yum-config-manager --enable epel-testing
The list of shipped python packages is not complete. For example, M2Crypto is shipped because it is a dependency of fts3, but it is not mentioned.
...to work with MySQL 8
If you want to use DIRACOS in the vanilla cern/slc6-base container you would execute
curl -O http://diracos.web.cern.ch/diracos/master/diracos-master.tar.gz
curl -O https://raw.githubusercontent.com/DIRACGrid/DIRACOS/master/tests/integration/test_import.py
yum install tar -y
tar xf diracos-master.tar.gz -C /tmp
export DIRACOS=/tmp/diracos
source $DIRACOS/diracosrc
then if you run the test pytest test_import.py
it fails with
# pytest test_import.py
=============================================================== test session starts ===============================================================
platform linux2 -- Python 2.7.13, pytest-3.9.2, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/.hypothesis/examples')
rootdir: /, inifile:
plugins: cov-2.6.0, hypothesis-3.79.3
collected 109 items
test_import.py ..............................F.............................s................................................ [100%]
==================================================================== FAILURES =====================================================================
________________________________________________________________ test_module[git] _________________________________________________________________
moduleName = 'git'
@parametrize('moduleName', moduleNames)
def test_module(moduleName):
""" Try to import a module and check whether it is located in DIRACOS.
Modules that are in the ALLOWED_TO_FAIL list are shown as skipped and generate a warning
Modules that require graphic libraries on the system (GRAPHIC_MODULES) are skipped on container
"""
try:
module = __import__(moduleName)
# Test whether it is correctly imported from DIRACOS
try:
modulePath = module.__file__
# return true, if the common prefix of both is equal to directory
# e.g. /a/b/c/d.rst and directory is /a/b, the common prefix is /a/b
assert os.path.commonprefix([modulePath, diracosPath]) == diracosPath, \
"ERROR %s not from DIRACOS: %s" % (moduleName, modulePath)
# builtin modules like sys have no path
except AttributeError as e:
print "WARNING no path for %s" % moduleName
except ImportError as e:
msg = "could not import %s: %s" % (moduleName, repr(e))
print traceback.print_exc()
if moduleName in ALLOWED_TO_FAIL:
warnings.warn(msg)
pytest.skip("WARN: " + msg)
elif moduleName in GRAPHIC_MODULES:
warnings.warn(msg + "(Possibly due to system graphic libraries not present)")
pytest.skip("WARN: " + msg + "(Possibly due to system graphic libraries not present)")
else:
> pytest.fail("ERROR: " + msg)
E Failed: ERROR: could not import git: ImportError('Failed to initialize: Bad git executable.\nThe git executable must be specified in one of the following ways:\n - be included in your $PATH\n - be set via $GIT_PYTHON_GIT_EXECUTABLE\n - explicitly set via git.refresh()\n\nAll git commands will error until this is rectified.\n\nThis initial warning can be silenced or aggravated in the future by setting the\n$GIT_PYTHON_REFRESH environment variable. Use one of the following values:\n - quiet|q|silence|s|none|n|0: for no warning or exception\n - warn|w|warning|1: for a printed warning\n - error|e|raise|r|2: for a raised exception\n\nExample:\n export GIT_PYTHON_REFRESH=quiet\n',)
test_import.py:238: Failed
-------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------
None
-------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------
Traceback (most recent call last):
File "/test_import.py", line 212, in test_module
module = __import__(moduleName)
File "/tmp/diracos/usr/lib64/python2.7/site-packages/git/__init__.py", line 85, in <module>
raise ImportError('Failed to initialize: {0}'.format(exc))
ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
- be included in your $PATH
- be set via $GIT_PYTHON_GIT_EXECUTABLE
- explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
- quiet|q|silence|s|none|n|0: for no warning or exception
- warn|w|warning|1: for a printed warning
- error|e|raise|r|2: for a raised exception
Example:
export GIT_PYTHON_REFRESH=quiet
================================================= 1 failed, 107 passed, 1 skipped in 2.54 seconds =================================================
Installation using fullInstall in DIRAC/tests/Jenkins/dirac_ci.sh fails when the script wants to create a new DB in MySQL. It seems like MySQL 8 changes default authentication plugin from mysql_native_password
to caching_sha2_password
, as mentioned here.
Install log is here: log.txt.
The following environment variables were used:
DEBUG=True
DIRAC_RELEASE=v6r22-pre3
DB_USER=Dirac
DB_PASSWORD=Dirac
DB_ROOTUSER=admin
DB_ROOTPWD=password
DB_HOST=172.18.0.4
DB_PORT=3306
DIRACOSVER=v1r1
NoSQLDB_HOST=172.18.0.5
NoSQLDB_PORT=9200
repository=DIRACGrid
branch=rel-v6r22
DIRACBRANCH=rel-v6r22
as well as having MySQL 8.0 and ElasticSearch 6.6.0 in separate Docker containers. The server was also installed in a Docker container. The admin user in docker was created using
CREATE USER 'admin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'admin'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
quit
Solutions to this include:
ALTER USER admin IDENTIFIED WITH mysql_native_password BY 'password';
(DIRAC code)@petricm @andresailer
not urgent at all, but could you help me setup the release script and templates as you did for your DIRAC extension and the DIRAC repo ?
Thanks !
That will save a few download for SRPM
In [2]: srpm = '/eos/project/l/lhcbwebsites/www/lhcb-rpm/dirac/DIRACOS/SRPM/dcap-2.47.12-4.el6.src.rpm'
In [3]: urlSrpm = 'http://mirror.switch.ch/ftp/mirror/epel/6/SRPMS/Packages/d/dcap-2.47.12-4.el6.src.rpm'
In [7]: rpmUtils.miscutils.splitFilename( os.path.basename(srpm))
Out[7]: ('dcap', '2.47.12', '4.el6', '', 'src')
In [8]: rpmUtils.miscutils.splitFilename( os.path.basename(urlSrpm))
Out[8]: ('dcap', '2.47.12', '4.el6', '', 'src')
As mentioned in #73
There is a pre-release ready which fixes the bug: xrootd/xrootd#977 (comment)
Now that the tests are self contained for GitHub actions it should be simple to add to the GitLab CI pipeline.
I'm happy to implement it but thought I should ask first.
Any thoughts?
Things like https://github.com/DIRACGrid/DIRACOS/blob/master/diracos/bundlelib.py#L138 are becoming unreadable and error prone.
It maybe would be better to have templated shell scripts in separate files. This avoids one layer of escape characters
Dear all
we have released a new version of FTS (3.8.0)
http://cern.ch/go/z8RJ
gfal2 (2.16.0)
http://dmc.web.cern.ch/release/gfal2-2160
and Davix (0.6.9)
http://dmc.web.cern.ch/release/davix-069
They are now available on our RC repos and installed on our FTS pilot nodes.
Apart from bug fixes the main new features are:
- Automatic Session Reuse
http://fts3-docs.web.cern.ch/fts3-docs/docs/session_reuse.html#automatic-session-reuse
- Macaroons and Scitokens support ( thanks to Brian Bockelman)
http://fts3-docs.web.cern.ch/fts3-docs/docs/3rdpartycopy.html#macaroons-and-scitokens-support
We are going to enable them on the pilot by this week so they can be
validated
regards
Andrea
Should be compiled with setuid support disabled
Dumping before I go home and forget, nss
related dependencies need to be a consistent version and are not currently included in DIRACOS. This causes curl and git to be unable to use SSL. Example:
tar xvf diracos-v1r5.tar.gz
# Doesn't work
singularity exec --bind /cvmfs /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc -c 'source diracos/diracosrc; curl -v https://github.com > /dev/null'
# Binding /lib64 into the container fixes it
singularity exec --bind /lib64 --bind /cvmfs /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc -c 'source diracos/diracosrc; curl -v https://github.com > /dev/null'
# Hackily update (found using strace | grep "\"/lib64/")
cp /lib64/libfreebl3.so diracos/lib64/
cp /lib64/libsoftokn3.so diracos/lib64/
cp /lib64/libfreeblpriv3.so diracos/lib64/
cp /lib64/libfreeblpriv3.chk diracos/lib64/
cp /lib64/libfreeblpriv3.so diracos/lib64/
cp /lib64/libsoftokn3.chk diracos/lib64/
cp /lib64/libsoftokn3.so diracos/lib64/
cp /lib64/libnsssysinit.so diracos/lib64/
cp /lib64/libnssdbm3.so diracos/lib64/
cp /lib64/libnssdbm3.chk diracos/lib64/
cp /lib64/libnssdbm3.so diracos/lib64/
# Now works without binding
singularity exec --bind /cvmfs /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc -c 'source diracos/diracosrc; curl -v https://github.com > /dev/null'
Getting requirements for yum-3.2.29-81.el6.py27.usc4.src
--> python-2.7.13-2.el6.x86_64
--> gettext-0.17-18.el6.x86_64
--> intltool-0.41.0-1.1.el6.noarch
--> python-nose-0.10.4-3.1.el6.py27.usc4.noarch
--> python-2.7.13-2.el6.x86_64
--> rpm-python-4.8.0-59.el6.x86_64
--> Already installed : rpm-4.8.0-59.el6.x86_64
--> python-iniparse-0.3.1-2.1.el6.py27.usc4.noarch
--> python-2.7.13-2.el6.x86_64
--> python-urlgrabber-3.9.1-11.el6.py27.usc4.noarch
--> yum-metadata-parser-1.1.2-16.el6.py27.usc4.x86_64
--> pygpgme-0.1-18.20090824bzr68.el6.py27.usc4.x86_64
Error: Package: rpm-python-4.8.0-59.el6.x86_64 (base)
Requires: libpython2.6.so.1.0()(64bit)
Error: Package: rpm-python-4.8.0-59.el6.x86_64 (base)
Requires: python(abi) = 2.6
Installing: python-2.7.13-2.el6.x86_64 (local-py2.7)
python(abi) = 2.7
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
On a raw SLC6 docker image:
[root@a97b0495c41e releases]# rrdtool
rrdtool: error while loading shared libraries: libfreetype.so.6: cannot open shared object file: No such file or directory
[root@a97b0495c41e releases]# ssh
ssh: error while loading shared libraries: libfipscheck.so.1: cannot open shared object file: No such file or directory
For rrdtool, the issue is that libfreetype is declared as a dependency for building, not for running.
For ssh, it's not declared at all..
It says nowhere in the doc about pip packages
404 not found
INFO:root:Building http://dl.fedoraproject.org/pub/epel/6Server/SRPMS/Packages/l/lcgdm-1.9.2-1.el6.src.rpm with mockConfig /DIRACOS/mockConfigs/mock-build-diracos.cfg
DEBUG:root:SRPM file is an URL, download it first
DEBUG:root:Beginning file download: http://dl.fedoraproject.org/pub/epel/6Server/SRPMS/Packages/l/lcgdm-1.9.2-1.el6.src.rpm
DEBUG:root:Writing to /tmp/lcgdm-1.9.2-1.el6.src.rpm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
128 257 128 257 0 0 750 0 --:--:-- --:--:-- --:--:-- 1511
DEBUG:root:Checking existance of /DIRACOS/patches/lcgdm.patch
DEBUG:root:Rebuild SRPM with ['/usr/bin/mock', '-r', u'/DIRACOS/mockConfigs/mock-build-diracos.cfg', '--rebuild', u'/tmp/lcgdm-1.9.2-1.el6.src.rpm']
INFO: mock.py version 1.3.5 starting (python version = 2.6.6)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
ERROR: Cannot find/open srpm: /tmp/lcgdm-1.9.2-1.el6.src.rpm. Error: error reading package header
Traceback (most recent call last):
File "/usr/bin/dos-build-package", line 9, in <module>
load_entry_point('diracos==0.0.1', 'console_scripts', 'dos-build-package')()
File "/usr/lib/python2.6/site-packages/scripts/buildPackage.py", line 20, in main
diracoslib.buildPackage(package)
File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 463, in buildPackage
_buildFromSRPM(packageCfg)
File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 326, in _buildFromSRPM
_mockRebuild(srpmFile, mockConfig)
File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 355, in _mockRebuild
subprocess.check_call(cmd)
File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/mock', '-r', u'/DIRACOS/mockConfigs/mock-build-diracos.cfg', '--rebuild', u'/tmp/lcgdm-1.9.2-1.el6.src.rpm']' returned non-zero exit status 1
When this happens (I will put it on the doc):
check the website if there was just a minor version upgrade. If it is, fine, let's just use that one.
You can see there is lcgdm-1.10 so it is NOT a minor version.
In that case, just google for the exact RPM on google, and you can take it from any repo.
For example ftp://ftp.icm.edu.pl/vol/rzm8/linux-fedora/linux/epel/6/SRPMS/Packages/l/lcgdm-1.9.2-1.el6.src.rpm
So you edit the json file and replace the link (good test though, never tried an ftp link)
do what is said in the title
Would it be possible to use the xrootd5 release candidate for testing?
yum-utils-1.1.30-41(base) vs yum-utils-1.1.30-40(lhcb)
[root@90c8bd9bc220 ~]# dos-bundle DIRACOS/config/diracos.json
INFO:root:Bootstraping packaging of diracos
INFO: mock.py version 1.3.5 starting (python version = 2.6.6)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: chroot init
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled HW Info plugin
Start: yum install
Error: Package: yum-utils-1.1.30-41.el6.noarch (base)
Requires: python(abi) = 2.6
Available: python-2.7.13-2.el6.i686 (local-py2.7)
python(abi) = 2.7
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
ERROR: Command failed:
# /usr/bin/yum --installroot /var/lib/mock/epel-6-x86_64-install/root/ install @buildsys-build yum yum-utils python-devel xz-devel gcc git
Error: Package: yum-utils-1.1.30-41.el6.noarch (base)
Requires: python(abi) = 2.6
Available: python-2.7.13-2.el6.i686 (local-py2.7)
python(abi) = 2.7
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
Traceback (most recent call last):
File "/usr/bin/dos-bundle", line 9, in <module>
load_entry_point('diracos==0.0.1', 'console_scripts', 'dos-bundle')()
File "/usr/lib/python2.6/site-packages/scripts/bundleDiracOS.py", line 17, in main
diracoslib.bundleDIRACOS(cfg)
File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 609, in bundleDIRACOS
subprocess.check_call(bundleCmd)
File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mock', '-r', u'/root/DIRACOS/mockConfigs/mock-install-diracos.cfg', '--shell', 'python /tmp/bundlelib.py']' returned non-zero exit status 30
If you untar the bundle into a bare container you will find some broken links
find . -xtype l
./diracos/usr/lib64/python2.7/site-packages/PyQt4/uic/pyuic.py
./diracos/usr/lib64/python2.7/UserDict.py
./diracos/usr/lib64/python2.7/_abcoll.py
./diracos/usr/lib64/python2.7/_weakrefset.py
./diracos/usr/lib64/python2.7/abc.py
./diracos/usr/lib64/python2.7/codecs.py
./diracos/usr/lib64/python2.7/copy_reg.py
./diracos/usr/lib64/python2.7/fnmatch.py
./diracos/usr/lib64/python2.7/genericpath.py
./diracos/usr/lib64/python2.7/linecache.py
./diracos/usr/lib64/python2.7/locale.py
./diracos/usr/lib64/python2.7/ntpath.py
./diracos/usr/lib64/python2.7/os.py
./diracos/usr/lib64/python2.7/posixpath.py
./diracos/usr/lib64/python2.7/re.py
./diracos/usr/lib64/python2.7/sre.py
./diracos/usr/lib64/python2.7/sre_compile.py
./diracos/usr/lib64/python2.7/sre_constants.py
./diracos/usr/lib64/python2.7/sre_parse.py
./diracos/usr/lib64/python2.7/stat.py
./diracos/usr/lib64/python2.7/types.py
./diracos/usr/lib64/python2.7/warnings.py
./diracos/usr/lib64/libcom_err.so
./diracos/usr/lib64/libkeyutils.so
./diracos/usr/share/doc/runit-2.1.2/debian
./diracos/usr/share/doc/nordugrid-arc-5.4.0/arc.conf.reference
./diracos/usr/bin/voms-proxy-list
./diracos/usr/lib/jvm-exports/java-1.7.0-oracle.x86_64
./diracos/usr/lib/jvm-exports/java-1.8.0-oracle.x86_64
./diracos/etc/fonts/conf.d/58-dejavu-lgc-sans-mono.conf
./diracos/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans-mono.conf
./diracos/etc/fonts/conf.d/57-dejavu-sans-mono.conf
./diracos/etc/fonts/conf.d/20-unhint-small-dejavu-sans-mono.conf
all the python stuff links to /usr/lib64
which I don't think is correct
ll diracos/usr/lib64/python2.7/UserDict.py
lrwxrwxrwx. 1 root root 32 Oct 24 00:49 diracos/usr/lib64/python2.7/UserDict.py -> /usr/lib64/python2.7/UserDict.py
Mock can be instructed to use N cores to build with
config_opts['macros']['%_smp_mflags'] = "-j4"
it might be worth mentioning this or even better to add the option
dos-build-all-rpms -j4 /tmp/DIRACOS/config/diracos.json
Currently the DIRACOS installation of git
depends on the host's helper binaries:
$ git --exec-path
/usr/libexec/git-core
As DIRACOS needs to be runtime relocatable without editing the binaries there are several ways of working around this:
export GIT_EXEC_PATH=$DIRAC/diracos/usr/libexec/git-core
to the bashrc
diff --git a/git.spec b/git.spec
index f1482ff..baa9339 100644
--- a/git.spec
+++ b/git.spec
@@ -248,6 +248,7 @@ htmldir = %{_docdir}/%{name}-%{version}
prefix = %{_prefix}
ASCIIDOC8 = 1
ASCIIDOC_NO_ROFF = 1
+RUNTIME_PREFIX = YesPlease
EOF # Filter bogus perl requires
--
2.21.0
Unfortunately 2. has the caveat that git 1.7.1 only works if you use an absolute path to git
:
$ git --exec-path
git: exec_cmd.c:23: system_path: Assertion `argv0_path' failed.
Aborted
$ $(which git) --exec-path
/tmp/git-build/usr/libexec/git-core
This can be accepted as a "feature" or avoided by:
git
with a wrapper script that calls exec "${DIRAC}/something/git" "$@"
/proc
to read the exe
symlinkUpdating to a newer git seems like the cleanest option but I'm not sure how difficult it would be.
Currently singularity tries to find it's starter binary in /usr/libexec/singularity/bin/starter
instead of getting it from DIRACOS. It's found here:
Using the proper tools for scripts (arguments, logging, help, etc)
This allows to build the LD_LIBRARY_PATH and PATH at construction time. the bashrc from DIRAC should then just source this one.
Specify how to get the static version list for the requirements.txt
for i in $(awk -F '[>=]' {'print $1'} requirements.txt | grep -v git); do latest=$(pip install $i==0.0.0 2>&1 | grep 'from versions' | awk -F '[,:]' {'print $NF'} | sed -e 's/)//g' -e 's/ //g'); echo "$i==$latest"; done
All in the title :)
Trying to submit job we have in result
VERBOSE: Running command: arcsub -d DEBUG -c hepgrid5.ph.liv.ac.uk job2.xrsl
VERBOSE: System configuration file (/etc/arc/client.conf or /usr/etc/arc/client.conf) does not exist.
INFO: Cannot copy example configuration (/usr/share/arc/examples/client.conf), it is not a regular file
INFO: No configuration file could be loaded.
INFO: Using proxy file: /home/dirac/testARC/x509up_u15872
INFO: Using CA certificate directory: /etc/grid-security/certificates
DEBUG: Certificate format is PEM
DEBUG: Trying to check X509 cert with check_cert_type
DEBUG: Module Manager Init
ERROR: Invalid JobDescription:
&(executable="test.sh")
(inputFiles=(test.sh "test.sh"))
(stdout="test.out")
(stderr="test.err")
(outputFiles=("test.out" "") ("test.err" ""))
No job description parsers available
It seems ARC does not use plugins despite they are formally "installed".
At lxplus we have during job submission
....
DEBUG: Module Manager Init
DEBUG: Loaded /usr/lib64/arc/libaccJobDescriptionParser.so
DEBUG: Loaded JobDescriptionParserPlugin ARCJSDLParser
DEBUG: Loaded JobDescriptionParserPlugin JDLParser
DEBUG: Loaded JobDescriptionParserPlugin XRSLParser
VERBOSE: String successfully parsed as nordugrid:xrsl.
....
It seems ARC looks for plugins at /usr/lib64/arc only by default.
Resume:
ARC should be configured somehow.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.