shared setup should specify that `vsc` in a namespace package about vsc-install HOT 18 CLOSED

currenlty, vsc is not a pure namespace in vsc-install, it holds vsc.fancylogger for backward compatible reasons. but it can be a good default.
however, the magic is in the __init__, which should already be correct. can you ask the content of the vsc/init in both vsc-install egg and vsc-base?

from vsc-install.

I'm not saying the magic content in the __init__.py is wrong, not at all.

Only one __init__.py will ever be executed (the others will be scanned, but their contents will not be executed). That's the case already.

The warnings happen because Python (or setuptools, whatever) notices that the vsc/__init__.py files are present in multiple different places on the filesystem (i.e. in different .eggs subdirectories).

If we register vsc as a namespace package in the setup.py, we're just telling Python that's exactly what we want to do, and it will stop warning about this.

Note that we get basically the same warning in the case where vsc-base is available both in the EasyBuild installation prefix and on the OS (there it is wrong, and the warning is legitimate)
We'll need to check whether the warning is still going to occur in that case. We would want that, I think...

from vsc-install.

well, the magic content in the eb __init__ is wrong. i also do not understand the original error message, as eb doesn't use pkg_resources.

the warnings can't come only from your exlpanation, as i've never seen them, and i have quite a few more packages extending vsc namespace installed.

btw, how do you know that only one __init__ is imported?

(but i won't object the change or anything, it can only improve stuff)

from vsc-install.

I looked into how the __init__s are picked up before. The reason we need to put the same thing in every vsc/__init__.py (or easybuild/__init__.py) are exactly because only one is actually executed (the first one that is found in the Python search path).

The magic in easybuild/__init__.py is wrong in the sense that it's not zip-safe yet, that's true. We'll fix that.

Note that the warnings w.r.t. the easybuild namespace are triggered via vsc/__init__.py which imports pkg_resources, see the error messages mentioned in the description in easybuilders/easybuild-framework#1588.

from vsc-install.

so importing pkg_resources triggers something in another namespace too? yuck.
there is no difference to what we use

import pkg_resources
pkg_resources.declare_namespace(__name__)

and what is recommended

__import__('pkg_resources').declare_namespace(__name__)

right?

wrt __init__, i thought they were all executed, and the last one might do stupid things, like not extending. (how is each path added if they are not executed?

[stdweird@spikev ~]$ python -v -c 'import vsc;print vsc.__path__' 2>&1 |grep vsc/_
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_utils-1.7.4-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_config-1.21-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_ldap-1.3.4-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_ldap_extension-1.8-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_filesystems-0.28.2-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_accountpage_clients-0.6-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_postgres-1.2-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_administration-0.28.2-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/vsc_quattor-0.6.13-py2.7.egg/vsc/__init__.pyc
# /home/stdweird/.local/lib/python2.7/site-packages/vsc_install-0.9.15-py2.7.egg/vsc/__init__.pyc matches /home/stdweird/.local/lib/python2.7/site-packages/vsc_install-0.9.15-py2.7.egg/vsc/__init__.py
import vsc # precompiled from /home/stdweird/.local/lib/python2.7/site-packages/vsc_install-0.9.15-py2.7.egg/vsc/__init__.pyc
# /home/stdweird/.local/lib/python2.7/site-packages/vsc_base-2.4.17-py2.7.egg/vsc/__init__.pyc matches /home/stdweird/.local/lib/python2.7/site-packages/vsc_base-2.4.17-py2.7.egg/vsc/__init__.py
import vsc # precompiled from /home/stdweird/.local/lib/python2.7/site-packages/vsc_base-2.4.17-py2.7.egg/vsc/__init__.pyc
import vsc # loaded from Zip /home/stdweird/.local/lib/python2.7/site-packages/mympingpong-0.7.0-py2.7.egg/vsc/__init__.pyc

from vsc-install.

The paths are added when the first __init__.py is executed, which goes hunting for the other __init__.py's. It doesn't execute them, just find them (stat vs open).

from vsc-install.

@stdweird: I don't see how the import vs __import__ would be different, but I think it's recommended not to use __import__

from vsc-install.

@boegel don't think so

[stdweird@spikev tmp]$ echo "print 'a';__import__('pkg_resources').declare_namespace(__name__)" > a/abcd/__init__.py
[stdweird@spikev tmp]$ echo "print 'b';__import__('pkg_resources').declare_namespace(__name__)" > b/abcd/__init__.py
[stdweird@spikev tmp]$ python -c 'import abcd;print abcd.__path__' 
a
b
['/tmp/b/abcd', '/tmp/a/abcd']

from vsc-install.

was missing

export PYTHONPATH="a:b"
mkdir -p a/abcd b/abcd

(yes, abc is taken 😄 )

from vsc-install.

also look at the order, b is added last, but to the front
happy times not giving a crap about pythonpath!

from vsc-install.

@stdweird: the behaviour is different with pkg_resources .declare_namespace than with pkgutil.extend_path (which is what we were using before); also note that the order is correct here

$ echo "print 'a';__path__ = __import__('pkgutil').extend_path(__path__, __name__)" > a/abcd/__init__.py
$ echo "print 'b';__path__ = __import__('pkgutil').extend_path(__path__, __name__)" > b/abcd/__init__.py
$ python -c 'import abcd;print abcd.__path__'                                                           
a
['/private/tmp/a/abcd', '/private/tmp/b/abcd']

from vsc-install.

oh boy, good to know.

from vsc-install.

I'm not sure it matters a lot though...

Do we care in what order the paths to vsc.utils or vsc.testing are added to sys.path?

from vsc-install.

only when you are testing and the modules your are testing are in the path (e.g. because it's installed).
and in case of EB, when someone is developing their own flavour of e.g. an easyblock that is part of the release. setting the location of your custom version in PYTHONPATH might/will not be enough to force the lookup order (which is similar to testing, but in case of EB, counts as a vlid non-tetsing usecase).

from vsc-install.

I indeed had to take this into account when switching to declare_namespace in the EasyBuild framework, see easybuilders/easybuild-framework#1593.

For --include-easyblocks, I'm now hard injecting the path to custom easyblocks in easybuild.easyblocks.__path__; it's the only way to be sure that they get preference over others.

I also had to tweak the tests quite a bit, since we're playing around with injecting custom (test) easyblocks all over the place.

from vsc-install.

@boegel we need a sort function that, given a module, sorts the __path__ according to PYTHONPATH

from vsc-install.

@stdweird :)

from vsc-install.

fixed with #22

from vsc-install.