Basic functionality should be similar to quickpkg from portage.

Name ideas: ppkg, pbinpkg

For example, if you're working in a repo that isn't in the repos.conf (or PORTDIR_OVERLAYS) we should support trying to dynamically add it in similar to how 'ebuild' from portage works.

A generic method could be handy that takes a path, tests if it's within a valid repo structure, and then temporarily appends it to the available repos in the config.

Should work the same as portage.

Our build scripts use multiple jobs to run emerge in parallel, would be nice if pkgcore could as well.

python setup.py \
    build_ext -I "${VIRTUAL_ENV}"/include/site/python2.7 \
    install --disable-man-pages --disable-html-docs

causes pkgcore to ignore the --disable-* options. Passing the commands separately works.

This is probably broken due to not installing snakeoil and other deps required for building all the docs since some internal snakeoil/pkgcore modules are used to create the docs currently.

This will probably require adding a requirements.txt file to the repo or similar to make sure the deps are installed before docs are generated.

Tests should be moved into the root directory of the repo since we don't need to distribute and install them in the main module; however, note that there is some test code that should be kept in the pkgcore namespace since it's useful outside of the pure testing usecase (e.g. FakePkg). Not installing tests will probably become more important once we start reworking them and adding a lot more resolver tests, bash tests, and perhaps even integration style tests for pmerge, pinspect, etc.

Most of the test setup will have to be rewritten to support this but perhaps we can remove more snakeoil and pkgcore distutils hacks at the same time.

This should work similar to __shopt_push/__shopt_pop and __IFS_push/__IFS_pop from isolated-functions.lib. In fact all three methods probably should share a common, generic push/pop stack implementation similar in functionality to the implementations in eutils.eclass.

With variable pushing and popping we will be able to easily do things such as:

__var_push LC_COLLATE C
files=( /path/to/some/files/*.{diff,patch} )
__var_pop

which enables a stable sorting method while building the patch files list without having to manually save and restore the value of LC_COLLATE.

Currently running pkgcore from a git checkout causes the system location for pkgcore plugin's to be regenerated as well if pkgcore is installed.

This fails if we're running pkgcore as a regular user since they don't have write access to the system location.

If an ebuild has an invalid mirror or similar issue using wrong metadata an unhandled exception is thrown. We should instead try to return helpful error messages as long as --debug isn't enabled.

A current example is the tree are the more recent mail-client/mutt ebuilds that have "bitbucket" mirrors in SRC_URI but those were removed from the thirdpartymirrors file in the repo profile a while ago.

When I have */*::some_overlay in package.mask, this happens.

# pmerge -NuDav --debug
Error in configuration:
Traceback follows:
  File "/usr/lib/python2.7/site-packages/pkgcore/util/commandline.py", line 827, in main
    main_func, options = argparse_parse(subcommands, args, options)
  File "/usr/lib/python2.7/site-packages/pkgcore/util/commandline.py", line 738, in argparse_parse
    namespace = parser.parse_args(args, namespace=namespace)
  File "/usr/lib/python2.7/site-packages/pkgcore/util/commandline.py", line 600, in parse_args
    delayed(args, attr)
  File "/usr/lib/python2.7/site-packages/pkgcore/util/commandline.py", line 349, in __call__
    self.invokable(namespace, attr)
  File "/usr/lib/python2.7/site-packages/pkgcore/util/commandline.py", line 230, in store_default
    obj = config.get_default(config_type)
  File "/usr/lib/python2.7/site-packages/pkgcore/config/central.py", line 565, in get_default
    "Failed instantiating default %s %r" % (type_name, defaults[0][0])))
  File "/usr/lib/python2.7/site-packages/pkgcore/config/central.py", line 560, in get_default
    return defaults[0][1].instantiate()
  File "/usr/lib/python2.7/site-packages/pkgcore/config/central.py", line 143, in instantiate
    compatibility.raise_from(errors.InstantiationError(self.name))
  File "/usr/lib/python2.7/site-packages/pkgcore/config/central.py", line 139, in instantiate
    self._instance = self._instantiate()
  File "/usr/lib/python2.7/site-packages/pkgcore/config/central.py", line 211, in _instantiate
    "exception caught from %r" % (errors._identify_functor_source(self.type.callable),)))
  File "/usr/lib/python2.7/site-packages/pkgcore/config/central.py", line 206, in _instantiate
    self._instance = callable_obj(*pargs, **configdict)
  File "/usr/lib/python2.7/site-packages/pkgcore/ebuild/domain.py", line 346, in __init__
    filtered = self.generate_filter(generate_masking_restrict(mask_atoms),
  File "/usr/lib/python2.7/site-packages/pkgcore/ebuild/domain.py", line 77, in generate_masking_restrict
    return generate_collapsed_restriction(masks, negate=True)
  File "/usr/lib/python2.7/site-packages/pkgcore/ebuild/atom.py", line 844, in generate_collapsed_restriction
    d[a.key].append(a)

Error was:
 Failed instantiating default domain 'livefs domain'
 Failed instantiating section 'livefs domain'
 Failed instantiating section 'livefs domain': exception caught from 'pkgcore.ebuild.domain.domain'
 'AndRestriction' object has no attribute 'key'

before 41dc095 (I hope the invocation is correct):

$ ./bin/pmerge -NuDav  --debug
DEBUG:pkgcore:updating cache '/usr/lib/python2.7/site-packages/pkgcore/plugins/plugincache' for new plugins
ERROR:pkgcore:Cannot write cache for /usr/lib/python2.7/site-packages/pkgcore/plugins/plugincache: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/pkgcore/plugins/.update.plugincache'. Try running pplugincache.
Error in configuration:
Traceback follows:
  File "/home/test/pkgcore/pkgcore/util/commandline.py", line 827, in main
    main_func, options = argparse_parse(subcommands, args, options)
  File "/home/test/pkgcore/pkgcore/util/commandline.py", line 738, in argparse_parse
    namespace = parser.parse_args(args, namespace=namespace)
  File "/home/test/pkgcore/pkgcore/util/commandline.py", line 600, in parse_args
    delayed(args, attr)
  File "/home/test/pkgcore/pkgcore/util/commandline.py", line 349, in __call__
    self.invokable(namespace, attr)
  File "/home/test/pkgcore/pkgcore/util/commandline.py", line 230, in store_default
    obj = config.get_default(config_type)
  File "/home/test/pkgcore/pkgcore/config/central.py", line 565, in get_default
    "Failed instantiating default %s %r" % (type_name, defaults[0][0])))
  File "/home/test/pkgcore/pkgcore/config/central.py", line 560, in get_default
    return defaults[0][1].instantiate()
  File "/home/test/pkgcore/pkgcore/config/central.py", line 143, in instantiate
    compatibility.raise_from(errors.InstantiationError(self.name))
  File "/home/test/pkgcore/pkgcore/config/central.py", line 139, in instantiate
    self._instance = self._instantiate()
  File "/home/test/pkgcore/pkgcore/config/central.py", line 211, in _instantiate
    "exception caught from %r" % (errors._identify_functor_source(self.type.callable),)))
  File "/home/test/pkgcore/pkgcore/config/central.py", line 206, in _instantiate
    self._instance = callable_obj(*pargs, **configdict)
  File "/home/test/pkgcore/pkgcore/ebuild/domain.py", line 177, in __init__
    raise_from(Failure("failed reading '%s': %s" % (fp, e)))
  File "/home/test/pkgcore/pkgcore/ebuild/domain.py", line 171, in __init__
    val.extend(action(x) for x in iter_read_bash(fs_obj.location))
  File "/home/test/pkgcore/pkgcore/ebuild/domain.py", line 171, in 
    val.extend(action(x) for x in iter_read_bash(fs_obj.location))
  File "/home/test/pkgcore/pkgcore/util/parserestrict.py", line 125, in parse_match
    r = map(convert_glob, tsplit)
  File "/home/test/pkgcore/pkgcore/util/parserestrict.py", line 47, in convert_glob
    "'*'- '%s' is disallowed however" % token)

Error was:
 Failed instantiating default domain 'livefs domain'
 Failed instantiating section 'livefs domain'
 Failed instantiating section 'livefs domain': exception caught from 'pkgcore.ebuild.domain.domain'
 domain failure: failed reading '/etc/portage/package.mask': globs must be composed of [\w-.+], with optional '*'- '*::sabayon' is disallowed however
 globs must be composed of [\w-.+], with optional '*'- '*::sabayon' is disallowed however

Debug output is as follows:

inserting MutatedPkg(built ebuild: x11-libs/gtk+-3.14.6, overrides=('depends', 'rdepends', 'post_rdepends'))
not viable     ( ( ('repo.livefs',) EqualityMatch: =False || ( ('category',) == virtual && ('package_is_real',) EqualityMatch: =False ) ) && =x11-libs/gtk+-3.14.6 ) for ( ( ('repo.livefs',) EqualityMatch: =False || ( ('category',) == virtual && ('package_is_real',) EqualityMatch: =False ) ) && =x11-libs/gtk+-3.14.6 )  no matches
failed- [<KeyedAndRestriction negate=False type=None finalized=True restrictions=(<OrRestriction negate=False type='package' finalized=True restrictions=(<PackageRestriction attr='repo.livefs' restriction=<EqualityMatch False negate=False @0x7f6afb289518> @0x7f6afb2fc9b0>, <AndRestriction negate=False type='package' finalized=True restrictions=(<PackageRestriction attr='category' restriction=<StrExactMatch 'virtual' @0x7f6afb2fca00> @0x7f6afb2fca50>, <PackageRestriction attr='package_is_real' restriction=<EqualityMatch False negate=False @0x7f6afb289680> @0x7f6afb2fcaa0>) @0x7f6afb287940>) @0x7f6afb287998>, <atom =x11-libs/gtk+-3.14.6 @#7f6af7abe910>) @0x7f6af7ae31d0>]
insertion of MutatedPkg(built ebuild: x11-libs/gtk+-3.14.6, overrides=('depends', 'rdepends', 'post_rdepends')): ([<KeyedAndRestriction negate=False type=None finalized=True restrictions=(<OrRestriction negate=False type='package' finalized=True restrictions=(<PackageRestriction attr='repo.livefs' restriction=<EqualityMatch False negate=False @0x7f6afb289518> @0x7f6afb2fc9b0>, <AndRestriction negate=False type='package' finalized=True restrictions=(<PackageRestriction attr='category' restriction=<StrExactMatch 'virtual' @0x7f6afb2fca00> @0x7f6afb2fca50>, <PackageRestriction attr='package_is_real' restriction=<EqualityMatch False negate=False @0x7f6afb289680> @0x7f6afb2fcaa0>) @0x7f6afb287940>) @0x7f6afb287998>, <atom =x11-libs/gtk+-3.14.6 @#7f6af7abe910>) @0x7f6af7ae31d0>], <pkgcore.resolver.plan.resolver_frame object at 0x7f6af7dc26d8>)
Unhandled Exception occurred:
Traceback follows:
  File "/usr/lib64/python2.7/site-packages/pkgcore/util/commandline.py", line 875, in main
    exitstatus = main_func(options, out, err)
  File "/usr/lib64/python2.7/site-packages/pkgcore/scripts/pmerge.py", line 535, in main
    resolver_inst.load_vdb_state()
  File "/usr/lib64/python2.7/site-packages/pkgcore/resolver/plan.py", line 332, in load_vdb_state
    (pkg.versioned_atom, ret))

One example that's needed when using userpriv and non-public git repos is SSH_AUTH_SOCK. This list should probably be maintained in pkgcore/ebuild/const.py.

We probably don't want to use the same exact format as emerge.log. Also, part of this is providing a pkgcore-based tool like qlop that parses the log and provides a cli interface.

Something like the following should work:

pinspect profile --repo gentoo masked_use default/linux/amd64/13.0

using 3592e9e here:

$ mkdir -p foo/metadata foo/profiles/base
$ cat <foo/metadata/layout.conf
masters = portage-stable chromiumos
profile-formats = portage-2
repo-name = stout
thin-manifests = true
use-manifests = true
EOF
$ echo chromiumos:bar > foo/profiles/base/parent
$ pinspect $PWD/foo/profiles/base
Unhandled Exception occurred:
Traceback follows:
  File "/home/vapier/pkgcore/pkgcore/util/commandline.py", line 839, in main
    exitstatus = main_func(options, out, err)
  File "/home/vapier/pkgcore/pkgcore/ebuild/inspect_profiles.py", line 55, in __call__
    out.write("\n".join(x.path for x in namespace.profile.stack))
  File "/home/vapier/snakeoil/snakeoil/klass.py", line 148, in __get__
    obj = self.function(instance)
  File "/home/vapier/pkgcore/pkgcore/ebuild/profiles.py", line 465, in stack
    return tuple(f(self.node))
  File "/home/vapier/pkgcore/pkgcore/ebuild/profiles.py", line 460, in f
    for x in node.parent_paths:
  File "/home/vapier/snakeoil/snakeoil/klass.py", line 148, in __get__
    obj = self.function(instance)
  File "/home/vapier/pkgcore/pkgcore/ebuild/profiles.py", line 116, in _load_and_invoke
    compatibility.raise_from(ProfileError(profile_path, filename, e))
  File "/home/vapier/pkgcore/pkgcore/ebuild/profiles.py", line 109, in _load_and_invoke
    return func(self, data)
  File "/home/vapier/pkgcore/pkgcore/ebuild/profiles.py", line 186, in parent_paths
    raise ValueError("repo mapping is unset")

Error was:
 failed parsing profile '/home/vapier/foo/profiles/base', file 'parent'. error: repo mapping is unset
 repo mapping is unset

Currently it doesn't recurse on deep and has other issues due to being implemented directly in the pmerge script and run against the inputted atom list instead of being put directly into the resolver where it belongs.

Currently this is broken due to repo changes.

Currently when running pmerge from the git repo as a regular user, it errors out with the following if you hit Ctrl-C when it asks if you want to merge the resolved packages:

ERROR pkgcore: failed writing binpkg Packages cache to '/var/gentoo/pkgs/Packages'; permissions issue [Errno 13]
: Permission denied: '/var/gentoo/pkgs/.update.Packages'

It seems like we could update the cache a bit later or at some other point so this isn't an issue.

It would be helpful to be able to write longer descriptions and examples in the man pages that aren't in the argparse descriptions for commands, but still keep some sort of diffing process around to be able to easily see if we're missing commands or description updates compared to the scripts.

Currently the default looks like the following for simple cases:

REQUIRED_USE requirement wasn't met
Failed to match: contains [gpm]
from: contains [gpm]
for USE: lzma bzip2 abi_x86_64 amd64 fbcon zlib jpeg kernel_linux ssl elibc_glibc unicode tiff X ipv6 userland_GNU
pkg: www-client/links-2.9

while portage's looks like:

!!! The ebuild selected to satisfy "links" has unmet requirements.

• www-client/links-2.9::gentoo USE="X bzip2 fbcon ipv6 jpeg lzma ssl tiff unicode zlib -directfb -gpm -livecd (-suid) (-svga)" ABI_X86="64"

  The following REQUIRED_USE flag constraints are unsatisfied:
  !livecd? ( fbcon? ( gpm ) )

  The above constraints are a subset of the following complete expression:
  !livecd? ( fbcon? ( gpm ) ) svga? ( suid )

We should move beyond our current flat-file based vdb at some point to something like a sqlite-based version which would probably be stored in /var/db/pkgcore/. This will also require a converter script to go at least from vdb -> vdb2 (and back for bonus points).

If this is properly spec-ed out, it might be viable to use the same generic format (and code) to generate per-repo caches that can configurably be used instead of the raw ebuild files.

Used by our current build scripts with portage.

Warn subj for requested packages.

Subj.

Often it appears error messages are printed twice in a row. This probably has something to do with how pkgcore is handling exceptions and re-raising them.

Specifically this appears to be how the dump_error method from pkgcore.util.commandline is outputting exception chains possibly in relation to the raise_from compat implementation in snakeoil.compatibility.

It appears that bash-4.3 doesn't reinitialize declarations of empty index arrays properly causing issues when the envs are dumped and reloaded. The most often hit issue is when an ebuild uses something that makes use of the estack functionality from eutils.eclass. This adds the string "declare -a _ESTACK_eshopts='()'" to the env to be dumped and on reload bash skips it, causing issues for code trying to append to the array. A quick example of the breakage can be seen when trying to install app-vim/colorschemes.

Upstream appeared to have a similar bug related to empty vars that was fixed in 4.3_p20 so this is probably a similar situation.

We need to figure out a simple testcase so upstream can reproduce and fix this.

Currently pebuild only supports atom arguments. In order to support similar workflows as ebuild from portage we should accept paths or filenames of ebuilds provided they're within a configured repo.

So we can have them outputted via the argparse error message if an unsupported one is passed as an argument.

Repo config objects are passed when using something like pmaint sync gentoo while raw repo objects are passed when using something such as pinspect eclass_usage gentoo. This causes issues since in once the the aliases attribute is a dict and in the other it's a tuple.

We should either only pass one type of repo object, use similar types of the aliases attribute, or conditionalize the object loading method that tries to iterate over available aliases.

In theory, it doesn't seem like it should be too hard to add exclude support before a resolver rewrite occurs.

We should have our own equivalent to leverage pkgcore internals and also provide more various output formats one of which should be semi-parseable.

With a failure message of the following: 'AndRestriction' object has no attribute 'slotdep_str'

Reason being the nodeps causes mutated pkgs to be merged that are of the AndRestriction type instead of the usual DepSet type where the slotdep_str method is declared.

Probably would be useful to support the same sort of framework that portage does so hooks could be used across PMs.

pmerge: error: unrecognized arguments: -b --binpkg-respect-use --with-bdeps=y

We should figure out a way to dynamically build a list of export blocks for EAPI specific functions, i.e. we shouldn't block exports from eclasses for similarly named functions but instead override them with internal PM implementations when EAPI provided functionality comes into existence.

The current method involves explicitly adding the related functions to the INTERNAL_FUNCS list the only current affected function being "usex" that is defined in EAPI 5 and in the eutils.eclass for EAPI < 5. This requires us to track implementations within eclasses which is problematic at best.

What we need is a function or script that takes an argument of a file (or files) to load and returns the list of exported functions related to that file. This would allow us to determine what functions would be exported when a certain EAPI is used so we could filter them from the dumped env.

Note that this probably has to be done with a clean env to start otherwise externally defined functions would be returned as well. If we are forced to use something like env -i bash -c /path/to/script < /path/to/sourced/file we should aim to only perform this once on startup and not per env dump since the content shouldn't change between phases that are allowed to munge the env.

I can probably do the zsh support (mostly based off or directly using gentoo-zsh-completions for the pkg-related stuff) but the bash support will have to come from someone else.

PORTDIR and PORTDIR_OVERLAYS is slowly moving away and this is the replacement. Probably not going to support env settings via PORTAGE_REPOSITORIES unless we devise something more sane.

We should support py3.3 and up.

For example:

pquery -a --repo vdb --restrict-revdep RDEPEND=dev-lang/perl

would only return installed pkg atoms that rdep on perl. Currently it grabs all deps regardless of type.

Among other uses this enables a hacky revdep-rebuild method, for example:

pquery -a --repo vdb --restrict-revdep RDEPEND=dev-libs/icu | xargs pmerge -1

We should have a testing framework in place so we can easily add unit tests for bash functions.

Currently we don't even take the repo-name setting from layout.conf into account.

Both dispatch-conf and etc-update have interesting features and uses that the other lacks. It would be useful to have an equivalent external to portage with good testsuite support.

Possible names (note that pconfig is already used): ?

We should probably respect certain environment variables so certain settings can be overridden for easier testing such as USE.

Looks like it was broken in 27f61fd. :)

# pmerge -NuDav --debug
 * Scanning for changed USE...
Unhandled Exception occurred:
Traceback follows:
  File "/usr/lib/python2.7/site-packages/pkgcore/util/commandline.py", line 839, in main
    exitstatus = main_func(options, out, err)
  File "/usr/lib/python2.7/site-packages/pkgcore/scripts/pmerge.py", line 474, in main
    for inst_pkg in installed_repos.itermatch(restrict):

Error was:
 local variable 'restrict' referenced before assignment

For example, running:

pquery --license GPL-2 --repo gentoo --debug

gives the backtrace:

repo: <filterTree raw_repo=<pkgcore.ebuild.repository._ConfiguredTree raw_repo=<ebuild _UnconfiguredTree location='/var/gentoo/repos/gentoo' @0x7f0c3067b710> wrapped=['provides', 'license', 'fetchables', 'require
d_use', 'post_rdepends', 'restrict', 'depends', 'rdepends', 'src_uri', 'iuse_effective'] @0x7f0c30069810> restriction=<AndRestriction negate=False type='package' finalized=True restrictions=(<OrRestriction negate
=False type='package' finalized=True restrictions=(<pkgcore.restrictions.delegated.delegate object at 0x7f0c3007c680>, <pkgcore.restrictions.delegated.delegate object at 0x7f0c3007c758>) @0x7f0c3007d100>, <pkgcor
e.restrictions.delegated.delegate object at 0x7f0c301d9878>, <pkgcore.restrictions.delegated.delegate object at 0x7f0c301d98c0>) @0x7f0c3007d1b0> sentinel=True @0x7f0c30069bd0>
restrict: <PackageRestriction attr='license' restriction=<ContainmentMatch2 ('GPL-2',) all=False @0x7f0c36a588e8> @0x7f0c32bbb820>
Unhandled Exception occurred:
Traceback follows:
  File "/usr/lib64/python2.7/site-packages/pkgcore/util/commandline.py", line 875, in main
    exitstatus = main_func(options, out, err)
  File "/usr/lib64/python2.7/site-packages/pkgcore/scripts/pquery.py", line 860, in main
    pkgs = list(pkgs)
  File "/usr/lib64/python2.7/site-packages/pkgcore/repository/configured.py", line 60, in <genexpr>
    return (x for x in self.raw_repo.itermatch(restrict, **kwds) if x.is_supported)
  File "/usr/lib64/python2.7/site-packages/pkgcore/repository/prototype.py", line 259, in _internal_match
    if match_func(pkg):
  File "/usr/lib64/python2.7/site-packages/pkgcore/restrictions/packages.py", line 117, in force_True
    return self.restriction.force_True(pkg, self.attr, attr)
  File "/usr/lib64/python2.7/site-packages/pkgcore/restrictions/values.py", line 467, in force_True
    filter, desired_false=false, desired_true=true):
  File "/usr/lib64/python2.7/site-packages/pkgcore/restrictions/boolean.py", line 217, in iterative_quad_toggling
    if desired_true(rest, pvals):
  File "/usr/lib64/python2.7/site-packages/pkgcore/restrictions/values.py", line 459, in true
    return pkg.request_enable(attr, r)
  File "/usr/lib64/python2.7/site-packages/pkgcore/package/conditionals.py", line 170, in request_enable
    if reqs.force_True(self):

Error was:
 force_True() takes at least 4 arguments (2 given)

Make sure to check the negation as well since the calling convention appears the same there.

This involves explaining to the user why a certain pkg is masked via license, keywords, or mask files instead of just ignoring masked versions entirely.

We should probably involve refactoring pkgcore.ebuild.domain into this work since most of the visibility code will have to be split out of there to make things more sane.