Add an option that rebuilds all dependent ports when a port is rebuilt (i.e. png may have a major version bump requiring a reinstall of all dependent software).

Objective

Part 8 of "sigev" milestone.

Adapt to sigev to be compatible with Python 3.x, or minimise differences (with documentation) to allow easy porting.

Some ports, such as net/cvsupchk, require a dependencies to have completed a specific stage, such as extract, not installation.

Objective

Part 6 of "sigev" milestone

Port portbuilder to use sigev code

When trying to update all installed ports via portbuilder -baU I see the following exception (which gets logged in the logfile at /tmp/portbuilder/portbuilder):

[...]
[     3.7609] (D) Stage.work()> Port 'net/avahi-app': starting stage Config
[     3.7641] (D) DependLoader._find_method()> Port 'net/avahi-app': resolving using method 'build'
[     3.7649] (D) Stage._finalise()> Port 'net/avahi-app': finished stage Config
[     3.7685] (D) Stage.work()> Port 'net/avahi-app': starting stage Depend
[     4.1654] (D) Stage.work()> Port 'security/nss': starting stage Config
[     4.1684] (D) DependLoader._find_method()> Port 'security/nss': resolving using method 'build'
[     4.1692] (D) Stage._finalise()> Port 'security/nss': finished stage Config
[     4.1726] (D) Stage.work()> Port 'security/nss': starting stage Depend
[     4.2747] (D) Stage.work()> Port '': starting stage Config
[     4.2795] (D) DependLoader._find_method()> Port '': resolving using method 'build'
[     4.2804] (D) Stage._finalise()> Port '': finished stage Config
[     4.2839] (D) Stage.work()> Port '': starting stage Depend
[     4.2864] (D) Stage._finalise()> Port '': finished stage Depend
[     4.2953] (D) Stage.work()> Port '': starting stage Checksum
[     4.2969] (D) Stage._finalise()> Port '': finished stage Checksum
[     4.2982] (D) StageBuilder._cleanup()> Port '': completed job for stage Checksum
[     4.2990] (D) Stage.work()> Port '': starting stage Fetch
[     4.3000] (D) Stage._finalise()> Port '': finished stage Fetch
[     4.3064] (D) StageBuilder._cleanup()> Port '': completed job for stage Fetch
[     4.3073] (D) StageBuilder._port_ready()> Port '': queuing job for stage Build
[     4.3088] (D) Stage.work()> Port '': starting stage Build
[     4.316] (EXCEPTION)

Traceback from signal connection <2.0000> (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/libpb/builder.py", line 463, in _add
    super(BuildBuilder, self)._add(dependjob, port, pending)
  File "/usr/local/lib/python2.7/site-packages/libpb/builder.py", line 348, in _add
    builders[self.stage.prev].add(port).connect(self._stage_resolv)

Traceback from signal emitter <2.0076> (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/libpb/stacks/base.py", line 97, in _finalise
    self.done()
  File "/usr/local/lib/python2.7/site-packages/libpb/job.py", line 60, in done
    self.emit(self)

Traceback from exception (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/libpb/builder.py", line 405, in _stage_resolv
    self._port_ready(port)
  File "/usr/local/lib/python2.7/site-packages/libpb/builder.py", line 433, in _port_ready
    self.queue.add(stagejob)
  File "/usr/local/lib/python2.7/site-packages/libpb/queue.py", line 51, in add
    self._run()
  File "/usr/local/lib/python2.7/site-packages/libpb/queue.py", line 88, in _run
    job.run(self)
  File "/usr/local/lib/python2.7/site-packages/libpb/job.py", line 52, in run
    self.work()
  File "/usr/local/lib/python2.7/site-packages/libpb/stacks/base.py", line 78, in work
    self._do_stage()  # May throw job.JobStalled()
  File "/usr/local/lib/python2.7/site-packages/libpb/stacks/mutators.py", line 70, in _do_stage
    self._pre_make()
  File "/usr/local/lib/python2.7/site-packages/libpb/stacks/build.py", line 173, in _pre_make
    self._make_target(("all",), BATCH=True, NO_DEPENDS=True)
  File "/usr/local/lib/python2.7/site-packages/libpb/stacks/mutators.py", line 74, in _make_target
    pmake = make.make_target(self.port, targets, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/libpb/make.py", line 80, in make_target
    stdout = open(port.log_file, 'a')
IOError: [Errno 21] Is a directory: '/tmp/portbuilder/'

I guess this is due to ports that have been removed in the past (e.g. bsdpan-Purple-0.01 ! Comparison failed ) and while I'm not surprised to see this happen it would be nice if portbuilder gave a better error message.

Objective

Part 2 of "sigev" milestone

Create external event bridge using kevent and adapt (context) delegator to use it.

KEvent

• EVFILT_PROC
• EVFILT_VNODE
• EVFILT_TIMER

The package stack should include a PkgConf stage to check if the package has the configuration the user selected, mimicing what happens in the repo stack.

Install and package stages run concurrently, but pkgng locks its database on each of them, so one of them might fail if they both try to acquire the lock at the same time. I've already seen such a problem in the logs:

# make -C /usr/ports/x11/xterm package -DNO_DEPENDS -DBATCH
===>  Building package for xterm-285
pkg: sqlite: database is locked (pkgdb.c:2363)
*** [do-package] Error code 1

This sounds like a pretty serious issue.

Rebuild ports that have changed options

Let
/ be an amd64 environment
/i386 be an i386 environment

then the following will happen:

portbuilder x11/xorg

success

portbuilder -C /i386 x11/xorg

success

chroot /i386 portbuilder x11/xorg

stall

When using a i386 python then some builds will stall however portbuilder itself does not experience a deadlock.

Further investigation is required to determine what the problem is. (Threading, kevent, int overflow)

The user may want to rebuild a port, allow that to be specified.

Objective

Part 5 of "sigev" milestone.

"Port" subprocess to asynchronous using sigev.

A package shouldn't be installed unless its dependencies match those of the port after make config was run. A stage must be added, common to package and repostacks.

David suggests that a Pkg class should be created to handle all this information. Unless he does that in the next days, I think I'll start fixing this by creating the new stage (similar to RepoConf). The work can be converted in the future, anyway.

Objective

Part 1 of milestone "sigev"

Create a library that does "pure" signal/event/delegation, with unit tests.

Signal Classes

• Signal
• SignalProperty
• EdgeSignal
• SingleSignal

Delegation

• Standard Delegator (event loop)
• Context Managed Delegator

Objective

Part 4 of "sigev" milestone

"Port" queue to asynchronous using sigev

Queue

• dynamic priority of queued jobs
• directly handle consumption

A bug (at least as I was told) in pkgng makes pkg install downgrade its dependencies. pkg fetch and pkg add must be used (even if the "bug" is fixed, older pkgng should be supported for a while), thus the RepoFetch stage must be enabled. This should also give some speedup, as it would parallelise package fetching (sharing the fetch queue).

I'll see if I can work on this.

pkg.info(repo=True) doesn't work with pkg_* tools, so the whole repo stack isn't even started. Additionally, pkg.add(repo=True) calls pkg_add -r $pkgname, but pkg_add -r needs LATEST_LINK. The former can be fixed making the Repo stage return True when using pkg_* (an abstract method should be created for this, like pkg.exists() or similar), the latter by using the RepoFetch stage.

This is not high priority, in my opinion.

portbuilder crashes when resizing the terminal, mostly when enlarging.
Traceback:

Traceback (most recent call last):
File "/usr/local/bin/portbuilder", line 146, in run_loop
run()
File "/usr/local/lib/python2.7/site-packages/libpb/event.py", line 136, in run
self._queue()
File "/usr/local/lib/python2.7/site-packages/libpb/event.py", line 160, in _queue
for ev in self._kq.control(None, 2, timeout):
OSError: [Errno 4] Interrupted system call

Traceback (most recent call last):
File "/usr/local/bin/portbuilder", line 419, in
main()
File "/usr/local/bin/portbuilder", line 124, in main
run_loop(delegate.no_port, options)
File "/usr/local/bin/portbuilder", line 173, in run_loop
run()
File "/usr/local/lib/python2.7/site-packages/libpb/event.py", line 136, in run
self._queue()
File "/usr/local/lib/python2.7/site-packages/libpb/event.py", line 160, in _queue
for ev in self._kq.control(None, 2, timeout):
OSError: [Errno 4] Interrupted system call

Objective

Part 3 of "sigev" milestone.

Port portbuilder to use sigev code

Allow a port specific override of the --method command.

When in multi-repo mode, several pkgng commands print a warning header. portbuilder should just ignore that (or, better, it should be smart and detect the desired output).

portbuilder sets as automatic every port passed as an argument; when -a is used, this means all installed ports. The "auto" flag should be set from scratch only for new ports passed as arguments; if a port already exists, it should inherit the flag.

I suggest to fetch the flag in pkg.info() (something like pkg query %a:%n-%v:%o), to avoid calling pkg query %a for every port, as the difference in time is 0.05s vs. 5s with 700+ packages. PKGDB.ports could then become a dictionary of tuples, dictionaries or whatever (with PKGDB.status() being expanded to return information about the property), or a Port-like class could be made for packages (and stored in the dictionary) to allow for future expansion, but I don't know how this would affect performance.
Another possibility would be to run pkg query %a (which should be called via pkg.query()) in PkgInstaller._post_pkg_add(). It would add very low overhead only on installation, but I believe it would be much better to have it in Port.flags from the beginning.

Something like portmaster -x $port would be really useful - with multiple iterations of the option accepted -, causing $port to be ignored when it's pkg.OLDER and -u or -U is being used; its dependency tree should be considered resolved, too, and none of its dependencies should be queued for update unless it was explicitly specified or it's brought in by another (non-excluded) port.