release-engineering / kobo Goto Github PK

run reads output in chunks. If a unicode character happens to be split between two chunks, decoding fails and there is a crash.

This bug triggered a failure in Fedora 31 updates-testing compose: https://pagure.io/releng/failed-composes/issue/237

Worker.update_worker call save but it does not set any variable, which is nonsense.

It should set worker attributes based on function parameters.

https://github.com/release-engineering/kobo/blob/master/kobo/hub/models.py#L253

Code should be:

def update_worker(self, enabled, ready, task_count):
    """Update worker attributes. Return worker_info.

    Update only if data provided from a worker differs.
    """
    if (self.enabled, self.ready, self.task_count) != (enabled, ready, task_count):
        self.enabled = enabled
        self.ready = ready
        self.task_count = task_count
        self.save()

    return self.export()

This code

force_list({'a': True, 'b': True}.keys())

should return ["a", "b"] but instead it returns [dict_keys(['a', 'b'])] (list of dict_items).

This only happens on Python 3.

File kobo/worker/main/py makes use of a private variable which is not available on Python 3.

log_level = logging._levelNames.get(conf.get("LOG_LEVEL", "DEBUG").upper())

This causes the error bellow

AttributeError: 'module' object has no attribute '_levelNames'

Just two questions as per title. Thank you.

It would be convenient if it was possible to concatenate lists in configuration files. Particularly if one file is included in another, it would be nice to be able to define common base in one file and append something in another.

Proposed syntax:

a = [1, 2, 3]

a = a + [4]

There is already a precedent for having operators in the configuration: it supports string interpolation with % operator.

The pkgset.FileCache object behaves like a dict, but misses __setitem__ method. This means that a user needs to access file_cache member. This leaks abstraction details.

Hi.

Could you please make a new release? There is a lot of changes (Python 3 compatibility for example) and it would be nice to release a new version of RPM packages based on a new version of Kobo.

Thanks!

As I understand it, Kobo supports loading configuration values from a file:

And the extent of runtime configuration for a (global) profile is generally implemented through environment variables in a downstream client:

Users would have to write wrapper scripts to ensure the correct profile is set if using this approach.

Could configuration profiles be implemented as a CLI option? The option could look like this:

$ downstream-kobo-client --help
Usage: downstream-kobo-client <command> [args] [--help]

Options:
  -h, --help           show this help message and exit
  --username=USERNAME  specify user
  --password=PASSWORD  specify password
  --profile=PROFILE    specify profile

Which could run like this:

$ downstream-kobo-client --profile=${PROFILE}

And this in turn could look in a designated configuration folder like so:

/etc/downstream-kobo-client/${PROFILE}.conf

Class SafeCookieTransport() which is in xmlrpc submodule uses old (2.7) interface of xmlrpclib library.
Following code snippet shows the bug:

import xmlrpc.client
from kobo.xmlrpc import SafeCookieTransport

url = 'https://errata.devel.redhat.com/errata/xmlrpc.cgi'

xmlrpc = xmlrpc.client.ServerProxy(url, transport=SafeCookieTransport())

print(xmlrpc.get_advisory_cdn_file_list('RHSA-2018:2942-03'))

Here is the output:

Traceback (most recent call last):
  File "xml_rpc_test.py", line 9, in <module>
    print(xmlrpc.get_advisory_cdn_file_list('RHSA-2018:2942-03'))
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/home/araszka/.local/lib/python3.6/site-packages/kobo/xmlrpc.py", line 450, in _single_request
    self.send_request(h, handler, request_body)
  File "/home/araszka/.local/lib/python3.6/site-packages/kobo/xmlrpc.py", line 557, in send_request
    return xmlrpclib.SafeTransport.send_request(self, connection, handler, request_body)
TypeError: send_request() missing 1 required positional argument: 'debug'

New version of xmlrpc library for python 3 and its interface have been updated and 'debug' argument has been added to send_request() function.
https://github.com/python/cpython/blob/3.7/Lib/xmlrpc/client.py#L1266

In TaskManager class, here's the body of fork_task currently:

def fork_task(self, task_info):
    self.log_debug("Forking task %s" % self._task_str(task_info))

    pid = os.fork()
    if pid:
        self.log_info("Task forked %s: pid=%s" % (self._task_str(task_info), pid))
        return pid

    # in no circumstance should we return after the fork
    # nor should any exceptions propagate past here
    try:
        # set process group
        os.setpgrp()

        # set a do-nothing handler for sigusr2
        # do not use signal.signal(signal.SIGUSR2, signal.SIG_IGN) - it completely masks interrups !!!
        signal.signal(signal.SIGUSR2, lambda *args: None)

        # set a default handler for SIGTERM
        signal.signal(signal.SIGTERM, signal.SIG_DFL)

        # run the task
        self.run_task(task_info)
    finally:
        # die
        os._exit(os.EX_OK)

If run_task raises any exception, it's never logged, and the process exits with 0 exit code giving no indication of an error.

It seems that any exceptions here should be logged, so that problems from run_task (e.g. failed XML-RPC calls to hub) leave some evidence behind.

Current Travis builds run in Ubuntu Trusty, where Koji is not available so tests which depend on RPM and Koji are skipped.

Possibilities:

• Use docker to run tests in Travis https://docs.travis-ci.com/user/docker/
• Wait until Travis switches to Ubuntu Xenial where Koji might be somehow available

Have a look at this error handling logic in kobo/django/xmlrpc/dispatcher.py, _marshaled_dispatch method:

        try:
            if dispatch_method is not None:
                response = dispatch_method(method, params)
            else:
                response = self._dispatch(method, params)
            # wrap response in a singleton tuple
            response = (response,)
            response = xmlrpclib.dumps(response, methodresponse=1, allow_none=self.allow_none, encoding=self.encoding)

        except xmlrpclib.Fault as fault:
            response = xmlrpclib.dumps(fault, allow_none=self.allow_none, encoding=self.encoding)

        except:
            # report exception back to server
            if settings.DEBUG:
                from kobo.tback import Traceback
                response = xmlrpclib.dumps(
                    xmlrpclib.Fault(1, u"%s" % Traceback().get_traceback()),
                    allow_none=self.allow_none, encoding=self.encoding)
            else:
                response = xmlrpclib.dumps(
                    xmlrpclib.Fault(1, "%s: %s" % (sys.exc_type.__name__, sys.exc_info()[1])),
                    allow_none=self.allow_none, encoding=self.encoding)

Consider what happens if the XML-RPC method raises any unexpected exception. Except for DEBUG mode, the exception details (traceback) will be discarded. This can make it virtually impossible to identify which code raised an exception. Even in DEBUG mode, the exception info is only passed to the client, so the root cause of an error on the server requires looking into logs on the client.

This should be improved so that unexpected exceptions during XML-RPC methods are always logged on the server.

The code for interrupt_tasks and timeout_tasks are very similar.

def timeout_tasks(request, task_list):
    result = True
    for task_id in task_list:
        task = Task.objects.get_and_verify(task_id=task_id, worker=request.worker)
        if task:
            try:
                task.timeout_task(recursive=True)
            except:
                raise
                result = False
    return result

The strange part is the result variable, this function only returns True, if for some reason task.timeout_task raises an error result = False is never reached because it re-raises the error.

Some options available for fixing this:

• Do not return anything and re-raise
• Do not re-raise and return False
• Do not re-raise and return the actual count of tasks changed. If the input was 10 tasks but only 5 were interrupted return the number 5.

worker log:

2016-04-19 09:46:42 [INFO    ] Waking up task #24993 [...].
2016-04-19 09:46:42 [INFO    ] Task #24993 exited with status 0
2016-04-19 09:46:42 [INFO    ] Task has finished: 24993
2016-04-19 09:46:42 [INFO    ] kill_group: Process (pgrp 9253) exited
2016-04-19 09:47:03 [WARNING ] Closing interrupted tasks: [24993]
2016-04-19 09:47:03 [ERROR   ] <Fault 1: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/kobo/django/xmlrpc/dispatcher.py", line 95, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 418, in _dispatch
    return func(*params)
  File "/usr/lib/python2.6/site-packages/kobo/hub/decorators.py", line 24, in _new_func
    return func(request, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/kobo/hub/xmlrpc/worker.py", line 111, in interrupt_tasks
    task.interrupt_task(recursive=True)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/kobo/hub/models.py", line 737, in interrupt_task
    task.interrupt_task(recursive=True)
  File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line 371, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/kobo/hub/models.py", line 733, in interrupt_task
    raise Exception("Cannot interrupt task %d, state is %s" % (self.id, self.state))
Exception: Cannot interrupt task 24994, state is 3

At 9:42, process of 24993 finished with 0 so kobo correctly closed the task and cleaned up:

2016-04-19 09:46:42 [INFO    ] Task #24993 exited with status 0
2016-04-19 09:46:42 [INFO    ] Task has finished: 24993
2016-04-19 09:46:42 [INFO    ] kill_group: Process (pgrp 9253) exited

I believe that here comes the race condition. In next cycle, we can see

2016-04-19 09:47:03 [WARNING ] Closing interrupted tasks: [24993]

which is incorrect since task is already closed. When checking code:

if task_info["state"] == TASK_STATES["OPEN"] and task_info["id"] not in self.pid_dict:
    # an interrupted task appears to be open, but running task manager doesn't track it in it's pid list
    # this happens after a power outage, for example
    interrupted_list.append(task_info["id"])
    finished_tasks.add(task_info["id"])
    continue

this condition is evaluated as true which means that the task is still open even though it's suppose to be closed (because it doesn't have an entry in pid_dict)

So, why it's not closed? Looking at code, this is how tasks are supposed to be closed

try:
    task.run()
except ShutdownException:
    ...

thread.stop()
if failed:
    hub.worker.fail_task(task.task_id, task.result)
else:
    hub.worker.close_task(task.task_id, task.result)

Since that code contains very little logging, it's hard to figure out what could possibly go wrong.

_utf8_chunk and _tail fails raises an error:

 TypeError: memoryview: a bytes-like object is required, not 'str'

These functions also have different returns on Python 2 and 3. For py2 it returns str for py3 it returns bytes, this must be due to unicode errors.

StateEnumField is broken:

ERROR: test_to_python (__main__.TestStateEnumField)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_types.py", line 206, in setUp
    self.field = StateEnumField(self.state_enum, default='NEW')
  File "/home/dmach/github/kobo-github/kobo/django/fields.py", line 48, in __init__
    super(StateEnumField, self).__init__(*args, **kwargs)
  File "/home/dmach/github/kobo-github/devel/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 162, in __init__
    self.choices = choices or []
TypeError: unbound method _set_choices() must be called with StateChoiceFormField instance as first argument (got StateEnumField instance instead)

Hello,

I'd expect that if rpmlib.parse_nvra() would call strip() or any other tool to get rid of possible newline character on the end of string.

I expect that "\n" isn't valid arch anyway.

Execution:
lkocman@xxx> python -c 'from kobo.shortcuts import run; import kobo.rpmlib; print kobo.rpmlib.parse_nvra(run("rpm -q rhsm-tools")[1])'

Actual results:
{'src': False, 'name': 'rhsm-tools', 'epoch': '', 'version': '1.26', 'release': '1.el6eng', 'arch': 'noarch\n'}
lkocman@rcm-dev:tmp>

Expected results (execution is enforcing .strip() on rpm output)
lkocman@rcm-dev:tmp> python -c 'from kobo.shortcuts import run; import kobo.rpmlib; print kobo.rpmlib.parse_nvra(run("rpm -q rhsm-tools")[1].strip())'
{'src': False, 'name': 'rhsm-tools', 'epoch': '', 'version': '1.26', 'release': '1.el6eng', 'arch': 'noarch'}
lkocman@rcm-dev:tmp>

From kobo hub, if you request a log file which doesn't exist and one of the following is true:

• file ends in .htm
• file ends in .html
• format=raw

Then you'll get a successful but empty response like this:

$ curl -v 'http://pub-dev-hub/pub/task/56/log/notexist.htm'
* About to connect() to pub-dev-hub port 80 (#0)
*   Trying 172.17.0.3...
* Connected to pub-dev-hub (172.17.0.3) port 80 (#0)
> GET /pub/task/56/log/notexist.htm HTTP/1.1
> User-Agent: curl/7.29.0
> Host: pub-dev-hub
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Mon, 04 Sep 2017 23:46:54 GMT
< Server: Apache/2.2.15 (Red Hat)
< Vary: Cookie
< Content-Length: 0
< Connection: close
< Content-Type: text/html; charset=UTF-8
< 
* Closing connection 0

Expected behavior: should get a 404 response.

The current latest version of kobo on pypi is 0.5.2 which has some incompatible code with python3, could you upload the latest version to pypi?

We can install the latest version via pip with github link, but it's a little inconvenient to maintain the setup.py to parsing the requirements in our projects.

Though not documented, the intended behavior of kobo.shortcuts.run seems to be:

• it passes through any keyword arguments supported by Popen
• for stdout, it returns whatever type Popen would return (which depends on those keyword arguments)

But on Python 3, this is broken in every text mode case other than universal_newlines. This is because the code has special cases to deal with the universal_newlines keyword argument, but in Python 3 this is one of only several arguments for enabling text mode (and it's not the preferred one).

Steps to reproduce

(on Python 3)

>>> from kobo.shortcuts import run

>>> run('echo Hello')
(0, b'Hello\n')

>>> run('echo Hello', universal_newlines=True)
(0, 'Hello\n')

>>> run('echo Hello', text=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rmcgover/src/kobo/kobo/shortcuts.py", line 338, in run
    lines = out_reader.read(buffer_size)
  File "/home/rmcgover/dev/python3/lib64/python3.7/codecs.py", line 500, in read
    data = self.bytebuffer + newdata
TypeError: can't concat str to bytes

>>> run('echo Hello', encoding='utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rmcgover/src/kobo/kobo/shortcuts.py", line 338, in run
    lines = out_reader.read(buffer_size)
  File "/home/rmcgover/dev/python3/lib64/python3.7/codecs.py", line 500, in read
    data = self.bytebuffer + newdata
TypeError: can't concat str to bytes

>>> run('echo Hello', errors='strict')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/rmcgover/src/kobo/kobo/shortcuts.py", line 338, in run
    lines = out_reader.read(buffer_size)
  File "/home/rmcgover/dev/python3/lib64/python3.7/codecs.py", line 500, in read
    data = self.bytebuffer + newdata
TypeError: can't concat str to bytes

Actual behavior

In cases 3-5 above, exceptions are raised.

Expected behavior

In cases 3-5 above, output is returned as a str.

sys.exc_type has been deprecated a long time ago and was removed in Python 3 which causes this error:

                response = xmlrpclib.dumps(
>                   xmlrpclib.Fault(1, "%s: %s" % (sys.exc_type.__name__, sys.exc_info()[1])),
                    allow_none=self.allow_none, encoding=self.encoding)
E               AttributeError: module 'sys' has no attribute 'exc_type'

venv3/lib/python3.7/site-packages/kobo/django/xmlrpc/dispatcher.py:112: AttributeError

With Django 1.8, starting with an empty database, with INSTALLED_APPS containing 'django.contrib.auth' and 'kobo.django.auth', manage.py migrate will fail with:

ERRORS:
auth.User.groups: (fields.E304) Reverse accessor for 'User.groups' clashes with reverse accessor for 'User.groups'.
HINT: Add or change a related_name argument to the definition for 'User.groups' or 'User.groups'.
auth.User.user_permissions: (fields.E304) Reverse accessor for 'User.user_permissions' clashes with reverse accessor for 'User.user_permissions'.
HINT: Add or change a related_name argument to the definition for 'User.user_permissions' or 'User.user_permissions'.
kobo_auth.User.groups: (fields.E304) Reverse accessor for 'User.groups' clashes with reverse accessor for 'User.groups'.
HINT: Add or change a related_name argument to the definition for 'User.groups' or 'User.groups'.
kobo_auth.User.user_permissions: (fields.E304) Reverse accessor for 'User.user_permissions' clashes with reverse accessor for 'User.user_permissions'.
HINT: Add or change a related_name argument to the definition for 'User.user_permissions' or 'User.user_permissions'.

It is possible to silence this by use of SILENCED_SYSTEM_CHECKS, but then migrations will fail because both apps try to create the table for the Users model.

It's unclear how to make this work; the migration in kobo.django.auth doesn't create related tables such as Groups. So, disabling the migrations from django.contrib.auth doesn't work (the User.username field will be left at the default 30 chars, defeating the purpose of kobo_auth's User) and disabling the migrations from django.contrib.auth doesn't work (kobo's migrations will fail to create necessary tables).

This traceback was thrown when running pungi-4.1.21, which switched to python3:

Traceback (most recent call last):
  File "/usr/bin/pungi-koji", line 468, in <module>
    main()
  File "/usr/bin/pungi-koji", line 254, in main
    run_compose(compose, create_latest_link=create_latest_link, latest_link_status=latest_link_status)
  File "/usr/bin/pungi-koji", line 362, in run_compose
    init_phase.start()
  File "/usr/lib/python3.6/site-packages/pungi/phases/base.py", line 64, in start
    self.run()
  File "/usr/lib/python3.6/site-packages/pungi/phases/init.py", line 41, in run
    write_global_comps(self.compose)
  File "/usr/lib/python3.6/site-packages/pungi/phases/init.py", line 84, in write_global_comps
    get_file_from_scm(scm_dict, tmp_dir, logger=compose._logger)
  File "/usr/lib/python3.6/site-packages/pungi/wrappers/scm.py", line 249, in get_file_from_scm
    scm.export_file(scm_repo, i, scm_branch=scm_branch, target_dir=tmp_dir)
  File "/usr/lib/python3.6/site-packages/pungi/wrappers/scm.py", line 147, in export_file
    self.run_process_command(tmp_dir)
  File "/usr/lib/python3.6/site-packages/pungi/wrappers/scm.py", line 48, in run_process_command
    universal_newlines=True)
  File "h", line 326, in run
    lines = proc.stdout.read(buffer_size)
  File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
Frame decode in /usr/lib64/python3.6/encodings/ascii.py at line 26
<CODE>

We should add some Django 1.11 configuration to the Travis CI. This would be needed to ensure the changes for issue #44 are working.

In scope:

• There's at least one environment in tox.ini targeting Django 1.11
• The environment is covered by Travis CI
• The overall result of the test run for that new environment is successful
  • (This might require setting up some tests to xfail or skip.)
  • (If xfail is used, best to make them strict xfails, so that tests must be updated as bugs are fixed)

Not in scope:

• Fixing all bugs or test xfails/skips needed for Django 1.11

Task.__lock fails when running unit tests on Python 3.

cursor.rowcount always returns -1 on Python 3, on Python 2 it returned the actual rows count affected. Unit tests are using SQLite, but this problem seems to happen with other drivers.

The biggest problem here is that cursor.rowcount is used to raise or not exceptions. We need to fix it or use Django ORM instead of a cursor.

Worker.timeout_task interrupt subtasks if recursive is True and the task has subtasks.
The subtasks are set to INTERRUPTED but they should be set to TIMEOUT.

This is the current code of timeout_task:

    def timeout_task(self, recursive=True):
        """Set the task state to timeout."""
        try:
            self.__lock(self.worker_id, new_state=TASK_STATES["TIMEOUT"], initial_states=(TASK_STATES["OPEN"], ))
        except (MultipleObjectsReturned, ObjectDoesNotExist):
            raise Exception("Cannot interrupt task %d, state is %s" % (self.id, self.get_state_display()))

        if recursive:
            for task in self.subtasks():
                task.interrupt_task(recursive=True)
        self.logs.gzip_logs()

It's a very silly mistake, looks like it was copy and paste and someone forgot to change it.

There is no kobo-0.5.1 git tag despite such version of kobo was released in Fedora. It is also not clear to me why the last version is tagged 0.5.2 instead of kobo-0.5.2 as all the previous releases. Please keep the release tags consistent.

When can the python-3-dev branch merge to master and land in a point release?

It happens when a running task has a child which is already in closed state and child should be interrupted. Interrupt happens if new patch is added to scan, so it tries to interrupt the parent repeatedly. However, the parent is trying to shut down the child which has already finished (esp. in closed state) and query is looking for opened task, which throws an exception. I think the best idea is to add condition when the query finds no results, and if the state of the task is in finished state, do not throw exception (maybe write something to log).

set_task_weight from kobo/hub/xmlrpc/worker.py fails because it uses task.setWeight which is not implemented on Task model.

When a task is running in a kobo worker, it's the responsibility of the LoggingThread to send log messages back to the hub.

This is done by the following loop:

    def run(self):
        """Send queue content to hub."""
        while self._running or not self._queue.empty() or self._send_data:
            if self._queue.empty():
                self._event.wait(5)

            self._event.clear()
            while True:
                try:
                    self._send_data += self._queue.get_nowait()
                except six.moves.queue.Empty:
                    break

            if not self._send_data:
                continue

            now = int(time.time())
            if self._running and len(self._send_data) < 1200 and now - self._send_time < 5:
                continue

            if isinstance(self._send_data, six.text_type):
                self._send_data = self._send_data.encode('utf-8')

            try:
                self._hub.upload_task_log(StringIO(self._send_data), self._task_id, "stdout.log", append=True)
                self._send_time = now
                self._send_data = ""
            except Fault:
                continue

Problem: if self._hub.upload_task_log raises an error (other than an XML-RPC Fault), then the logging thread simply stops. However, the main thread of the task being executed doesn't necessarily stop.

The end result of this is that if the kobo hub has a temporary outage while a task is in progress, that task might continue executing but all logs after the outage would be lost.

It would be better if the LoggingThread kept retrying the task log uploads, for as long as the task's main thread is alive.

Steps to reproduce

• Start a long-running task
• Take the kobo hub down temporarily during task execution

Actual behavior

• Task may continue running but with no further log messages uploaded

Expected behavior

• If task continues running, then it resumes uploading logs after kobo hub is restored.

Function main_loop has this code:

try:
    log_file = conf.get("LOG_FILE", None)
    logger = logging.Logger("TaskManager")
    logger.setLevel(logging.DEBUG)
    if log_file:
        log_level = logging._levelNames.get(conf.get("LOG_LEVEL", "DEBUG").upper())
        kobo.log.add_rotating_file_logger(logger, log_file, log_level=log_level)

    tm = TaskManager(conf=conf, logger=logger)
except Exception as ex:
    raise
    sys.stderr.write("Error initializing TaskManager: %s\n" % ex)
    sys.exit(1)

If some exception occurs it's re-raised instead of executing the code above the raise line. This code is unreachable and it's difficult to know what should be the behavior.

If kobo.shortcuts.run is called with a buffer_size other than -1, it can generate useless additional whitespace.

Can be trivially reproduced like:

$ python
Python 2.7.5 (default, Aug  2 2016, 04:20:16) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from kobo.shortcuts import run
>>> x = run("echo Hello world", stdout=True, buffer_size=2)
He ll o  wo rl d

The reason is that run internally uses the print method, which is documented as:

A space is written before each object is (converted and) written, unless the output system believes it is positioned at the beginning of a line.

In the above example, there's an extra space introduced every 2 characters since run will loop, reading a buffer of size 2, then printing it.

Since the intent is to pass through the stdout from the process unmodified, this should be changed to use write rather than print.

There are many modules in this library, some of which bring lots of dependencies like koji or django. It would be nice to split the library into multiple smaller libraries so that people can install only what they need.

This is already done to some level in Fedora RPM packaging with subpackages:

kobo
    kobo-admin
    python3-kobo
    python3-kobo-admin
    python3-kobo-client
    python3-kobo-django
    python3-kobo-hub
    python3-kobo-rpmlib
    python3-kobo-worker

For example in Pungi we use only these modules: log, pkgset, rpmlib, shortcuts and threads. No Django needed, yet when installing with pip during development, it gets pulled in.

TaskManager has a custom implementation for get_query_set that returns only non-achieved Tasks.

The problem is, it does not filter at all in any version after 1.6 because this function was renamed as it shows here.

This means that if we call Task.objects.all() it does not filter achieved tasks and this can result in errors.

If we rename the funcion it could work, but it needs to be tested.

Currently, we release kobo to PyPI as a single egg containing all kobo components. This doesn't make much sense since kobo consists of at least the following components:

• hub
• worker
• client
• utility classes

These are generally deployed to different environments and it doesn't make sense to have them all packaged together as one "all or nothing" egg.

It means that we're discouraged from defining any requirements in the kobo egg, since anyone using that may be forced to pull in a lot more dependencies than they need. For example, somebody using some of the kobo utility classes shouldn't have to install Django; and it should not be necessary to install Django to (for example) create a container image for a kobo worker via pip.

We should instead be shipping separate eggs, probably at least 4 (plus one for backwards-compat), and these should define their dependencies correctly.

For example, we could release eggs:

• kobo-hub
• kobo-worker
• kobo-client
• kobo-utils
• kobo (deprecated egg which simply depends on the other eggs)

Kobo fails on Python 3 due missing smart_unicode function.

    from django.conf import settings
    from django.core.exceptions import ImproperlyConfigured
    from django.core.urlresolvers import reverse
    from django.utils.safestring import mark_safe
>   from django.utils.encoding import smart_unicode
E   ImportError: cannot import name 'smart_unicode' from 'django.utils.encoding'

venv3/lib/python3.7/site-packages/kobo/django/menu/__init__.py:95: ImportError

Currently if you try to set key starting with underscore, KeyError is raised.
https://github.com/release-engineering/kobo/blob/master/kobo/conf.py#L92-L95

This is confusing for users encountering this exception, as the documentation says:

Raised when a mapping (dictionary) key is not found in the set of existing keys.

I assume there is some reasoning why these keys are rejected, but I suggest using different exception.

If get_next_task is called and TaskManager is currently locked and has awaited tasks it will fail to run the awaited tasks if they are not exclusive because of missing extra fields.

The reason is that take_task crash if the task is not exclusive because it requires extra fields from arch and channel if flat=True on get_awaited_tasks these extra fields are not provided.

Code that requires extra fields

Getting the following error when I do python manage.py migrate

Traceback (most recent call last):
  File "manage.py", line 13, in <module>
    execute_from_command_line(sys.argv)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
    output = self.handle(*args, **options)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 82, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
    self.build_graph()
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/loader.py", line 206, in build_graph
    self.load_disk()
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/loader.py", line 108, in load_disk
    migration_module = import_module(migration_path)
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/kobo/django/xmlrpc/migrations/0001_initial.py", line 8, in <module>
    class Migration(migrations.Migration):
  File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/kobo/django/xmlrpc/migrations/0001_initial.py", line 22, in Migration
    ('user', models.ForeignKey(blank=True, to=settings.AUTH_USER_MODEL, null=True)),
TypeError: __init__() missing 1 required positional argument: 'on_delete'

I was able to apply the migrations after editing /kobo/django/xmlrpc/migrations/0001_initial.py by adding on_delete=models.CASCADE to line number 22.

Is there any work around without editing this?

Passing a Context or a RequestContext is still possible when the template is loaded by a DjangoTemplates backend but it’s deprecated and isn't supported since Django 1.10.

kobo depends on python-krbV, which is python2-only and dead. We need to use something else to be able to use dependent functionality on Python 3.

Django 1.11 is next version with LTS and the last version with Python 2.7 support so Kobo should support it. Also, Django 1.8 EOL will be in April 2018.

https://www.djangoproject.com/download/

Related to: #2

kobo should provide a command for resetting the priority of an already-scheduled task. The use-case is to adjust the relative priorities of tasks so that desired tasks are executed first, when the queue is full.

We already have "create-task". Suggestion: this should be a new "modify-tasks" counterpart. Only certain fields of a task could be modified; initially, at least the priority.

Example:

Usage: <command> modify-tasks task_id [task_id...]

Options:
  --priority=PRIORITY     priority

LoggingThread fails on Python 3 due to encoding errors.

It fails on Python 3 because string.encode returns a bytes-like object and StringIO should receive an str object.

Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/kobo/kobo/worker/logger.py", line 54, in run
    self._hub.upload_task_log(StringIO(self._send_data), self._task_id, "stdout.log", append=True)
TypeError: initial_value must be str or None, not bytes

Please update kobo release on PyPI to match upstream one.

Latest kobo release on PyPI: 0.3.8
Latest upstream release: 0.5.2

The method cancel_subtasks has the following line:

result &= task.cancel_task()

cancel_task does not return a boolean, actually, it does not return anything.

This causes the error:

TypeError: unsupported operand type(s) for &=: 'bool' and 'NoneType'

Function shutdown_worker from kobo/hub/xmlrpc/client.py allows passing None as the worker_name parameter.

If my guess is correct, this means any worker can catch the "shutdown task" and be killed which is not good.

If the command fails, an error message is printed to stderr. If a program using the function wants to run in quiet mode, there is no easy way to silence it.

The message could for example only be printed when can_fail is True. Otherwise an exception is raised with the same error message, so calling code can print it as it sees fit.