Looking at setup.py it seems like the dependencies are only specified using a lower limit, something which is bound to cause reproducibility issues in the future:

install_requires=[
        'PyYAML>=3.0',
        'distro>=1.0.0',
        'jsonschema>=2.5.0',
    ],

When you install a certain kas version you never know if it's going to be compatible with its dependencies, something it will inevitably not be as time goes on. This should be changed to a reasonable upper bound which depends on how the dependencies manage their versions.

Apparently shellcheck is not executed on container-entrypoint. I'll open a MR closing this issue if I have the time later on.

(side note: linking issues to MRs is a quite neat thing for the workflow as it's all happening on the same platform. TBH I really don't understand why you would rather make users use google for that... I mean... GOOGLE?)

Kas should have a default value for KAS_REPO_REF_DIR and clone a repository there itself as a transparent step before checking out a refspec and eventually applying patches, because 1) it requires its own naming convention for these cached repos and 2) it knows the origin URLs of the said repos.

This would allow us to share KAS_REPO_REF_DIR alongside SSTATE_DIR and DL_DIR between multiple builds.

Cc: @rossburton

I am behind a proxy which blocks git:// so I use a .gitconfig to replace git:// with https://, I tried to use kas but fell foul of this issue.

kas generates the local.conf and bblayers.conf in the build directory from the headers (local_conf_header and bblayers_conf_header) and other kas variables (machine, distro, and the layers in repos).

Is it possible to use the existing template configs in each layer with kas?
i.e. meta-xxx/conf/local.conf.sample and meta-xxx/conf/local.conf.sample, that are copied by oe-init-build-env if we set TEMPLATECONF=meta-xxx/conf
I cannot find any information how to do this in docs/userguide.rst.

I'm looking for the same functionality of 'repo manifest -r -o my-manifest.xml' of google repo in Kas.
Is there something to export all git hash stuff from the meta layers into a new exported YML file?

Hi,

thanks for your great work.

When I try to run the following command I get an error :

$ KAS_WORK_DIR=$PWD/tmp-build/ DL_DIR=$PWD/downloads ./kas-container shell test.yml -c "bitbake package-index"
usage: kas [-h] [--version] [-d] {build,checkout,for-all-repos,shell} ...
kas: error: unrecognized arguments: /repo/test.yml

How can I call a shell bitbake command with the docker container solution? With plain kas command it works as expected.

Strictly speaking, auto.conf is where automatically written configuration should go, not local.conf.

I was only thinking about this because of the possibility of using kas as a starting point for builds. If it wrote to auto.conf then re-running kas could just rewrite the managed configuration, leaving local.conf for the user to further alter.

Hello there!

I'd like my bblayers.conf to include a layer which lives in a subdirectory of my repository. However, the layer gets referenced in bblayers.conf in an absolute way.

In kas.yml I define the layer to live in subdirectory meta-manwag of the repository:

repos:
  meta-manwag:
    path: meta-manwag

But the generated bblayers.conf turns it into an absolute path.

BBLAYERS ?= " \
    [...]
    /home/manuel/bora-proj/poky/meta-poky \
    /home/manuel/bora-proj/poky/meta-yocto-bsp \
    meta-manwag"

The documentation says the following regarding the path keyword:

The path where the repository is stored. If the url and path is missing, the repository where the current configuration file is located is defined. If the url is missing and the path defined, this entry references the directory the path points to. If the url as well as the path is defined, the path is used to overwrite the checkout directory, that defaults to kas_work_dir + repo.name. In case of a relative path name kas_work_dir is prepended.

As per documentation KAS_WORK_DIR shall default to the current work directory. I tried setting it explicitely (KAS_WORK_DIR=$( pwd) kas shell my.yml), but that didn't help.

This problem also shows with ghcr.io/siemens/kas/kas:2.3.3. Strangely, I have some other kas container with digest sha256:7772f9bfe9db3e61effdb8ef76dbd0b2d778668dc6d1581cb76cf0363046d4c2 in my repository, which works just fine. I yet have to find a way to diff it with ghcr.io/siemens/kas/kas:2.3.3.

If you want, you can recreate this problem in the project it occured:

1. git clone https://gitlab.com/manuel_wagesreither/bora-proj.git
2. kas shell kas/bora-qemux86-64.yml

Thank you for creating kas! Managing layers with google repo tool was horrible.
Best regards,
Manuel

When I try to execute kas with python3.7, the following error message appears:

Traceback (most recent call last):
  File "/usr/local/bin/kas", line 7, in <module>
    from kas.kas import main
  File "/usr/local/lib/python3.7/site-packages/kas/kas.py", line 51, in <module>
    from .libkas import kasplugin
  File "/usr/local/lib/python3.7/site-packages/kas/libkas.py", line 168
    task = asyncio.async(repo.fetch_async())
                       ^
SyntaxError: invalid syntax

However, when I execute kas with python3.5 it runs without problems:
python3.5 -m kas

When trying to build a isar based project, I noticed, that kas-docker has a problem with git submodules.

When executing the same kas-docker build in a "normal" git repository, everything works fine. Executing the command in a git submodule fails to detect the proper git-root:

$ kas-docker --isar -v build kas/boards/some-project.yml
+ mkdir -p /home/user/isar/isar
+ docker run -v /home/user/isar/isar:/repo:ro -v /home/user/isar/isar:/work:rw --workdir=/work -e USER_ID=1000 -e GROUP_ID=1000 --rm -t -i -e SHELL='/bin/bash' -e TERM='xterm-256color' --cap-add=SYS_ADMIN --cap-add=MKNOD --privileged --device /dev/loop20 kasproject/kas-isar build /repo/kas/boards/some-project.yml
2019-08-13 11:10:04 - INFO     - kas 1.0 started
2019-08-13 11:10:04 - INFO     - /repo/kas$ git rev-parse --show-toplevel
2019-08-13 11:10:04 - INFO     - /repo/kas$ hg root
2019-08-13 11:10:04 - INFO     - /repo/kas/board$ git rev-parse --show-toplevel
2019-08-13 11:10:04 - INFO     - /repo/kas/board$ hg root
2019-08-13 11:10:04 - INFO     - /repo/kas$ git rev-parse --show-toplevel
2019-08-13 11:10:04 - INFO     - /repo/kas$ hg root
2019-08-13 11:10:04 - INFO     - Using /repo/kas as root for repository isar
2019-08-13 11:10:04 - INFO     - /work/isar$ git cat-file -t xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

git rev-parse --show-toplevel command cannot work with reverence in the .git file, since the top-level repository is not mounted inside the docker container.

Why does kas require the git rev-parse --show-toplevel in the project root folder? It seems unnecessary for proper cloning of the meta-layers defined in the kas-file.

Hey,
I'm trying to set an environment variable from the command line. In one of my recipes I set:

WANT_TARGET_BRANCH ?= master

How can I set that variable when calling kas? Would come very handy in my CI pipeline.
I tried something like

WANT_TARGET_BRANCH=X bash -c 'kas shell dev.yml -c "bitbake my_tool"'

and

kas shell dev.yml -c "WANT_TARGET_BRANCH=X bitbake my_tool"

Neither was working so far.
Best

Hey there,

I'm super happy with how kas brings a lot more structure to my build system. However, I'm stuck now at about 60% build progress with the following exception which I do not get when not using my kas configuration.
I was wondering if you had a clue if I forgot to set some variable or where that might come from

NOTE: Using umask 0o002 (not 22) for sstate packaging
DEBUG: Executing python function sstate_hardcode_path
NOTE: Removing hardcoded paths from sstate package: 'find /home/pyrocko/yocto/qs-kas/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/kbd/2.2.0-r0/sstate-build-package/ \( -name "*.la" -o -name "*-config" -o -name "*_config" -o -name "postinst-*" \) -type f | xargs grep -l -e '/home/pyrocko/yocto/qs-kas/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/kbd/2.2.0-r0/recipe-sysroot' -e '/home/pyrocko/yocto/qs-kas/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/kbd/2.2.0-r0/recipe-sysroot-native' -e '/home/pyrocko/yocto/qs-kas/build/tmp/hosttools' | tee /home/pyrocko/yocto/qs-kas/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/kbd/2.2.0-r0/sstate-build-package/fixmepath | xargs --no-run-if-empty sed -i -e 's:/home/pyrocko/yocto/qs-kas/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/kbd/2.2.0-r0/recipe-sysroot-native:FIXMESTAGINGDIRHOST:g' -e 's:/home/pyrocko/yocto/qs-kas/build/tmp/work/arm1176jzfshf-vfp-poky-linux-gnueabi/kbd/2.2.0-r0/recipe-sysroot:FIXMESTAGINGDIRTARGET:g' -e 's:/home/pyrocko/yocto/qs-kas/build/tmp/hosttools:FIXME_HOSTTOOLS_DIR:g''
DEBUG: Python function sstate_hardcode_path finished
DEBUG: Executing python function sstate_report_unihash
WARNING: KeyError in .
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:sstate_report_unihash(d)
     0003:
File: '/home/pyrocko/yocto/qs-kas/poky/meta/classes/sstate.bbclass', lineno: 840, function: sstate_report_unihash
     0836:    report_unihash = getattr(bb.parse.siggen, 'report_unihash', None)
     0837:
     0838:    if report_unihash:
     0839:        ss = sstate_state_fromvars(d)
 *** 0840:        report_unihash(os.getcwd(), ss['task'], d)
     0841:}
     0842:
     0843:#
     0844:# Shell function to decompress and prepare a package for installation
File: '/home/pyrocko/yocto/qs-kas/poky/bitbake/lib/bb/siggen.py', lineno: 555, function: report_unihash
     0551:
     0552:            if "." in self.method:
     0553:                (module, method) = self.method.rsplit('.', 1)
     0554:                locs['method'] = getattr(importlib.import_module(module), method)
 *** 0555:                outhash = bb.utils.better_eval('method(path, sigfile, task, d)', locs)
     0556:            else:
     0557:                outhash = bb.utils.better_eval(self.method + '(path, sigfile, task, d)', locs)
     0558:
     0559:            try:
File: '/home/pyrocko/yocto/qs-kas/poky/bitbake/lib/bb/utils.py', lineno: 420, function: better_eval
     0416:    if extraglobals:
     0417:        ctx = copy.copy(ctx)
     0418:        for g in extraglobals:
     0419:            ctx[g] = extraglobals[g]
 *** 0420:    return eval(source, ctx, locals)
     0421:
     0422:@contextmanager
     0423:def fileslocked(files):
     0424:    """Context manager for locking and unlocking file locks."""
File: '<string>', lineno: 1, function: <module>
  File "<string>", line 1, in <module>

File: '/home/pyrocko/yocto/qs-kas/poky/meta/lib/oe/sstatesig.py', lineno: 591, function: OEOuthashBasic
     0587:
     0588:                update_hash("\n")
     0589:
     0590:            # Process this directory and all its child files
 *** 0591:            process(root)
     0592:            for f in files:
     0593:                if f == 'fixmepath':
     0594:                    continue
     0595:                process(os.path.join(root, f))
File: '/home/pyrocko/yocto/qs-kas/poky/meta/lib/oe/sstatesig.py', lineno: 551, function: process
     0547:                    add_perm(stat.S_IXOTH, 'x')
     0548:
     0549:                if include_owners:
     0550:                    try:
 *** 0551:                        update_hash(" %10s" % pwd.getpwuid(s.st_uid).pw_name)
     0552:                        update_hash(" %10s" % grp.getgrgid(s.st_gid).gr_name)
     0553:                    except KeyError:
     0554:                        bb.warn("KeyError in %s" % path)
     0555:                        raise
Exception: KeyError: 'getpwuid(): uid not found: 1000'

DEBUG: Python function sstate_report_unihash finished
DEBUG: Python function sstate_task_postfunc finished

Thanks for taking a glimpse! :)

Hi,
We want to create a download mirror as part of our kas build procedure. So we want to send --runonly=fetchbitbake argument as part of our kas build procedure. kas build <yml file> --runonly=fetch

But the command fails with kas: error: unrecognized arguments: --runonly=fetch
So tried with just any args (ex: -n). This also fails. So, Are we doing something wrong? What is the proper command to send extra arguments to the bitbake command?

Is there any way to create a custom build/conf/templateconf.cfg with kas?

When trying to run kas-docker I got the following error message:

| I: Running command: chroot /work/build/tmp/work/jailhouse-demo-arm64/isar-bootstrap-target/1.0-r0/rootfs /debootstrap/debootstrap --second-stage
| chroot: failed to run command ‘/debootstrap/debootstrap’: No such file or directory
| WARNING: exit code 127 from a shell command.
| 
ERROR: Task (mc:qemu-arm64-jailhouse-demo:/work/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) failed with exit code '1'

The cause of the error was not that debootstrap/debootstrap was actually missing, but that /proc/sys/fs/binfmt_misc/qemu-aarch64 contained the wrong path to the qemu-interpreter. Due to a different binfmt-support setup on the host.

I don't know wether this is actually solvable in kas-docker, but at least this ticket might help somone to find the cause of the error.

devtool modifies the local layers/config to add a layer which overwrites specific recipe sources to local checkouts. This is massively useful when developing code to run within a yocto-built environment. But re-entering the kas shell overwrites the config changes that devtool made. It'd be great if kas could do something to integrate with devtool more seamlessly

Clicking on documentation link in README points to wrong page.

• link: https://kas.readthedocs.io/
• redirect: https://kas.readthedocs.io/en/0.19.0/
• expected: https://kas.readthedocs.io/en/1.0/

Hi,

It seems that kas has hardcoded build as default name for the build directory inside working directory.
It is quite problematic if we want to build multiple images from the same set of layer repositories.

For example, I have custom layer with local changes and I want to test it with multiple target images (e.g. for different machines). I am unable to do that with kas.

Some solution would be to provide an extra cmd line argument --build where I could specify what should be the BitBake workdir. This way paths where repositories are cloned/stored and build dir are separated.

Or maybe there is some mechanism to do that already?
Thanks!

Please let us know how to use the multiconfig:* feature. Do we put conf/multiconfig/foo.conf in local layer (meta-custom)?

Hello,
I am unable to install kas using pip. pip install kas results in the following error:

» pip install kas
Collecting kas
  Using cached kas-2.6.tar.gz (25 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-necuf4_r/kas/setup.py'"'"'; __file__='"'"'/tmp/pip-install-necuf4_r/kas/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-necuf4_r/kas/pip-egg-info
         cwd: /tmp/pip-install-necuf4_r/kas/
    Complete output (9 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-necuf4_r/kas/setup.py", line 29, in <module>
        from kas import __version__
      File "/tmp/pip-install-necuf4_r/kas/kas/__init__.py", line 28, in <module>
        from .configschema import CONFIGSCHEMA
      File "/tmp/pip-install-necuf4_r/kas/kas/configschema.py", line 34, in <module>
        with open(os.path.join(cwd, 'schema-kas.json'), 'r') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-necuf4_r/kas/kas/schema-kas.json'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I'm using pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8) on Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-89-generic x86_64)

Installing an old version with pip install kas==2.5 worked fine.

Environment

Docker image: kasproject/kas:2.2
SSH private key in PEM RSA format

Expected behavior

running SSH_PRIVATE_KEY=/path/to/ssh/key kas build /mnt/kas-foo.yml in the docker container should add the SSH key to the ssh-agent using ssh-add.

Actual behavior

builder@e76a2ef8b3c4:/tmp$ SSH_PRIVATE_KEY=/path/to/ssh/key kas build /mnt/kas-foo.yml 
2020-09-09 11:08:45 - INFO     - kas 2.2 started
2020-09-09 11:08:45 - INFO     - /mnt$ git rev-parse --show-toplevel
2020-09-09 11:08:45 - INFO     - adding SSH key
2020-09-09 11:08:45 - ERROR    - failed to add ssh key: b'Error loading key "(stdin)": invalid format\n'

Additional information

Manually running ssh-agent -s and ssh-add /path/to/ssh/key works:

builder@e76a2ef8b3c4:/tmp$ ssh-agent -s
SSH_AUTH_SOCK=/tmp/ssh-Ds4LFCbiHB34/agent.122; export SSH_AUTH_SOCK;
SSH_AGENT_PID=123; export SSH_AGENT_PID;
echo Agent pid 123;
builder@e76a2ef8b3c4:/tmp$ ssh-add /path/to/ssh/key
Identity added: /path/to/ssh/key (/path/to/ssh/key)

Hi!
I am missing the feature to set the values of a configuration key in local_conf_header via an environment variable that is present in the host that calls kas.

A use case for this might be a version string or a pre-provision token that might change for different kas builds where the only difference might be a single value.

Committing patches creates a commit hash that does not exist in the source repo. Yocto buildhistory reports that hash which results in a loss of traceability. It would be interesting if kas provided an option to suppress commit of applied patches.

Hey,
the order of layers in BBLAYERS in the generated build directory deviates from the order in my yml configuration.

After removing bblayers.conf (just to see that I'm not mistaken) I run kas build mconfig.yml. This is BBLAYERS:

BBLAYERS ?= " \
    /home/pyrocko/yocto/qs-kas/meta-mender/meta-mender-core \
    /home/pyrocko/yocto/qs-kas/meta-mender/meta-mender-raspberrypi \
    /home/pyrocko/yocto/qs-kas/meta-openembedded/meta-networking \
    /home/pyrocko/yocto/qs-kas/meta-openembedded/meta-oe \
    /home/pyrocko/yocto/qs-kas/meta-openembedded/meta-python \
    /home/pyrocko/yocto/qs-kas/meta-openembedded/meta-webserver \
    /home/pyrocko/yocto/qs-kas/meta-quakesaver/. \
    /home/pyrocko/yocto/qs-kas/meta-quakesaver/meta-qs-raspberrypi \
    /home/pyrocko/yocto/qs-kas/meta-scipy \
    /home/pyrocko/yocto/qs-kas/poky/meta \
    /home/pyrocko/yocto/qs-kas/poky/meta-poky \
    /home/pyrocko/yocto/qs-kas/poky/meta-yocto-bsp"
 

And this is the order of my yml file:

  meta-quakesaver:
    layers:
      .:
      meta-qs-raspberrypi:

  meta-scipy:

  meta-mender:
    layers:
      meta-mender-core:
      meta-mender-raspberrypi:
      meta-mender-core:

  meta-openembedded:
    layers:
      meta-oe:
      meta-python:
      meta-networking:
      meta-webserver:

  poky:
    layers:
       meta:
       meta-poky:
       meta-yocto-bsp:

As you can see it's not reversed but just different. The actual config includes another config.yml but that one doesn't define any layers or repos. Any idea why that happens?

Hi,
I am using meta-raspberrypi that is using kas.

1. I need to add a layer to the build and modify the kas file how can I do that without modifying the yaml file in meta-raspberrypy?
2. as a developer I need to run a lot of bitbake commands is possible to launch the shell?

I saw that your docker images (https://hub.docker.com/r/kasproject/kas) have not been updated in a year which can be a security risk for people unsing that image. Would be better to have the docker build and deploy somehow hooked into your github actions.

2020-04-08 11:35:21 - INFO     - /work/isar-siemens$ git apply /repo/isar-patches/simatic-ipc-modules-uri.patch
2020-04-08 11:35:21 - INFO     - Patch applied. (patch path: /repo/isar-patches/simatic-ipc-modules-uri.patch, repo: isar-siemens, patch entry: simatic-ipc127e-static-ip)
:
2020-04-08 11:35:21 - INFO     - /work/isar-siemens$ git apply /repo/isar-patches/simatic-ipc127e-static-ip.patch
2020-04-08 11:35:21 - INFO     - Patch applied. (patch path: /repo/isar-patches/simatic-ipc127e-static-ip.patch, repo: isar-siemens, patch entry: simatic-ipc127e-static-ip)

my kas.yml:

repos:
  isar-siemens:
    url: [email protected]:ebsy/debian/isar-siemens.git
    patches:
      simatic-ipc-modules-uri:
        repo: my-repo
        path: isar-patches/simatic-ipc-modules-uri.patch
      simatic-ipc127e-static-ip:
        repo: my-repo
        path: isar-patches/simatic-ipc127e-static-ip.patch

I also observed that kas try under some unknown condition to re-apply the simatic-ipc-modules-uri.patch again and fails the startup, but I cannot reproduce this case currently...

Hey
I refactored a project of to encapsulate some logic into a sub layer (lets say meta-specific-config). Thus it looks a bit like this:

meta-x:
  recipes ...
  meta-specific-config:
    recipes ...

Unfortunately kas, doesn't include the base layer recipes but only the meta-specific-config in the bblayers if I explicitly set the layers attribute in the kas configuration.
What my bblayers should contain for a specific configuration is:

  meta-x \
  meta-x/meta-specific-config \

How can I achieve that?

It is better to have a manpage for kas command.

Thanks.

Hi,

I'm trying to build core-image-minimal on a Mac using kas-container with docker and docker-machine. However, when running a very minimal example, I'm getting the following error message:

$ ./kas-container build kas.yml
2020-12-19 22:35:49 - INFO     - kas 2.3.3 started
2020-12-19 22:35:49 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:35:49 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:35:49 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:35:49 - INFO     - /work$ git clone -q https://git.yoctoproject.org/git/poky /work/poky
2020-12-19 22:37:11 - INFO     - Repository poky cloned
2020-12-19 22:37:11 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:11 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:11 - INFO     - /work/poky$ git status -s
2020-12-19 22:37:12 - INFO     - /work/poky$ git rev-parse --verify -q origin/thud
2020-12-19 22:37:12 - INFO     - e52122a3e6912575ff401a4af6ac1bf3070092bc
2020-12-19 22:37:12 - INFO     - /work/poky$ git checkout -q e52122a3e6912575ff401a4af6ac1bf3070092bc -B thud
2020-12-19 22:37:17 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:18 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:18 - INFO     - /work/poky$ /tmp/tmp3plp2byg/get_bb_env /work/build
2020-12-19 22:37:18 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:18 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:18 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:18 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:18 - INFO     - /work/build$ /work/poky/bitbake/bin/bitbake -k -c build core-image-minimal
NOTE: Retrying server connection... (Traceback (most recent call last):
  File "/work/poky/bitbake/lib/bb/main.py", line 446, in setup_bitbake
    server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset)
  File "/work/poky/bitbake/lib/bb/server/process.py", line 402, in __init__
    self.sock.bind(os.path.basename(sockname))
PermissionError: [Errno 1] Operation not permitted
)
WARNING: /work/poky/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
  logger.info("Retrying server connection... (%s)" % traceback.format_exc())

NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection...
ERROR: Unable to connect to bitbake server, or start one
2020-12-19 22:37:39 - ERROR    - Command returned non-zero exit status 1

Contents of kas.yml:

# Every file needs to contain a header, that provides kas with information
# about the context of this file.
header:
  # The `version` entry in the header describes for which configuration
  # format version this file was created for. It is used by kas to figure
  # out if it is compatible with this file. The version is an integer that
  # is increased on every format change.
  version: 1
# The machine as it is written into the `local.conf` of bitbake.
machine: qemux86-64
# The distro name as it is written into the `local.conf` of bitbake.
distro: poky
repos:
  # This repo contains the kas.yml file - there is no need to fetch it again.
  # It's current revision will be used to perform the build.
  meta-custom:
  # Here we include a list of layers from the poky repository to the
  # bblayers.conf:
  poky:
    url: https://git.yoctoproject.org/git/poky
    refspec: thud
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:

I have a layer that uses LFS to provide some archives. When I use kas to checkout/build that layer my LFS pointers are not replaced by the actual files. If I only checkout the layer (using kas), navigate to it and then run git lfs pull on it, I get the archives and I can proceed as usual.
If I use bitbake shell and try to git lfs pull in that layer I get:

Skipping object checkout, Git LFS is not installed.

Am I doing something wrong here? Or does kas not work well with LFS?

Yocto Context

https://www.yoctoproject.org/docs/3.1.3/mega-manual/mega-manual.html#required-git-tar-python-and-gcc-versions describes how to install a tarball containing the minimal build tools (git, tar, python, gcc) to bootstrap a full yocto build using bitbake. Installing without root permissions is explicitly supported and seems to work pretty well from my experience. As described in the manual, the binaries from the buildtools tarball can be added to PATH by sourcing the provided environment file manually or for example by sourcing it from .profile.

Setup Context

My setup is this: I am provided an unprivileged user on a Linux system. I can do whatever I want in this user's home directory. I explicitly do not want to make any assumptions about the build tools provided by the host system and want to use Yocto's Pre-Built buildtools tarball to bootstrap the build system. This seems to be inline with the goals of kas and the goal of reproducible builds. Afterall, my build environment is unaffected by changes made to the host system.

• I use the python version provided by the tarball to setup a python venv in which I install kas via pip
• kas requires the git binary provided by the tarball to clone repositories

Problem

kas resets the PATH variable to /usr/sbin:/usr/bin:/sbin:/bin in https://github.com/siemens/kas/blob/2.2/kas/libkas.py#L231. This goes against my goals of using the pre-built buildtools tarball to achieve reproducible builds. Indeed, this results in kas attempting to use only system-provided tools and failing if some tools are missing on the system (although I explicitly installed them as per the Yocto manual and everything works fine without kas).

Related Reports

There is #27 which describes a similar problem and related discussion at https://groups.google.com/g/kas-devel/c/JXD7wQLu6Cg/m/u0q1uBDVBgAJ. The root cause of this issue is the same, however, my issue concerns setting up the dependencies for kas and bitbake in the first place.

Hey,
I usually now work within our proprietary meta layer within the downloaded kas sources. I update things, run kas build, if everything works fine I update the kas configuration with that latest commit.
To enhance that workflow I was wondering if a refspec=${AUTOREV} given that a branch (or if not fallback to master) is provided wouldn't be an option.
Sure, explicit is better than implicit. On the other hand we already know (and I love) the ${AUTOREV} feature from bitbake during development.

Best
Marius

Hi, first of all I wanted to thank you for this tool, it makes maintaing yocto projects a lot easier (a lot better than repo and custom init scripts)!

I am currently building a project which does not use poky, but openembedded-core instead. Everything seems to work fine, with the exception that bitbake needs to be cloned separately (poky includes it, but openembedded-core does not). If I add it to the kas-project.yml the repository is correctly cloned and kas executes normally, however when starting bitbake it complains that the folder "bitbake" (which is the clone destination of bitbake repository) does not contain the layer.conf file.

This is due to the fact that the bitbake repository is not a layer, still it is automatically included by kas in the generated bblayers.conf file.

This is confirmed by removing the bitbake repo from the kas-project.yml file after it has been first cloned: in this case it starts building with no complaining!

Is there a way to simply clone a repository but avoid including it in the bblayers.conf file?

The fetch() method for git will call git fetch --all ; this causes issues because I have 12+ forks added as remotes for one of my layers. It seems like this should be able to only fetch 'origin', which has been created by kas.

class GitRepo(RepoImpl):
    """
        Provides the git functionality for a Repo.
    """
...
    def fetch_cmd(self):
        return ['git', 'fetch', '--all']

Kas latest release is 2.3.3 but the latest Docker image is 2.2. Can a 2.3.3 docker image be published?

when using local git server over self signed https server this error happend:

server certificate verification failed. CAfile: none CRLfile: none

also using export GIT_SSL_NO_VERIFY=1 not solved the problem.

I'm trying to run this following command in order to get all the repositories used:
./kas-container.sh for-all-repos 'echo "REMOTE_URL: ${git ls-remote --get-url}"' {kas_config}

unfortunately this feature is not supported yet, I have tried to implement the following feature but it seems a bit more complicated then I have anticipated.
The first reason is that the parsing order is different on for-all-repos command

build - kas build [-c TASK] config
shell - kas shell [-c COMMAND] config
for-all-repos - kas for-all-repos config command

So some new flags/some hacks required to make them work together. or a bit deeper change rather than just adding a param.
The other issue I have encountered is that for some reason I can't run for-all-repos without the -d flag

Example:

docker run ghcr.io/siemens/kas/kas:2.4 for-all-repos

error: exec: "for-all-repos": executable file not found in $PATH

docker run ghcr.io/siemens/kas/kas:2.4 -d for-all-repos

usage: kas for-all-repos [-h] [--skip SKIP] [--force-checkout] [--update]
config command
kas for-all-repos: error: the following arguments are required: config, command

I would be happy to implement the following feature with some guildness. Thanks!

If you need to patch 3rd repos, e.g. poky, it is cumbersome to clone and provide the sources on your servers/accounts. Instead it would be helpful if kas could apply patches onto of those 3rd party repos.

repos:
  meta-foo:
  poky:
     url: "upstream-projects"
     refspec: 388e83aa...33
     patches: 
        add-shallow-tar-file-support : "folder1"
        add-wic-extensions: "folder2"

The problem here is, that it is not clear what base path for the folders.

One way to address the path problem is to use another pair of dictoraries

repos:
  meta-foo:
  poky:
     url: "upstream-projects"
     refspec: 388e83aa...33
     patches: 
        add-shallow-tar-file-support:
          repo: meta-foo
          path: "folder1"
        add-wic-extensions: 
          repo: meta-foo
          path: "folder2"

Another thing is how to handle after patches have been applied. The refspec changes but the good thing is that with the quilt import we have normal git commits in the tree. So we can figure out if the repos was dirty or not.

Yes, this is what bitbake does and kas starts to be more and more like bitbake. Is it bad, don't know.

Also posted in mailing list.

Log for minimal example:

┌─[1] - [2020-10-09T10:04:36]
[pts/15][ngenetzky@ngenetzky-novo:/tmp/kas-bug-report][27593]
└─>( set -x && cat kas.yml && grep 'VERSION=' $(which kas-docker) && kas-docker shell ./kas.yml )                                                                                                              
+-zsh:20> cat kas.yml
header:
  version: 3
  includes:
    - repo: not-a-repo
      file: kas/board/qemux86-64.yml
repos:
  poky:
    url: git://git.yoctoproject.org/poky
    refspec: "51f6145f8f99a02df1dad937684f014b0172e72a" # yocto-2.6.4 # thud
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:
+-zsh:20> which kas-docker
+-zsh:20> grep '--color=auto' '--exclude-dir=.bzr' '--exclude-dir=CVS' '--exclude-dir=.git' '--exclude-dir=.hg' '--exclude-dir=.svn' '--exclude-dir=.idea' '--exclude-dir=.tox' 'VERSION=' /home/ngenetzky/.local/bin//kas-docker
        KAS_IMAGE_VERSION="2.1.1"
+-zsh:20> kas-docker shell ./kas.yml
2020-10-10 03:05:36 - INFO     - kas 2.1.1 started
2020-10-10 03:05:36 - INFO     - /repo$ git rev-parse --show-toplevel
2020-10-10 03:05:36 - INFO     - /repo$ hg root
2020-10-10 03:05:36 - ERROR    - Set of coroutines/Futures is empty.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/kas/kas.py", line 162, in main
    sys.exit(kas(sys.argv[1:]))
  File "/usr/local/lib/python3.7/dist-packages/kas/kas.py", line 150, in kas
    if plugin().run(args):
  File "/usr/local/lib/python3.7/dist-packages/kas/shell.py", line 112, in run
    macro.run(ctx, args.skip)
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 63, in run
    command.execute(ctx)
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 106, in execute
    while all(executor(c) for c in self.commands):
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 106, in <genexpr>
    while all(executor(c) for c in self.commands):
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 104, in executor
    return command.execute(ctx)
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 310, in execute
    repos_fetch(ctx.missing_repos)
  File "/usr/local/lib/python3.7/dist-packages/kas/libkas.py", line 171, in repos_fetch
    loop.run_until_complete(asyncio.wait(tasks))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/usr/lib/python3.7/asyncio/tasks.py", line 354, in wait
    raise ValueError('Set of coroutines/Futures is empty.')
ValueError: Set of coroutines/Futures is empty.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/kas/kas.py", line 162, in main
    sys.exit(kas(sys.argv[1:]))
  File "/usr/local/lib/python3.7/dist-packages/kas/kas.py", line 150, in kas
    if plugin().run(args):
  File "/usr/local/lib/python3.7/dist-packages/kas/shell.py", line 112, in run
    macro.run(ctx, args.skip)
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 63, in run
    command.execute(ctx)
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 106, in execute
    while all(executor(c) for c in self.commands):
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 106, in <genexpr>
    while all(executor(c) for c in self.commands):
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 104, in executor
    return command.execute(ctx)
  File "/usr/local/lib/python3.7/dist-packages/kas/libcmds.py", line 310, in execute
    repos_fetch(ctx.missing_repos)
  File "/usr/local/lib/python3.7/dist-packages/kas/libkas.py", line 171, in repos_fetch
    loop.run_until_complete(asyncio.wait(tasks))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/usr/lib/python3.7/asyncio/tasks.py", line 354, in wait
    raise ValueError('Set of coroutines/Futures is empty.')
ValueError: Set of coroutines/Futures is empty.

PermissionError: [Errno 13] Permission denied: '/downloads'

I've written some cunning/horrible code in Bitbake that uses modules on the host for improved logging. Outside of Kas, this works fine. Inside Kas, the imports fail.

Hi,

I am trying to use kas to build yocto and I am experimenting building TI Processor SDK with kas. You clan find related information via the link below:

https://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Overview_Building_the_SDK.html

For building the SDK, two external compilers are used:

$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz
$ tar -Jxvf gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf.tar.xz -C $HOME
$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/8.3-2019.03/binrel/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz
$ tar -Jxvf gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu.tar.xz -C $HOME

and some exports are done:

$ export TOOLCHAIN_PATH_ARMV7=$HOME/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf
$ export TOOLCHAIN_PATH_ARMV8=$HOME/gcc-arm-8.3-2019.03-x86_64-aarch64-linux-gnu

I did not know how to add into the yaml file, it would be nice if we can support that.

Kind regards,
Mustafa

Running:

kas build myboard.yml:mysubmodule/mirrors.yml

Results in error:

ERROR    - All concatenated config files must belong to the same repository or all must be outside of versioning control

The file mysubmodule/mirrors.yml truly isn't from the same Repo but also isn't handled by KAS. It is just a git submodule and is nowhere mentioned in myboard.yml. Shouldn't KAS allow to use it?

The use case is that we have shared KAS include files which set common source mirrors, sstate-cache, build server options and others. And it would be nice to be able to set them on the command line to be able to choose certain configs dynamically.

The ideal way would be to have KAS check out the git repo with common kas files and still be able to reference them from the command line. But the documentation says that's prohibited. My current work-around is to share them with a git submodule which isn't handled by KAS. But as mentioned above, KAS refuses to include files from it which is really strange.

Hey,

Is it possible to use environment variables inside kas config files? That would be very convenient to inject e.g. a deploy token in a CI pipeline.
For example:

repos:
  meta-something:
    url: https://oauth2:${OAUTH_DEPLOY_TOKEN}@git.somecompany.org/reponame
    refspec: ${DEPLOY_REFSPEC}

Thanks in advance!
Marius

It looks like kas supports multi-config targets in the configuration file but this isn't covered by the documentation at all. Examples would also be helpful.

See fa15757 and 230c5a9.

Say I have a base YAML file that but want to build it for a number of variations: six different machines, with musl or glibc, and two different distributions.

Currently that means 6 * 2 * 2 = 24 different yaml files would be needed. It would be good if kas supported matrix builds, similar to Travis, so that these variations could be simply listed in a top-level yml that expands out to all of the required builds.

My kas in CI just failed with the following mysterious errors:

1202020-10-13 10:53:46 - INFO     - /builds/poky$ git remote set-url origin https://git.yoctoproject.org/git/poky
1222020-10-13 10:53:46 - INFO     - Changing remote URL to https://git.yoctoproject.org/git/poky failed with error: 

Note the absence of error messages.

The problem was that kas was running as a normal user but the poky checkout was owned by root.