containers / build Goto Github PK

another build tool for container images (archived, see https://github.com/rkt/rkt/issues/4024)

License: Apache License 2.0

Go 98.62% Shell 1.38%

build's Introduction

This project is currently unmaintained

acbuild was originally created to be the tool used to build AppC images. Due to the introduction of the Open Container Initiative, development on AppC was officially suspended in November, 2016. While acbuild has the ability to also produce OCI images, it is not the only tool capable of doing so. In its current state, acbuild is not currently maintained. If you wish to become a maintainer of acbuild, feel free to start contributing and ask for direct commit access via the issue/PR tracker.

For those looking for an OCI image manipulation tool that is actively maintained, umoci or buildah might be able to fill the role.

acbuild - another container build tool

acbuild is a command line utility to build and modify container images.

It is intended to provide an image build workflow independent of specific formats; currently, it can output the following types of container images:

ACI, the container image format defined in the App Container (appc) spec.
OCI, the format defined in the Open Containers Image Format specification

http://www.youtube.com/watch?v=WcnIDm80y68

Rationale

We needed a powerful tool for constructing and manipulating container images that made it easy to iteratively build containers, both from scratch and atop existing images. We wanted that tool to integrate well with Unix mechanisms like the shell and Makefiles so it would fit seamlessly into well-known administrator and developer workflows.

Installation

Dependencies

acbuild can only be run on a Linux system, and has only been tested on the amd64 architecture.

For trying out acbuild on Mac OS X, it's recommended to use Vagrant. Instructions on how to do this are a little further down in this document.

acbuild requires a handful of commands be available on the system on which it's run:

cp
modprobe

Additionally systemd-nspawn is required to use the default engine for acbuild run. Thus on Ubuntu the systemd-container package needs to be installed.

Prebuilt Binaries

The easiest way to get acbuild is to download one of the releases from GitHub.

Build from source

The other way to get acbuild is to build it from source. Building from source requires Go 1.5+.

Follow these steps to do so:

Grab the source code for acbuild by git cloneing the source repository:
```
cd ~
git clone https://github.com/containers/build acbuild
```
Run the build script from the root source repository directory:
```
cd acbuild
./build
```
Or, if you want to build in docker (assuming $PWD exists and contains acbuild/ on your Docker host):
```
cd acbuild
./build-docker
```
A bin/ directory will be created that contains the acbuild tool. To make sure your shell can find this executable, append this directory to your environment's $PATH variable. You can do this in your .bashrc or similar file, for example:
```
vi ~/.bashrc
```

and put the following lines at the end of the file:

export ACBUILD_BIN_DIR=~/acbuild/bin
export PATH=$PATH:$ACBUILD_BIN_DIR

Building acbuild with rkt

If rkt is installed on the system, acbuild can also be built inside of a rkt container with the following command:

./build-rkt

Trying out acbuild using Vagrant

For users with Vagrant 1.5.x or greater, there's a provided Vagrantfile that can quickly get you set up with a Linux VM that has both acbuild and rkt. The following steps will grab acbuild, set up the machine, and ssh into it.

git clone https://github.com/containers/build acbuild
cd acbuild
vagrant up
vagrant ssh

Documentation

Documentation about acbuild and many of its commands is available in the Documentation directory in this repository.

Examples

Check out the examples directory for some common applications being packaged into ACIs with acbuild.

Related work

https://github.com/sgotti/baci
https://github.com/appc/spec/tree/master/actool - particularly the build and patch-manifest subcommands. acbuild may subsume such functionality, leaving actool as a validator only.
https://github.com/blablacar/dgr

build's People

Contributors

Stargazers

Watchers

build's Issues

acbuild: Failed to create directory /home/ajung/src/x/.acbuild/currentaci/rootfs/sys/fs/selinux: Read-only file system ON FEDORA 23

I am trying to build an image using acbuild (1:1 example taken from

https://coreos.com/blog/rkt-0.10.0-with-new-api-service/
).

This fails on Fedora 23.


[root@vmd9666 x]# cat x
acbuild begin quay.io/listhub/alpine
acbuild set-name example.com/nginx
acbuild run apk update
acbuild run apk add nginx
acbuild set-exec -- /usr/sbin/nginx -g "daemon off;"
acbuild port add http tcp 80
acbuild mount add html /usr/share/nginx/html
acbuild label add arch amd64
acbuild label add os linux
acbuild write --overwrite nginx-latest-linux-amd64.aci
acbuild end
[root@vmd9666 x]# cd /opt/PDFreactor/^C
[root@vmd9666 x]# sh x


Downloading quay.io/listhub/alpine: [==========================] 2.63 MB/2.63 MB
Failed to create directory /home/ajung/src/x/.acbuild/currentaci/rootfs/sys/fs/selinux: Read-only file system
Failed to create directory /home/ajung/src/x/.acbuild/currentaci/rootfs/sys/fs/selinux: Read-only file system
Timezone Europe/Berlin does not exist in container, not updating container timezone.
fetch http://dl-4.alpinelinux.org/alpine/v3.2/main/x86_64/APKINDEX.tar.gz
v3.2.3-104-g838b3e3 [http://dl-4.alpinelinux.org/alpine/v3.2/main]
OK: 5299 distinct packages available
Failed to create directory /home/ajung/src/x/.acbuild/currentaci/rootfs/sys/fs/selinux: Read-only file system
Failed to create directory /home/ajung/src/x/.acbuild/currentaci/rootfs/sys/fs/selinux: Read-only file system
Timezone Europe/Berlin does not exist in container, not updating container timezone.
(1/2) Installing pcre (8.37-r1)
(2/2) Installing nginx (1.8.0-r1)
Executing nginx-1.8.0-r1.pre-install
Executing busybox-1.23.2-r0.trigger
OK: 7 MiB in 17 packages

*: add script mode

I think most scripts users will write to use acbuild will follow the same general formula:

start the build
perform some operations, if there's an error abort the build
end the build

This formula doesn't require any of bash's complex control flow like for loops, and the resulting script must be run as root if it uses the run command. This will require careful inspection of any build scripts the user downloads, since blindly running shell scripts as root is terrifying.

I propose being able to write a file composed solely of comments and acbuild commands, in which acbuild will exec itself with every line that isn't a comment or empty. In the event of an error acbuild can call abort to clean up after itself, and this offers a little more security that an untrusted shell script.

Such a file could look like this:

# start the build
begin
# we're using alpine
dep add quay.io/coreos/alpine-sh
# this is an nginx container
run -- apk install nginx
# end the build
end ./mycoolapp.aci

And could be invoked like: acbuild script ./build-my-cool-app.acb

Support using rkt cas?

Spitballing. It'd be cool to have a mode where acbuild uses rkt's cas as its backend store for images. That could mean you could rapidly iterate on images using acbuild and then be able to test them using rkt without incurring the overhead of having to load them into the store every time.

Obviously we don't want to couple acbuild to rkt so we would probably implement it using some kind of plug-in. Or if they completely share(d) the backend format it could in theory be as simple as pointing acbuild at the rkt cas directory? (obvious concerns there with compatibility)

/cc @chancez

*: add implicit layered mode

A mode should be added where every acbuild command implicitly creates a new layer (akin to how a Dockerfile works). This would probably make the most sense as a flag on the begin command.

modify flag shouldn't require root

derek@energia ~/aci> acbuild --modify ./alpine-latest-linux-amd64.aci set-name localhost/alpine
"/bin/bbsuid": permission denied - call write as root

files copied into container ought to have a uid/gid mapping

as of 0e884ec

vbatts@valse ~/src/appc/acbuild/examples/acserver (master *) $ ./build-acserver.sh 
Building acserver...
Beginning build with an empty ACI
Setting name of ACI to example.com/acserver
Copying host:acserver to aci:/bin/acserver
Copying host:/home/vbatts/opt/gopath/src/github.com/appc/acserver/templates to aci:/templates
Adding port "http"="tcp"
Adding mount point "acis"="/acis"
Setting exec command [/bin/acserver]
Writing ACI to acserver-latest-linux-amd64.aci
vbatts@valse ~/src/appc/acbuild/examples/acserver (master *) $ tar tvf acserver-latest-linux-amd64.aci
drwxr-xr-x 1000/1001         0 2015-11-06 05:12 rootfs
drwxr-xr-x 1000/1001         0 2015-11-06 05:12 rootfs/bin
-rwxr-xr-x 1000/1001   7158435 2015-11-06 05:12 rootfs/bin/acserver
drwxrwxr-x 1000/1001         0 2015-11-06 04:58 rootfs/templates
-rw-rw-r-- 1000/1001      2464 2015-11-06 04:58 rootfs/templates/index.html
-rw-r--r-- root/root       345 2015-11-06 05:12 manifest

Add support for manipulating isolators

acbuild should have functionality for adding/removing/modifying isolators in an ACI.

util: replace UnTar with an external library

There are issues with the UnTar in its current state, and it's not a particularly easy thing to write. It should be replaced with a library.

*: add explicit layered mode

I think a command like acbuild layer should be supported that would add a new layer into the current ACI.

acbuild exec -- wget http://example.com/very-big-file.deb
acbuild exec -- dpkg -i ./very-big-file.deb
acbuild exec -- rm ./very-big-file.deb
acbuild layer

The above would produce a layered image, but the file very-big-file.deb would not exist in any of the layers.

*: generate annotations listing acbuild steps

It would be neat if acbuild kept track of the acbuild commands that get run on an ACI in the image's annotations.

It would provide inside into how a given image gets built.

Concurrent build issues

If I'm invoking two acbuilds acbuild where the first is using begin and the second is using --modify, the second one fails to start because the lock for the first is created in the current directory, and the second cannot acquire the lock.

It would be great if the .acbuild context was named after the build, and the lock was associated with a particular .acbuild context (or aci if using --modify).

Ex:
acbuild begin path/to/example.aci --name example-app where --name must be unique for this particular build, to allow for multiple concurrent builds.
This would result in something like:
.acbuild-example-app and a lock named .acbuild-example-app.lock in the current directory.

acbuild --modiy path/to/example.aci set name coreos.com/example could produce a lock named something like:
.path_to_example.aci.lock

You could still only have one instance of an acbuild begin in a single directory, but you could have as many acbuild --modify running in the same directory, assuming each have their own ACI to modify (still limited to one acbuild per aci).

Add support for event handlers

Issue untar'ing some ACIs

Certain ACIs cause the following error when acbuild attempts to untar them:

extracttar error: exit status 1, output: error extracting tar: error extracting tarball: archive/tar: invalid tar header

An example of an ACI that causes this is at https://aci.gonyeo.com/nginx-latest-linux-amd64.aci. The application was made with buildroot, and the ACI was built with actool. rkt has no issues running the ACI.

Nginx example produces ACI with empty rootfs

Running the example in the README created an ACI with nothing in the rootfs. Not sure if this is a bug or if I'm missing something.

Here's my build file

$ cat build.sh 
#!/bin/bash

acbuild begin
acbuild dependency add quay.io/fermayo/ubuntu
acbuild run -- apt-get update
acbuild run -- apt-get -y install nginx
acbuild set-exec /usr/sbin/nginx
acbuild set-name example.com/ubuntu-nginx
acbuild write ubuntu-nginx.aci
acbuild end

After building the ACI file and attempting to run it I got the following error message.

$ sudo rkt run --insecure-skip-verify ubuntu-nginx.aci 
rkt: using image from file /usr/bin/stage1-coreos.aci
rkt: using image from file /home/eric/p/acbuild/ubuntu-nginx.aci
rkt: using image from local store for image name quay.io/fermayo/ubuntu
[ 8331.672240] nginx[4]: Error: Unable to open "/usr/sbin/nginx": No such file or directory

Further inspection of the ACI file shows an empty rootfs directory

$ tar -tvf ubuntu-nginx.aci 
drwxr-xr-x 1000/1000         0 2015-11-08 12:54 rootfs
-rw-r--r-- root/root       271 2015-11-08 12:54 manifest

The manifest file looks pretty reasonable

$ tar -xf ubuntu-nginx.aci manifest -O | jq .
{
  "acKind": "ImageManifest",
  "acVersion": "0.7.1+git",
  "name": "example.com/ubuntu-nginx",
  "labels": [
    {
      "name": "arch",
      "value": "amd64"
    },
    {
      "name": "os",
      "value": "linux"
    }
  ],
  "app": {
    "exec": [
      "/usr/sbin/nginx"
    ],
    "user": "0",
    "group": "0"
  },
  "dependencies": [
    {
      "imageName": "quay.io/fermayo/ubuntu"
    }
  ]
}

Running Fedora 23. Here's some additional version data

$ sudo acbuild version
acbuild version v0.1.1-38-g0e884ec
appc version 0.7.1+git
$ sudo rkt version
rkt version 0.10.0
appc version 0.7.1
$ sudo systemd-nspawn --version
systemd 222
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN
$ uname -r
4.2.5-300.fc23.x86_64

Missing support for eventHandlers

Example taken from https://github.com/appc/spec/blob/master/spec/aci.md#image-manifest-schema

...
    "app": {
...
        "eventHandlers": [
            {
                "exec": [
                    "/usr/bin/data-downloader"
                ],
                "name": "pre-start"
            },
            {
                "exec": [
                    "/usr/bin/deregister-worker",
                    "--verbose"
                ],
                "name": "post-stop"
            }
        ],
...

"acbuild" features overlap with "actool patch-manifest"

The command actool patch-manifest has some features that overlap with acbuild:

https://github.com/appc/spec/blob/master/actool/manifest.go#L56

Should actool get the missing features and then actool patch-manifest be deprecated?

squash: add implementation

A new command, squash, should be added to support squashing together the given ACI with its dependencies.

acbuild begin
acbuild dep add quay.io/coreos/alpine-sh
acbuild squash

would be equivalent to

acbuild begin ./alpine-sh

*: deal with versioning

We need to start thinking about how acbuild works w.r.t versions of the spec.
Initially at least acbuild needs to be versioned, I suggest it could just be 1:1 with the spec itself (i.e. acbuild --version would be equal to the version of appc that is vendored in)
Next would be to start thinking about forwards/backwards compatibility.

end: image signing

It would be really convenient to be able to end a build and have acbuild produce the .asc file in addition to the .aci file, if it's handed the keyrings.

Something like the following:

acbuild end --keyring ./keys.pub --secret-keyring ./keys.sec mycoolapp-latest-linux-amd64.aci

Would produce both a mycoolapp-latest-linux-amd64.aci file and a mycoolapp-latest-linux-amd64.aci.asc file in the current directory.

Missing support for workingDirectory

Example taken from https://github.com/appc/spec/blob/master/spec/aci.md#image-manifest-schema

...
    "app": {
...
        "workingDirectory": "/opt/work",
...

Release v0.2.0

Issue for blocking references on the next acbuild release.

run: remove need for --

It would be very convenient if cobra could be configured in such a way that it would stop trying to parse flags once it sees the run command, so that it's not necessary to put -- in between run and the command.

Not able to run container started with appc image built by acbuild

Hi
I used the nginx example as below to create aci image in my CoreOS node.
acbuild begin
acbuild dependency add quay.io/fermayo/ubuntu
acbuild run -- apt-get update
acbuild run -- apt-get -y install nginx
acbuild set-exec /usr/sbin/nginx
acbuild set-name example.com/ubuntu-nginx
acbuild write ubuntu-nginx.aci
acbuild end

aci image got created successfully, but rkt container starts and exits. rkt version used is 0.9.0 and coreos version is 845.0.0. Following are the logs:
core@core-01 ~ $ sudo rkt run --debug --insecure-skip-verify ubuntu-nginx.aci
rkt: using image from file /usr/share/rkt/stage1-coreos.aci
rkt: using image from file /home/core/ubuntu-nginx.aci
rkt: searching for app image quay.io/fermayo/ubuntu
rkt: remote fetching from url https://quay.io/c1/aci/quay.io/fermayo/ubuntu/latest/aci/linux/amd64/
rkt: fetching image from https://quay.io/c1/aci/quay.io/fermayo/ubuntu/latest/aci/linux/amd64/
Downloading ACI: [=============================================] 72.4 MB/72.4 MB
2015/11/08 05:25:26 Preparing stage1
2015/11/08 05:25:27 Writing image manifest
2015/11/08 05:25:27 Loading image sha512-973e0f0a1c8252b1e92c882c902e4a00a0c75284678c39334f7c8a14d1a2740c
2015/11/08 05:25:33 Writing image manifest
2015/11/08 05:25:33 Writing pod manifest
2015/11/08 05:25:33 Setting up stage1
2015/11/08 05:25:33 Wrote filesystem to /var/lib/rkt/pods/run/5005b059-212e-4f68-82f3-ba4a7d5529c6
2015/11/08 05:25:33 Pivoting to filesystem /var/lib/rkt/pods/run/5005b059-212e-4f68-82f3-ba4a7d5529c6
2015/11/08 05:25:33 Execing /init
2015/11/08 05:25:33 Loading networks from /etc/rkt/net.d
2015/11/08 05:25:33 Loading network default with type ptp
Spawning container rkt-5005b059-212e-4f68-82f3-ba4a7d5529c6 on /var/lib/rkt/pods/run/5005b059-212e-4f68-82f3-ba4a7d5529c6/stage1/rootfs.
Press ^] three times within 1s to kill container.
systemd 222 running in system mode. (-PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4
+SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
Detected virtualization systemd-nspawn.
Detected architecture x86-64.

Welcome to Linux!

Initializing machine ID from container UUID.
[ OK ] Created slice -.slice.
[ OK ] Listening on Journal Socket.
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Created slice system.slice.
[ OK ] Started Pod shutdown.
Starting Pod shutdown...
[ OK ] Created slice system-prepare\x2dapp.slice.
Starting Journal Service...
[ OK ] Started ubuntu-nginx Reaper.
Starting ubuntu-nginx Reaper...
[ OK ] Started Journal Service.
Starting Prepare minimum environment for chrooted applications...
[ OK ] Started Prepare minimum environment for chrooted applications.
[ OK ] Started Application=ubuntu-nginx Image=example.com/ubuntu-nginx.
Starting Application=ubuntu-nginx Image=example.com/ubuntu-nginx...
[ OK ] Reached target rkt apps target.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Halting system.
Container rkt-5005b059-212e-4f68-82f3-ba4a7d5529c6 has been shut down.

Any ideas?

Thanks
Sreenivas

*: context free mode

A context free mode should be added to support quick modifications to local ACI files.

Current tar implementation requires begin to be run as root

The ExpandTar function untars the ACI in a chroot, which requires root permissions to set up.

Feature Roadmap Overview

Overview for tracking the feature roadmap.

eventHandlers #106
workingDirectory #107
isolators #108

File removal in ACI

This issue should perhaps be put on appc/spec, but I'm temporarily putting it here to reduce noise.

As discussed OOB, we were planning to use pathWhitelist to represent file removal. However, a whitelist is much more clumsy to use than a black list for removing files. For instance, if I simply want to remove a single file hello.txt, I would essentially have to add to the whitelist all files and directories that are not hello.txt. This process is not only error-prone, but could also result in a large manifest.

I propose two solutions:

The simplest solution is simply to have a pathBlacklist. Intuitively, the blacklist would specify all files that we want to exclude from the final rendered image.

We could also borrow ideas from overlayfs:

whiteouts and opaque directories
--------------------------------

In order to support rm and rmdir without changing the lower
filesystem, an overlay filesystem needs to record in the upper filesystem
that files have been removed.  This is done using whiteouts and opaque
directories (non-directories are always opaque).

A whiteout is created as a character device with 0/0 device number.
When a whiteout is found in the upper level of a merged directory, any
matching name in the lower level is ignored, and the whiteout itself
is also hidden.

A directory is made opaque by setting the xattr "trusted.overlay.opaque"
to "y".  Where the upper filesystem contains an opaque directory, any
directory in the lower filesystem with the same name is ignored.

So in this approach, an ACI that removes files would have the "whiteouts" and "opaque directories" described above in its rootfs. When rendered, it would remove the corresponding files in the lower layer.

Personally, I like the second approach, as it keeps all the "state" in rootfs, reducing the manifest to the simple metadata file that it was supposed to be. It's also worth noting that the two approaches can coexist.

What are your thoughts? @klizhentas @jonboulle @jzelinskie

end: image pushing

It would be convenient (once acpush is finished) to be able to specify an image name, and have the image pushed to that location (optionally signing it with #15).

Something like the following:

acbuild end --keyring ./keys.pub --secret-keyring ./keys.sec quay.io/dgonyeo/mycoolapp

Would generate an aci and asc file, and then push them to quay. I imagine that in the event of push failure the build would be left in progress, so that a push could be reattempted or the ACI could be written to the local filesystem (also with the end command).

begin: error message when local aci path is invalid isn't good

If the local file doesn't exist acbuild will attempt to perform meta discovery on the provided start point. This results in misleading error messages when the aci doesn't exist:

begin: Get https://beginlocaltest.aci?ac-discovery=1: dial tcp: lookup beginlocaltest.aci on 10.7.2.1:53: no such host

Use github.com/spf13/cobra CLI library?

Coming from rkt/rkt#1028 starting with a CLI library like it should be useful (see rkt/rkt#632 (comment) for a list of features/requirementd) and avoid additional work now and in the future.

*: recipe for running acbuild in container

We should have acbuild packaged in an ACI and then be able to invoke it using any appc runtime, e.g. rkt. I expect this would just mount in the user's project/asset directly to a known location (/data or whatever) and then run the script they pass to it.

For example with rkt this might look like:

$ ls
app.js
build-nodejs.sh
$ rkt run --volume data,kind=host,source=$(pwd) appc.io/acbuild ./build-nodejs.sh

Where "data" would be defined as a mountpoint in the appc.io/acbuild image.
Then this would output the ACI in the same directory.

$ ls
app.js
build-nodejs.sh
nodejs-latest-linux-amd64.aci

build-jenkins in acbuild fails

vagrant@vagrant-ubuntu-vivid-64:/vagrant$ ./build-jenkins
Beginning build with an empty ACI
dependency add: ACIdentifier must contain only lower case alphanumeric characters plus "-._~/"

/tmp missing when calling `acbuild run`

Example:

root@core-01:/home/core/testcore-01 test # echo "test" > foo.txt
root@core-01:/home/core/testcore-01 test # rkt image export busybox busybox.aci
root@core-01:/home/core/testcore-01 test # ls
busybox.aci  foo.txt
root@core-01:/home/core/testcore-01 test # acbuild begin busybox.aci
root@core-01:/home/core/testcore-01 test # acbuild copy foo.txt /tmp/foo.txt
root@core-01:/home/core/testcore-01 test # acbuild run cat /tmp/foo.txt
Timezone UTC does not exist in container, not updating container timezone.
cat: can't open '/tmp/foo.txt': No such file or directory
run: exit status 1
root@core-01:/home/core/testcore-01 test # acbuild run ls /tmp
Timezone UTC does not exist in container, not updating container timezone.
root@core-01:/home/core/testcore-01 test # ls .acbuild/
currentaci/        depstore-expanded/ depstore-tar/
core-01 test # ls .acbuild/currentaci/rootfs/tmp
foo.txt

Add ability to set working directory in manifest

begin: image fetching

begin should support being given an image name instead of a local file as a starting point. acbuild would then download the ACI and use it as the starting point for the new build.

acbuild begin quay.io/coreos/alpine-sh

*: add some initial tests for every command

Each command supported by acbuild should have at least one functional test.

Use available acibuilder/fsdiffer?

In rkt/rkt#434 I proposed an aci builder library (you can find it here https://github.com/sgotti/acibuilder https://github.com/sgotti/fsdiffer). Maybe it can be improved to support also overlayfs and be used here (no need to vendor, just take it!)

Missing support for isolators

Example taken from https://github.com/appc/spec/blob/master/spec/aci.md#image-manifest-schema

...
    "app": {
...
        "isolators": [
            {
                "name": "resource/cpu",
                "value": {
                    "request": "250m",
                    "limit": "500m"
                }
            },
            {
                "name": "resource/memory",
                "value": {
                    "request": "1G",
                    "limit": "2G"
                }
            },
            {
                "name": "os/linux/capabilities-retain-set",
                "value": {
                    "set": ["CAP_NET_BIND_SERVICE"]
                }
            }
        ],
...

determine image references format

For many of its commands, acbuild reference an ACI (or several ACIs) that might be located locally on disk or potentially somewhere else (HTTP endpoint, AC discovery endpoint, ...). We should define a consistent format that this reference. We faced a similar question in rkt but haven't quite resolved it yet: rkt/rkt#715

modify should append ./ to the specified image

As a result of the recent begin changes:

derek@energia ~/aci> acbuild --modify alpine-latest-linux-amd64.aci set-name localhost/alpine
unknown host when fetching image, check your connection and local file paths must start with '/' or '.'

run: support different engines

Support should be added for alternate execution engines, like runc.

*: Use file locking to prevent concurrent invocations

Concurrent invocations in acbuild's current state can cause issues. A file lock should be acquired/released in the build context's directory to prevent this.

systemd-nspawn: prints timezone warning

Every time I've seen acbuild run used, the following warning is printed before the actual command is exec'd:

Timezone Europe/Berlin does not exist in container, not updating container timezone.

We should prevent the error from being printed.

Panic while uncompressing dependency

After updating to the lastest master this morning, I started getting the following error:

[jcollie@pc28043 rkt-logstash]$ acbuild version
acbuild version v0.1.0-4-g40f07a0
appc version 0.7.1+git
[jcollie@pc28043 rkt-logstash]$ sudo acbuild begin
[jcollie@pc28043 rkt-logstash]$ sudo acbuild dependency add dmacc.net/centos7 --image-id="sha512-baae275b452242b1ccdba64bff15a7a9" --label="version=0.1"
[jcollie@pc28043 rkt-logstash]$ sudo acbuild label add os linux
[jcollie@pc28043 rkt-logstash]$ sudo acbuild label add arch amd64
[jcollie@pc28043 rkt-logstash]$ sudo acbuild run --debug -- yum -y update
Running: [yum -y update]
Downloading dmacc.net/centos7: [===============================] 62.9 MB/62.9 MB
panic: runtime error: makeslice: len out of range==            ] 41.6 MB/62.9 MB

goroutine 1 [running]:
strings.Repeat(0x8ca320, 0x1, 0xfffffffffffffff5, 0x0, 0x0)
        /usr/local/go/src/strings/strings.go:464 +0x5f
github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress.DrawTextFormatBarForW.func2(0x4f38000, 0x3bfe5e9, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress/draw.go:114 +0xcf
github.com/appc/acbuild/registry.newIoprogress.func1(0x4f38000, 0x3bfe5e9, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/registry/fetch.go:441 +0x244
github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress.DrawTerminalf.func1(0x4f38000, 0x3bfe5e9, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress/draw.go:54 +0xf8
github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress.(*Reader).drawProgress(0xc820063d10)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress/reader.go:76 +0x1e1
github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress.(*Reader).Read(0xc820063d10, 0xc8204c6000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/Godeps/_workspace/src/github.com/coreos/ioprogress/reader.go:50 +0x115
io.copyBuffer(0x7fd0406453f8, 0xc820028228, 0x7fd040647418, 0xc820063d10, 0xc8204c6000, 0x8000, 0x8000, 0x4f30000, 0x0, 0x0)
        /usr/local/go/src/io/io.go:381 +0x247
io.Copy(0x7fd0406453f8, 0xc820028228, 0x7fd040647418, 0xc820063d10, 0xc820028228, 0x0, 0x0)
        /usr/local/go/src/io/io.go:351 +0x64
github.com/appc/acbuild/registry.Registry.uncompress(0xc8200f6520, 0x15, 0xc8200f65a0, 0x1a, 0x860100, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/registry/fetch.go:256 +0x5e2
github.com/appc/acbuild/registry.Registry.fetchACIWithSize(0xc8200f6520, 0x15, 0xc8200f65a0, 0x1a, 0x430100, 0xc8200f6be0, 0x11, 0xc820085b80, 0x1, 0x4, ...)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/registry/fetch.go:134 +0x556
github.com/appc/acbuild/registry.Registry.Fetch(0xc8200f6520, 0x15, 0xc8200f65a0, 0x1a, 0x540100, 0xc8200f6be0, 0x11, 0xc820085b80, 0x1, 0x4, ...)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/registry/fetch.go:58 +0x1c9
github.com/appc/acbuild/registry.Registry.FetchAndRender(0xc8200f6520, 0x15, 0xc8200f65a0, 0x1a, 0x100, 0xc8200f6be0, 0x11, 0xc820085b80, 0x1, 0x4, ...)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/registry/fetch.go:71 +0xf3
github.com/appc/acbuild/lib.(*ACBuild).renderACI(0xc820085980, 0x100, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/lib/run.go:200 +0x35c
github.com/appc/acbuild/lib.(*ACBuild).Run(0xc820085980, 0xc820062fa0, 0x3, 0x5, 0x0, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/lib/run.go:87 +0x4f1
main.runRun(0xd19fc0, 0xc820062fa0, 0x3, 0x5, 0x2)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/acbuild/run.go:48 +0x19f
main.runWrapper.func1(0xd19fc0, 0xc820062fa0, 0x3, 0x5)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/acbuild/acbuild.go:123 +0x73
github.com/appc/acbuild/Godeps/_workspace/src/github.com/spf13/cobra.(*Command).execute(0xd19fc0, 0xc820062f00, 0x5, 0x5, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/Godeps/_workspace/src/github.com/spf13/cobra/command.go:496 +0x6e3
github.com/appc/acbuild/Godeps/_workspace/src/github.com/spf13/cobra.(*Command).Execute(0xd17c00, 0x0, 0x0)
        /opt/acbuild/gopath/src/github.com/appc/acbuild/Godeps/_workspace/src/github.com/spf13/cobra/command.go:561 +0x180
main.main()
        /opt/acbuild/gopath/src/github.com/appc/acbuild/acbuild/acbuild.go:198 +0x94

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 19 [select]:
net/http.(*persistConn).writeLoop(0xc8200d33f0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 21 [IO wait]:
net.runtime_pollWait(0x7fd040647120, 0x72, 0xc8200121b0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8200aced0, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8200aced0, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8200ace70, 0xc820458000, 0x8000, 0x8000, 0x0, 0x7fd040641050, 0xc8200121b0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82015e018, 0xc820458000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820424ff0, 0x7fd0406472a0, 0xc82015e018, 0x5, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc8201b5b80, 0x99c017, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc8201b5b80, 0xc820456000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7fd03edcf960, 0xc8201b5b80, 0xc8200d34f8, 0xc820456000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc820442b60, 0xc820456000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8203f30e0)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8203f30e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200d34a0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 18 [IO wait]:
net.runtime_pollWait(0x7fd0406471e0, 0x72, 0xc8200121b0)
        /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8200acd10, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8200acd10, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8200accb0, 0xc820134000, 0x1000, 0x1000, 0x0, 0x7fd040641050, 0xc8200121b0)
        /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200281e0, 0xc820134000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820128000, 0x7fd0406472a0, 0xc8200281e0, 0x5, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc82007e840, 0x99c017, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc82007e840, 0xc820135000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7fd03edcf960, 0xc82007e840, 0xc8200d3448, 0xc820135000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc820423f40, 0xc820135000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8203f29c0)
        /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8203f29c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200d33f0)
        /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 22 [select]:
net/http.(*persistConn).writeLoop(0xc8200d34a0)
        /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:686 +0xc9d

Running git bisect points to the following commit:

a5ad61df3b0d636115d3543c82befc0c84ffdb48 is the first bad commit
commit a5ad61df3b0d636115d3543c82befc0c84ffdb48
Author: Derek Gonyeo <[email protected]>
Date:   Mon Oct 19 18:04:15 2015 -0700

    *: refactored library functions, added struct with common fields

    The library was refactored so that all the exposed functions were
    attached to a struct containing the information about the current
    context. This makes using the library much cleaner.

    The Registry struct had some of its fields renamed to be consistent with
    the new struct in the library.

    The functions for acquiring and releasing the lock were also moved from
    acbuild/acbuild.go to the new struct.

:040000 040000 0b4c18665811cff0026264baf86df63fafcc05ee 55039ab1fead9942c9614bbbe9f739b6f657cf0d M  acbuild
:040000 040000 a0e24013e636d4e8189af2c8d6d87acb0f914996 ce69a0ba1fd044cde549f7320f8aeef1f6ca40ee M  lib
:040000 040000 3a03acbd4d502a348d419705fb0f1257906a7060 d0ac54b87df26184a5bc3c884b8bc8c36fa50363 M  registry

Please "vendor" released/tagged dependencies whenever possible

Downstream package maintainers should be able to reuse packages that are usually built from released/tagged versions instead of random snapshot.

Whenever possible please bundle (vendor) released/tagged versions of components in Godeps. Thanks.

*: initial implementation

The README describes what acbuild should look like; now we need an initial implementation of the thing.

acbuild pod for building

It would be great to have some sort of pod that we can run for acbuild. Their would be a couple of moving parts:

the file accepter, a server that a developer could upload their source code and a acbuild script to
the image builder, would build whatever ends up in the shared mount from the file accepter
the image server, would serve up the end result images for the user to consume after the build completes

The basic idea would be to do an acbuild against a remote server by sending it the context of a currently checked out project like a Go project, rails app, etc.

fails to unmount bind mount

Occasionally when running acbuild I'll get an error like this:

umount: /home/jcollie/dev/rkt-logstash/.acbuild/target: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)

Once that happens and subsequent commands fail with:

run: remove .acbuild/target/tmp: directory not empty

It doesn't happen every time (maybe 5-10% of the time).

*: add Vagrantfile

For non-Linux users, it would be nice to have a very simple Vagrantfile to get them up and running with acbuild.

Example: https://github.com/coreos/rkt/blob/master/Vagrantfile

containers / build Goto Github PK

build's Introduction

This project is currently unmaintained

acbuild - another container build tool

Rationale

Installation

Dependencies

Prebuilt Binaries

Build from source

Building acbuild with rkt

Trying out acbuild using Vagrant

Documentation

Examples

Related work

build's People

Contributors

Stargazers

Watchers

Forkers

build's Issues

Recommend Projects

Recommend Topics

Recommend Org