Git Product home page Git Product logo

smith's Introduction

smith - microcontainer builder

smith

What is smith?

smith is a simple command line utility for building microcontainers from rpm packages or oci images.

Principles of microcontainers

  1. A microcontainer only contains the process to be run and its direct dependencies.

  2. The microcontainer has files with no user ownership or special permissions beyond the executable bit.

  3. The root filesystem of the container should be able to run read-only. All writes from the container should be into a directory called /write. Any unique config that an individual container instance will need should be placed into a directory called /read. Ephemeral files such as pid files can be written to /run.

Building and Running smith

You can build and run smith either as:

  • A Docker image
  • A Binary

Both methods are described below, but the Docker route is recommended as the simplest and easiest option.

Docker based smith

Dependencies

  • Docker

Method

  1. Clone smith:

git clone https://github.com/oracle/smith.git

  1. Build smith Docker image using the Dockerfile provided, optionally adding your own docker-repo-id to the tag:

sudo docker build -t [<docker-repo-id>/]smith .

  1. Set up an alias (or script) to run smith from the command line:
smith(){
    sudo docker run -it --rm \
    --privileged -v $PWD:/write \
    -v cache:/var/cache \
    -v /tmp:/tmp \
    -v mock:/var/lib/mock [<docker-repo-id>/]smith $@
}

You should now be able to start building microcontainers (see below).

Binary smith

wercker status

Building can be done via the Makefile:

make

Dependencies

Build
  • Docker
  • Go

To install go run sudo yum install golang-bin or sudo apt install golang-go as appropriate

Go dependencies are vendored in the vendor directory.

Runtime

To build from RPMs, smith requires:

  • mock

mock can have issues with non - RPM distros.

If you have problems installing or running smith natively on a non - RPM distro, best advice is to build it and run it in a Docker container (see above)

mock can be installed on Debian/Ubuntu with some extra care (see below). Specifically you need at least mock 1.2. Version 1.1.X will not work because the -r flag does not support abspath to the mock config file. Be aware that your smith builds may still fail.

Debian/Ubuntu specific instructions (Here be Dragons):

sudo apt install mock createrepo yum

# Fedora rawhide chroot (which mock uses by default) does not play well with
# Debian, so point /etc/mock/default.cfg to EPEL 7 (6 on Ubuntu):
sudo ln -s /etc/mock/epel-7-x86_64.cfg /etc/mock/default.cfg 
# rpm on Debian has a patch to macros that messes up mock so undo it. Note
# that updating your os will sometimes reset this file and you will have
# to run this command again.
sudo sed -i 's;%_dbpath\t.*;%_dbpath\t\t%{_var}/lib/rpm;g' /usr/lib/rpm/macros
# on debian/ubuntu for some reason yum tries to install packages for
# multiple archs, so it is necessary to update the yum.conf section in
# default.cfg to prevent that. If you switch your default.cfg you may
# have to do this again.
sudo sed -i '/\[main\]/a multilib_policy=best' /etc/mock/default.cfg

Whichever distro you are using check that your user is a member of the group mock:

$ groups

If your user is not a member of the group mock then add them:

$ usermod -aG mock <your_username>

On Oracle Linux edit your /etc/mock/site-defaults.cfg and add:

config_opts['use_nspawn'] = False

Installing smith

Installing can be done via the Makefile:

sudo make install

Using smith

To use smith, simply create a smith.yaml defining your container and run smith. If you want to overlay additional files or symlinks, simply place them into a directory called rootfs beside smith.yaml.

If you are building the same container multiple times without editing the package line, the -f parameter will rebuild the container without reinstalling the package.

Building Microcontainers

To build a "hello world" container with smith:

  1. Create a new directory and cd to it
mkdir cat
cd cat
  1. Create a smith.yaml file with the following contents:
package: coreutils
paths:
- /usr/bin/cat
cmd:
- /usr/bin/cat
- /read/data
  1. Create the rootfs directory. Smith will put the contents of the ./rootfs directory into the root directory of the image.

mkdir rootfs

  1. Create the read directory under rootfs

mkdir rootfs/read

  1. Create the file data under rootfs/read with the following content:

Hello World!

  • invoke smith with no parameters:

smith

Your image will be saved as image.tar.gz. You can change the name with a parameter:

smith -i cat.tar.gz

Smith has a few other options which can be viewed using "--help"

smith --help

Build Types

Smith can build from local rpm files or repositories. You can change the yum config by modifying your /etc/mock/default.cfg.

Smith can also build directly from oci files downloaded via the download command, or an oci directly from a docker repository. Simply specify either in your smith.yaml as package, for example:

package: https://registry-1.docker.io/library/fedora
paths:
- /usr/bin/cat
cmd:
- /usr/bin/cat
- /read/data

To build Smith directly from oci, the Docker command is slightly different:

smith(){
    docker run -it --rm \
    -v $PWD:/write \
    -v tmp:/tmp vishvananda/smith $@
}

Advanced Usage

For more detailed instructions on building containers, check out:

Upload

You can upload your image to a docker repository:

smith upload -r https://username:[email protected]/myrepo/cat -i cat.tar.gz

Images will be uploaded to the tag latest. You can specify an alternative tag name to use appending it after a colon:

smith upload -r https://registry-1.docker.io/myrepo/cat:newtag

It automatically uploads to registry-1.docker.io using docker media types. Otherwise it tries to upload using oci media types. If you want to upload to a private docker v2 registry that doesn't support oci media types, you can use the -d switch:

smith upload -d -r https://myregistry.com/myrepo/cat -i cat.tar.gz

You can specify a tag name to upload to by appending it to the name

Download

smith can also download existing images from docker repositories:

smith download -r https://registry-1.docker.io/library/hello-world -i hello-world.tar.gz

It will convert these to tar.gz oci layouts. The latest tag will be downloaded. To download an alternative tag, append it after a colon:

smith download -r https://registry-1.docker.io/library/hello-world:othertag

Contributing

Smith is an open source project. See CONTRIBUTING for details.

Oracle gratefully acknowledges the contributions to smith that have been made by the community.

Getting in touch

The best way to get in touch is Slack.

Click here to join the the Oracle Container Tools workspace.

Then join the Smith channel.

License

Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.

Smith is dual licensed under the Universal Permissive License 1.0 and the Apache License 2.0.

See LICENSE for more details.

smith's People

Contributors

bernardovale avatar brunoborges avatar crush-157 avatar termie avatar vishvananda avatar wcoekaer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

smith's Issues

Invalid Manifest when pushing to private registry

DEBU[0000] Uploading manifests/latest
ERRO[0000] Failed to upload image to http://user:pw@registry/repo/image: Put request returned invalid response 400:
{"errors":[{"code":"MANIFEST_INVALID","message":"manifest invalid","detail":{}}]}

I receive this error when I attempt to smith upload my image.tar.gz to the registry. Taking a peek at index.json, this line sticks out:

{"schemaVersion":2,"manifests":[{"mediaType":"application/vnd.smith.spec+json",},{"mediaType":"application/vnd.oci.image.manifest.v1+json","com.oracle.smith.version":"1.1.2.12.23acefc","org.opencontainers.image.created":"2017-10-31T21:19:31Z","org.opencontainers.image.ref.name":"latest"},"platform":{"architecture":"amd64","os":"linux"}}]}

Any idea why that mediaType is pointing to v1?

Wrong annotation keys

Smith makes index.json files with the obsolete annotation keys org.opencontainers.ref.name and org.opencontainers.created. Those keys were renamed before the final 1.0 release of the image spec (see opencontainers/image-spec@8ec42b7).

This is a critical bug, because skopeo version 0.1.23 now expects the 1.0 image spec format, and it is current in the rhel7 extras repo.

smith breaks standard commands on Alpine

I'm running smith inside an Alpine Docker container. After running smith some of the standard commands are broken, like sh and ls, but pwd still works.

~ # ls
Error loading shared library libssl.so.39: No such file or directory (needed by /bin/ls)
Error loading shared library libcrypto.so.38: No such file or directory (needed by /bin/ls)
Error relocating /bin/ls: X509_NAME_get_text_by_NID: symbol not found

This does not occur in a Debian container.

Building from local docker repo

Hello I'm trying to see if this tool can shrink my oracle-slim-7 docker image(I believe it's OCI compliant).

I have the image built locally but I don't understand how I can use smith to turn it into a microcontainer. Do I need to set up a whole registry to do this?

Hardcoded paths to ldconfig and chmod

I tried this tool on an Alpine based image that contains one fully statically linked executable that I would like to extract. The smith.yaml file looks like this:

type: oci
package: remarkify.gz.tar
paths:
  - /usr/local/bin/remarkify
cmd:
  - /usr/local/bin/remarkify

When I run the tool I get this output:

INFO[0000] Building in /tmp/smith-build-443054379
INFO[0000] Installing package
INFO[0000] Installing package remarkify.gz.tar
WARN[0000] pigz binary not found, falling back to internal gzip
INFO[0004] Removing /tmp/smith-unpack-0
WARN[0010] chmod failed:
WARN[0010] Could not make paths readable: Error starting chmod -R go+rX /usr/local/bin/remarkify: fork/exec /usr/bin/chmod: no such file or directory
WARN[0010] ldconfig failed:
INFO[0010] Failed to install remarkify.gz.tar: Error starting ldconfig -v -N -X /: fork/exec /usr/sbin/ldconfig: no such file or directory
INFO[0010] Removing /tmp/smith-build-443054379

The executable is there:

# ls /tmp/smith-unpack-0/usr/local/bin/remarkify
/tmp/smith-unpack-0/usr/local/bin/remarkify

Doing the same thing but with a Debian based image gives the same result. I also tried with a Fedora based image, this works. To me it looks like it tries to find /usr/bin/chmod and /usr/sbin/ldconfig inside the unpacked image. These two paths don't exist in the Debian or Alpine based images. In those images these tools are located in /bin/chmod and /sbin/ldconfig.

I also tried the final example on [1], same issue.

[1] https://hackernoon.com/how-to-build-a-tiny-httpd-container-ae622c37db39

Credentials not used for upload to private Docker repository with basic auth

When using smith upload -d -r https://jruser:[email protected]/reponame, the blobs POST is made without the username and password credentials. The private registry I'm using (artifactory) rejects the attempt with a 403 code because of the missing credentials.

(When I changed PrepPutObject to use req.SetBasicAuth, artifactory responded with a 404 due to a bug of its own, but that's beside the point.)

The code seems to be customized to the public Docker registry and its associated token server, and not to private ones which use the recommended basic auth setup.

Created image has invalid date

I created an image using smith. I uploaded it to a local docker registry using the -d flag to convert it to a Docker image. I then pulled down the image locally and it works perfectly fine. Running docker images shows that the image was created 292 years ago. I also pushed the image to my private GitLab instance, which contains a built-in Docker registry. Viewing the registry in GitLab resulted in a 500 page and an exception was thrown due to an invalid date.

One can argue that GitLab should be able to handle this since it works fine in Docker, both locally and in my local registry. But I think it would be really nice if smith could set the date correctly, if possible.

I've already reported an issue to GitLab: https://gitlab.com/gitlab-org/gitlab-ce/issues/34988.

Always get reponame must be specified when trying to upload

I have followed the instructions and everything is working smoothly until it comes time to upload. For the life of me I can't figure out why smith can't parse the repo name. I copied and pasted the exact command in the example.

/home/nwellinghoff/smith/smith-httpd# smith upload -r \

https://nwellinghoff:@registry-1.docker.io/nwellinghoff/smith-httpd
-i smith-httpd.tar.gz
WARN[0000] pigz binary not found, falling back to internal gzip
ERRO[0000] Failed to upload image to https://nwellinghoff:@registry-1.docker.io/nwellinghoff/smith-httpd: Reponame must be specified

I am able to upload the image manually by doing a
cat smith-httpd.tar.gz | docker import - nwellinghoff/smith-httpd:latest
then
docker push nwellinghoff/smith-httpd

but it would be nice if the smith command worked as well.

No Documentation

I'm currently working on trying to transition some containers to using Smith. I am interested in the project and am looking for documentation which would help with not having to read through the source (my current solution).

Is there a source of documentation that I can refer to? If so, it may be worth linking to it in the README.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.