openshift / source-to-image Goto Github PK
View Code? Open in Web Editor NEWA tool for building artifacts from source and injecting into container images
License: Apache License 2.0
A tool for building artifacts from source and injecting into container images
License: Apache License 2.0
Rightnow the STI source code is very 'sti-centric'. If I want to add my custom build type (heroku build-pack, generic Docker build, etc...) I can't easily plug this into the existing STI codebase. We should have better interfaces and have factories for this:
type Builder interface {
Build()
}
type Downloader interface {
Download()
}
What we have right now is something like:
type buildHandlerInterface interface {
cleanup()
setup(required []api.Script, optional []api.Script) error
determineIncremental() error
Request() *api.Request
Result() *api.Result
saveArtifacts() error
fetchSource() error
execute(command api.Script) error
wasExpectedError(text string) bool
build() error
}
and so on... the STI will be still 'default' build type, but we can also support different build models in the future.
This issue requires massive refactoring, so I don't think we can make it real before GA, but I want to keep this issue opened here that we don't forget about it.
Hi, I want to know if sti support the building from a war file to an image. I only find the examples about building from the source code into an image. But I found in this page : https://github.com/openshift/openshift-pep/blob/master/openshift-pep-013-openshift-3.md
in the " Build " section , it says:
"Example: post a WAR file to a source-to-images build that results in that WAR being deployed"
Could you give me some suggestions that how to build a war file to an image by sti build ? thank you!
Currently, we use ENV STI_SCRIPTS_URL image://...
in our Docker images. This variable points into default location for the STI scripts used for the STI build. I think it would be nice to switch this from the ENV variable to LABEL instructions. There are several benefits of doing that:
openshift.io/sti-scripts-url
)sti build
using source from the local filesystem and an image without sti-helper
fails silently when copying the source to upload/src
, as the upload directory does not exist yet.
The error output from cp
is suppressed, even with --loglevel=3
.
I am using sti from the command line and in most cases, I notice that it "hungs".
When I am using log level 3 I see that the build output always hungs at exactly the same line.
What is strange though is that when I check the docker logs, I can see that the assemble script has successfully finished and also that the container has exited gracefully. The target image of course is never created.
The last couple of days I've made so many attempts and only two of them were successful (I haven't done anything different).
Per @smarterclayton comment on my recent PR, I'm opening issue to verify if following comment is still valid (from https://github.com/openshift/source-to-image/blob/master/test/integration/images/sti-fake/Dockerfile)
RUN mkdir -p /sti-fake/src && \
# FIXME: The 'default' user has conflicting UID (1000). This should be fixed in
# how source-to-image handles UID's during isolation.
#
deluser default
if you only supply one or the other, the resulting image fails to build.
run STI and provide no tag on the image name, for an image repo where there is no "latest" tag, result is STI hangs forever in "pulling image"... seems like it must not be handling an error correctly:
$ docker pull ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift
FATA[0005] Tag latest not found in repository ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift
$ sti build https://github.com/bparees/session-app ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift badout --loglevel=5
I0506 18:54:38.631805 07628 docker.go:173] Pulling image ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift
(never errors/returns)
Note that if you provide an actual invalid tag, it does immediately error out. Not sure it's actually valid to have a repo with no "latest" tag in it?
$ sti build https://github.com/bparees/session-app ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift:badtag badout --loglevel=5
I0506 18:55:08.986159 08143 docker.go:173] Pulling image ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift:badtag
I0506 18:55:12.403179 08143 docker.go:177] An error was received from the PullImage call: Tag badtag not found in repository ce-registry.usersys.redhat.com/jboss-webserver3/tomcat8-openshift
Run hack/test-go.sh <pkg>; go tool cover -html=_output/go/tmp.out
for every package to look at coverage.
From the build log (a previously build test
image exists):
$ sti build server accursoft/ghc-network test --loglevel=3
download.go:112] Using image internal scripts from: image://opt/sti/save-artifacts
build.go:80] Clean build will be performed
If I specify the same scripts with -s
, it re-uses the build artifacts.
STI's exit code is always 0. This makes it difficult to determine if the build failed, for example, unless you scrape the invocation's output. I'd recommend that we exit non-zero when appropriate.
I think this will help to have much cleaner user experience with STI command line, where the user don't need to know all possible options/docker image names/etc to just trigger the STI build.
I'm proposing that this command:
sti build https://github.com/user/repo
will clone the GIT repo, then check if there is 'sti.json' (naming TBD) file and prefill the STI build options automatically (reusing the --use-config
logic)
$ git clone https://github.com/openshift/source-to-image
$ cd source-to-image/
$ make
docker build -t test .
Sending build context to Docker daemon 14.34 kB
Sending build context to Docker daemon
Step 0 : FROM openshift/base-centos7
# Executing 4 build triggers
Trigger 0, COPY ./.sti/bin/ /usr/local/sti
Step 0 : COPY ./.sti/bin/ /usr/local/sti
---> Using cache
Trigger 1, COPY ./contrib/ /opt/openshift
Step 0 : COPY ./contrib/ /opt/openshift
INFO[0000] contrib/: no such file or directory
Makefile:5: recipe for target 'build' failed
make: *** [build] Error 1
Adding a contrib folder works. If adding the folder as part of create is the solution, I can code this.
Before I do this, have I misunderstood something though?
I am trying to use the sti and provide a url to the sti scripts as follows:
sti build https://github.com/iocanel/test-sti.git fabric8/karaf-2.4 fabric8/test-sti -s file:/somepath/.sti/bin
The error I am getting is:
/bin/bash: assemble: No such file or directory
An error occured: Container exited with exit code: 127
It seems that it's not possible to redirect output of sti
command to somewhere:
$ sti build --loglevel=5 --forcePull=false https://github.com/goldmann/openshift-eap-examples --contextDir=custom-module 4e42b3029b98 test-jee-app > ~/sti/test.log
-bash: /home/goldmann/sti/test.log: No such file or directory
During investigation of a very big image sizes produced by STI I have found that STI ignores completely .gitignore or .dockerignore and puts every folder into the container it is building.
We have some temporarily files which are ignored in both git and dockerignore with significant weight(about 125MB) adding quite a lot of unnecessary data to production image. I cannot find a way to make STI ignore those files. Would be good if STI can read .dockerignore or .gitignore to get this files and do not put them inside the container.
Hi,
I was trying to play a little bit with STI to test some stuff with geard. I didn't manage to make if work in OS X or Fedora 20. When I run sti build test_sources/applications/html pmorie/fedora-mock sti_app
this is what I get:
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): localhost
INFO:sti.cmd.builder:Building new docker image
Traceback (most recent call last):
File "/usr/local/bin/sti", line 10, in <module>
sys.exit(main())
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sti/cmd/builder.py", line 362, in main
builder.main()
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sti/cmd/builder.py", line 348, in main
self.build(working_dir, build_image, source, is_incremental, user, app_image, env_str)
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sti/cmd/builder.py", line 245, in build
img = self.build_deployable_image(image_name, build_dir, tag, env_str, incremental_build)
File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sti/cmd/builder.py", line 229, in build_deployable_image
img, logs = self.docker_client.build(tag=tag, path=context_dir, rm=True)
ValueError: too many values to unpack
I don't really know Python so digging into the code to find out myself isn't that easy. Any help is appreciated, thanks.
Is it really necessary to have 'sti' in the package name?
What we do now:
import "github.com/openshift/source-to-image/pkg/sti/...."
What we should do:
import "github.com/openshift/source-to-image/pkg/...."
This comes from here: openshift/origin#1119 (diff). Generally speaking the idea is to stress test the building process itself without actually producing/committing image. STI currently requires that image and we should not especially when that PR gets into origin.
Switch hack/build-release.sh to use the same as openshift/origin
Currently, when customizing the STI scripts, one must take the original script and then add to it.
It would be nice if we supported some kind of additive mechnaism. Either by indicating it in the script, or by having a script of the same name with a number (eg: ##-assemble [happens before assemble] and assemble-## [happens after assemble])
As a go newbie, it took me about half an hour to work out where the sti
command was being installed. I eventually found it in _output
by reading the hack
sources.
Is this normal, or some kind of go convention? Should it be added to the installation instructions?
Since you split the env command-line argument on ',', the environment variable values themselves can't contain commas. I hit this trying to pass "-P some-repo,some-other-repo" to maven via MAVEN_ARGS
.
Verbosity wouldn't need to be passed around and more levels can be supported.
It would be useful to have the WORKDIR
set to the path where your application code lives in images created with STI.
That way, it would be possible to call scripts you might have in your code base passing its relative path to docker exec
or osc exec
.
docker exec -it <container> bash
would start in the "right" directory.
If there is no opposition to this idea, I could implement this myself given a pointer to where to start ๐
pkg/create/templates/docker.go?
Currently everything is sent to stderr
. The console output should be sent to stdout
instead. Only errors should be sent to stderr
.
Since the extraction of types.go
into their own package api
there a couple of fields unnecessarily public, this includes:
WorkingDir
Incremental
ExternalRequiredScripts
ExternalOptionalScripts
Currently we don't have ANY integration tests for onbuild strategy. We should have a couple at least!
Per this comment, we should review the code in cmd/sti/main.go#newCmdBuild and extract as much as we can from there into separate package that we'll be able to reduce the amount of duplication in code in S2I and origin.
It came up in conversation that it might be nice if the sti "program" accepted a flag that would: cause the resulting built Docker image to stay running instead of actually being a created image.
There may be cases where I want to test the result of my STI process locally without having to push the Docker image somewhere.
I guess this flag would essentially cause "run" to be executed immediately after "assemble" inside the instance of the builder image.
It's not a huge time savings, but it could potentially make a few other automation tasks easier... maybe?
My base image contains the following:
ENV STI_SCRIPTS_URL file:/opt/sti
it also adds the sti scripts under /opt/sti/
I am trying to use sti as follows:
sti build https://github.com/iocanel/test-sti.git fabric8/karaf-2.4 fabric8/test-sti
and get the error:
An error occured: Scripts download failed
I'm currently building sti-python image and found that STI swallows error messages from assemble script.
$ sti build 3.3/test/klein-test-app/ openshift/python-33-centos7 python-sample-app --forcePull=false
I0330 17:58:31.082271 19058 sti.go:371] ---> Installing application source
I0330 17:58:31.083148 19058 sti.go:371] ---> Building your Python application from source
I0330 17:58:31.083265 19058 sti.go:371] python setup.py install #develop
vs.
$ sti build 3.3/test/klein-test-app/ openshift/python-33-centos7 python-sample-app --forcePull=false --loglevel=1
I0330 17:58:46.910338 19213 sti.go:111] Building python-sample-app
I0330 17:58:46.916831 19213 sti.go:181] Using assemble from image:///usr/local/sti
I0330 17:58:46.916850 19213 sti.go:181] Using run from image:///usr/local/sti
I0330 17:58:46.916856 19213 sti.go:181] Using save-artifacts from image:///usr/local/sti
I0330 17:58:46.916862 19213 sti.go:119] Clean build will be performed
I0330 17:58:46.916867 19213 sti.go:130] Building python-sample-app
I0330 17:58:46.916877 19213 sti.go:313] No .sti/environment provided (no evironment file found in application sources)
I0330 17:58:47.090252 19213 sti.go:371] ---> Installing application source
I0330 17:58:47.091183 19213 sti.go:371] ---> Building your Python application from source
I0330 17:58:47.091352 19213 sti.go:371] python setup.py install #develop
E0330 17:58:47.193323 19213 sti.go:389] error in klein-test-app setup command: ('Invalid module name', 'klein-test-app')
I0330 17:58:47.351302 19213 main.go:202] An error occurred: non-zero (13) exit code from openshift/python-33-centos7
The swallowed line is:
E0330 17:58:47.193323 19213 sti.go:389] error in klein-test-app setup command: ('Invalid module name', 'klein-test-app')
when i testing the command : stil build , according to the readme.md I got the follwing error:
sti build git://github.com/bparees/openshift-jee-sample openshift/wildfly-8-centos test-jee-app
I0317 14:39:07.592323 26821 sti.go:111] Building test-jee-app
Cloning into '/tmp/sti424670445/upload/src'...
remote: Counting objects: 29, done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 29 (delta 1), reused 29 (delta 1), pack-reused 0
Receiving objects: 100% (29/29), 22.84 KiB | 12.00 KiB/s, done.
Resolving deltas: 100% (1/1), done.
Checking connectivity... done.
F0317 14:39:55.761126 26821 main.go:201] An error occurred: non-zero (13) exit code from openshift/wildfly-8-centos
anyone could give me some suggestions? thank you!
I would like the built image to run with a more restricted user than the one which performed the build. This could be supported by running assemble
as a non-default user, or by changing the image's default user after assembly.
It can be done now with docker run -u
, but it would be nice to support this automatically.
When trying to provide incremental build, I get the following result:
......
E0415 20:49:30.409228 17401 tar.go:158] Error reading next tar header: io: read/write on closed pipe
W0415 20:49:31.287679 17401 sti.go:125] Error saving previous build artifacts: timeout waiting for tar stream
......
The whole log of the incremental build is available at http://pastebin.test.redhat.com/276603
Currently the run
script is required. I would like it to have optional. This would let it reuse the CMD
command which was specified in the image we could possibly extend by creating the builder image.
Imagine the jboss/wildfly
image. This is upstream image has already defined a CMD
command. I don't see a reason to duplicate it just to satisfy the requirement of having the run
command. In my case I would just repeat what I have already defined, but this is unnecessary + adds another file and possibly could confuse people.
Without a run
script, the CMD
or ENTRYPOINT
from the extended image should be used as-is. In case the run script exist - it should override those.
Currently adding -v
flag to hack/test-integration.sh
shows only the logs from the integration tests itself. Whereas when searching for test error I'd like also to see STI internal which you turn on with --loglevel=3
.
Hi,
I'm wondering am I holding it wrong or have I found a problem with sti. I have the following images:
Base image -> ruby x version and all other deps like imagemagick and etc. Based on Dockerfile. Built by docker build in seconds.
Test image -> build by STI based on base image. Contains all gems from gemfile.
Staging image -> build by STI based on base image. Excludes test and development gems from gemfile, precompiles assets for staging.
Production image -> build by STI based on base image. Excludes test and development gems from gemfile, precompiles assets for production.
Sti allowed me to make process of building staging and production image significantly faster but at the end of the day I need to push those images to registry(docker hub). And that is where I found a problem.
If I look at our staging/production image I have a bunch of layers from base image and one giant layer (442MB) from sti:
IMAGE CREATED CREATED BY SIZE
6b2883b78afc 20 minutes ago /bin/sh -c tar -C /tmp -xf - && /tmp/scripts/ 442.5 MB
I'm ok with pushing this layer once BUT it gets regenerated EVERY time I run it. So even if building of the image is quite fast pushing it to the registry and then pulling it takes more time especially as number of servers grows.
Is there a way to cache this layer somehow and build on top of it ?
I would like to include STI scripts inside the image. Seeing as the image and sti scripts will likely be developed together, this seem like a reasonable requirement.
https://github.com/openshift/source-to-image/blame/master/README.md#L92
If a save-artifacts script exists, a prior image already exists, and the --clean option is not used, the workflow is as follows:
I'm not sure how this line should read, as the operation seems to have been flipped too? --clean=true is --incremental=false - but should probably update README. Would be good to also
Include builder image, source repo, date, time, image tag
My git repository is private, how to build from private git repository?
Please move the contents from the readme.md over to the real site
http://docs.openshift.org/latest/creating_images/sti.html
This way all your good work can be seen by the rest of the world
Hi is there a reason why last release is linux only? All previous ones where available for mac and windows too.
As the output of assemble is most like the most important thing end-users will want to see we should display it by default (or have special option for that --output-assemble?).
In Origin we should show this output, nothing else is important for the end user (if the end user is not debugging issue in STI itself)
Since just recently STI allows using arbitrary images for doing build, the problem arises when one of the base images includes ONBUILD ENTRYPOINT instruction. This results in an image where the .sti/bin/run
command will be appended after the ENTRYPOINT. This results in some weird behavior, depending what was specified as the ENTRYPOINT
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.