Hello
I tried to make the sysbox runtime work for our Jenkins setup but it ends up crashing (the whole Docker daemon).
System information
- AWS EC2 Instance
- Ubuntu 18.04 Server (
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200112 (ami-0b418580298265d5c)
)
- Upgraded kernel as documented via backports (
uname -r
= 5.3.0-40-generic
)
Jenkins Volumes (run directly on host using regular Docker runtime)
/var/run/docker.sock:/var/run/docker.sock
jenkins-workspace:/var/jenkins_home
Jenkins Pipeline
pipeline {
...
agent {
docker {
image 'censored:latest'
args '--runtime=sysbox-runc -v jenkins-maven-repository:/home/jenkins/.m2'
}
}
...
}
For completness, this is what Jenkins tries to run:
docker run -t -d -u 1000:1000 --runtime=sysbox-runc -v jenkins-maven-repository:/home/jenkins/.m2 -w /var/jenkins_home/jobs/maven-pipeline-ng/workspace --volumes-from 9f65cddc6f6cf5951e78cc7bbd67f64040b5877919c8860d0d550a1c6a3905cf -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** censored:latest cat
So basically the Jenkins container (which is a regular one) should start pipeline agents with sysbox runtime -> Agents will be siblings of Jenkins but be able to have isolated Docker inside
This leads to the following error however:
Failed to run image 'censored:latest'. Error: docker: Error response from daemon: OCI runtime create failed: container_linux.go:364: starting container process caused "process_linux.go:474: container init caused \"process_linux.go:441: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2020-03-06T13:50:22Z\\\\\\\" level=fatal msg=\\\\\\\"dial unix /var/run/docker/libnetwork/c0d26181f7af.sock: connect: connection refused\\\\\\\"\\\\n\\\"\"": unknown.
The whole server (not only Docker, literally everything) starts to behave strange afterwards and the only solution is to perform a reboot via AWS console (doesn't really work too). I.e. running reboot -h now
results in:
Failed to connect to bus: Connection refused
Failed to open /dev/initctl: No such device or address
Failed to talk to init daemon.
Manually creating censored:latest
containers within the Jenkins container works and Docker within such a container works as well when done without -v jenkins-maven-repository:/home/jenkins/.m2 -w /var/jenkins_home/jobs/maven-pipeline-ng/workspace --volumes-from 9f65cddc6f6cf5951e78cc7bbd67f64040b5877919c8860d0d550a1c6a3905cf -e
flags - so the problem must be with these.
Currently access to the server is not possible (stuck and un-stoppable) so I cannot further break down which of the flags alone is a problem (maybe --volumes-from
?). I will try once I regain access to the server, but hopefully the provided information is enough anyways.
Thanks.