Running out of loopback devices about dind HOT 31 CLOSED

I also hit a problem with the same symptoms. I can see in the output of losetup -l that the number of used loop devices continues to grow if I start nested docker daemons and don't stop them gracefully (in my case containers are always killed with SIGKILL).

I haven't been able to fix the leaking but I was able to mitigate it by an addition to the containers. What I found is that sometimes, the next available loop device would be a higher number than the existing device nodes /dev/loop* within the container, causing docker to give the "There are no more loopback device available" error. The error would occur much earlier than reaching the actual max number of loopback devices configured in the kernel.

I was able to fix that particular case by invoking this scriptlet in the container before docker is started, which ensures two free loopback devices exist:

#!/bin/bash
ensure_loop(){
  num="$1"
  dev="/dev/loop$num"
  if test -b "$dev"; then
    echo "$dev is a usable loop device."
    return 0
  fi

  echo "Attempting to create $dev for docker ..."
  if ! mknod -m660 $dev b 7 $num; then
    echo "Failed to create $dev!" 1>&2
    return 3
  fi

  return 0
}

LOOP_A=$(losetup -f)
LOOP_A=${LOOP_A#/dev/loop}
LOOP_B=$(expr $LOOP_A + 1)

ensure_loop $LOOP_A
ensure_loop $LOOP_B

Maybe it'd be possible to add something like that ^ into wrapdocker, if it helps anyone.

from dind.

The solution for me appears to be simply run:

service docker stop

at the end of my build script to gracefully stop the docker daemon started by wrapdocker. Then the number of loopback devices as per losetup -a does not grow (by 2) every drone build.

from dind.

I think I've figured out what I need to do. I need to umount /var/lib/docker when I exit the container. I wonder what the most reliable way to do this would be (depending on .bash_logout, or some such seems risky).

from dind.

Confirmed: Unmounting /var/lib/docker on exit fixes the error.

Question: Is this a more general issue that just shows it head early with devicemapper? If you mount something inside the docker container, but you don't unmount it before you destroy the container, does that mountpoint live somewhere on the host forever? Just wondering if the cgroups mounts also need cleaning up on exit?

from dind.

I am seeing the same issue when using wrapdocker inside of a drone build container. The idea is that my build process outputs a docker container. After about ten builds I get:

[error] attach_loopback.go:42 There are no more loopback device available.
loopback mounting failed

But I am using an unmodified version of wrapdocker so I'm unsure @d11wtq whether your solution is right for me. Again, only restarting my drone machine temporarily fixes the problem. The minimal .drone.yml file that reproduces this is:

image: ...
script:
    - ./.drone/build.sh

Where build.sh is:

#!/bin/bash
wrapdocker &
sleep 5
docker build ...

Any thoughts? Thanks.

from dind.

I did encounter a situation in which service docker stop results in:

* Stopping Docker: docker
start-stop-daemon: warning: failed to kill 1067: No such process
1 pids were not killed
No process in pidfile '/var/run/docker-ssd.pid' found running; none killed.

In this case I use the following for graceful shutdown of docker -d:

kill -15 `ps ax | grep "docker -d" | grep -v grep | awk {'print $1'}`

from dind.

I'm using this to stop wrapdocker :
start-stop-daemon --stop --pidfile "/var/run/docker.pid"

it seems the pid used by service docker is not always the same as wrapdocker :)

from dind.

@rohanpm it serves me well, thank you

from dind.

@rohanpm Works for me too, thank you! I quote your solution here: http://paislee.io/how-to-build-and-deploy-docker-images-with-drone/

from dind.

I'll close this issue since it appears to be solved. But feel free to re-open/comment if it's not the case!

Thanks,

from dind.

@rohanpm I've used your script in moby/moby#9117 - if you can confirm thats ok, that would be great :)

from dind.

@SvenDowideit, sure, no problem. (I saw the later discussion about maybe doing the same thing from golang code too.)

from dind.

ya :) much +1 to getting something done!

from dind.

I think this should be reopened and @rohanpm's contribution added to wrapdocker. Running dind on a CentOS 7 host results in the same error due to not enough loopback devices.

from dind.

Oh, sure. I'll be happy to look at a PR for this. Thanks a lot!

from dind.

I'm having the same issue with running dind on Centos7, it can't seem to actually start the docker daemon within the container.

from dind.

The .drone/build.sh script could be enhanced by using the trap command of the bash script. This error handler will be called before the bash script exits. So this could be a good place to clean up loop devices.

#!/bin/bash

handle_error() {
  echo "FAILED: line $1, exit code $2"
  echo "Remove loop device here ...."
  exit 1
}

trap 'handle_error $LINENO $?' ERR 

set -e
# start your build here ...

Just my two cents.

from dind.

I'm reopening but I don't use CentOS so I don't know how to help. I hope someone has a better idea!

from dind.

I am currently still haveing issues with running out of "Running out of loopback devices " on ubuntu 14:04 LTS with docker 1.5 on the host machine.

I hhave tried the following.

#66

Not sure where I am going wrong with this. Please help.

from dind.

This happens on the standard amazon linux AMI on EC2 as well.

from dind.

I am seeing this issue on CoreOS 668.2.0. The script by @rohanpm does not work in my instance. I get the following output:

root@slave6:~# ensure_loop $LOOP_A
/dev/loop1 is a usable loop device.
root@slave6:~# ensure_loop $LOOP_B
/dev/loop2 is a usable loop device.

from dind.

trap works for me alexanderilyin/docker-teamcity-agent@ef0cf6e

from dind.

Just to add, on CoreOS 717.3.0 this seems to work just fine with loopback devices starting with /dev/loop0...

from dind.

Problem still happening on CoreOS

from dind.

+1 Debian, Docker 1.6.2

from dind.

Same problem on CentOS 7.1
I started a container where I started another container. This problem occurred when I shut down the first container without having shut down the one inside.

from dind.

The first DinD container seems to start fine every time now,
but when I try to start a 2nd DinD I get the "no more loopback devices" again.

from dind.

I added a retry-loop and at the 3rd attempt also both containers are and stay running.

from dind.

For those of you using DinD for CI/testing, please have a look at this new blog post!

from dind.

@jpetazzo thx for post. After severals weeks of pain and suffering I've ended with exposing socket. I've just wanted to hear that it is not the worst solution from someone else.

from dind.

There is now an official docker:dind image upstream! I invite you to test it, since it is actively maintained. Thank you!

from dind.