Comments (20)
Make sure that things under /export
are owned by the galaxy user (uid 1450 I think). uwsgi and most other things are running as that user, so that should allow it to do whatever it needs to.
from docker-galaxy-stable.
The directory is created entirely by the the docker container, it doesn't exist at all prior to executing docker run
and many directories do get created.
from docker-galaxy-stable.
@lparsons can you provide me with more informations? Docker version? kinematic? Galaxy Docker version? I haven't seen this before.
Also can you look at the permissions for /export/galaxy-central/database/
?
Thanks and sorry for the trouble!
Bjoern
from docker-galaxy-stable.
I ran into this error yesterday with boot2docker/docker version 1.6.1. Today, there was an update to boot2docker/docker version 1.7 and I'm running into the same problem:
$ boot2docker version
Boot2Docker-cli version: v1.7.0
Git commit: 7d89508
$ docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): darwin/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64
I am not using Kinematic. I don't know how to determine the Galaxy Docker version, but it was freshly downloaded yesterday and again today.
from docker-galaxy-stable.
Permissions
Docker Container
$ docker exec grave_fermat ls -la /export
total 4
drwxr-xr-x 1 1000 staff 238 Jun 30 15:02 .
drwxr-xr-x 102 root root 4096 Jun 30 15:02 ..
drwxr-xr-x 1 1000 staff 1020 Jun 17 22:55 .distribution_config
drwxr-xr-x 1 1000 staff 204 Jun 29 17:23 galaxy-central
drwxr-xr-x 1 1000 staff 102 Jun 29 17:23 postgresql
drwxr-xr-x 1 1000 staff 68 Jun 17 22:56 shed_tools
drwxr-xr-x 1 1000 staff 102 Jun 29 17:23 var
$ docker exec grave_fermat ls -la /export/galaxy-central
total 0
drwxr-xr-x 1 1000 staff 204 Jun 29 17:23 .
drwxr-xr-x 1 1000 staff 238 Jun 30 15:02 ..
drwxr-xr-x 1 1000 staff 1088 Jun 29 17:23 config
drwxr-xr-x 1 1000 staff 272 Jun 17 05:54 display_applications
drwxr-xr-x 1 1000 staff 1904 Jun 17 22:55 tool-data
drwxr-xr-x 1 1000 staff 68 Jun 17 22:56 tool_deps
Boot2Docker VM
docker@boot2docker:/Users/lparsons/Documents/projects/galaxy_tools/galaxy_storage$ ls -la
total 0
drwxr-xr-x 1 docker staff 238 Jun 30 15:02 ./
drwxr-xr-x 1 docker staff 272 Jun 29 17:23 ../
drwxr-xr-x 1 docker staff 1020 Jun 17 22:55 .distribution_config/
drwxr-xr-x 1 docker staff 204 Jun 29 17:23 galaxy-central/
drwxr-xr-x 1 docker staff 102 Jun 29 17:23 postgresql/
drwxr-xr-x 1 docker staff 68 Jun 17 22:56 shed_tools/
drwxr-xr-x 1 docker staff 102 Jun 29 17:23 var/
OSX Host
$ ls -la
total 0
drwxr-xr-x 7 lparsons staff 238 Jun 30 11:02 ./
drwxr-xr-x 8 lparsons staff 272 Jun 29 13:23 ../
drwxr-xr-x 30 lparsons staff 1020 Jun 17 18:55 .distribution_config/
drwxr-xr-x 6 lparsons staff 204 Jun 29 13:23 galaxy-central/
drwxr-xr-x 3 lparsons staff 102 Jun 29 13:23 postgresql/
drwxr-xr-x 2 lparsons staff 68 Jun 17 18:56 shed_tools/
drwxr-xr-x 3 lparsons staff 102 Jun 29 13:23 var/
from docker-galaxy-stable.
I'm thinking this is the same issue I had with sharing /export between my VirtualBox VM (either Kitematic Boot2Docker or Vagrant Ubuntu) and my Windows host. Virtual machine shared folders commonly have problems keeping uid/gid straight between the host and guest. As you see in your B2D VM's ls, the shared folder is mapped with docker:staff (1000:50). Your Docker container has the staff group but not the docker user. But that doesn't really matter, because those things should be owned by the galaxy user (1450 by default) and postgres (default varies). I just submitted a patch to Bjorn that allows you to configure what the postgres user's UID/GID will be in the docker container (default 1550). Then, in the VM, you make sure to map /export/postgresql with 1550:1550, and /export/galaxy-central and /export/shed_tools with 1450:1450. /export/var should be mounted with root:root. In Kitematic on Windows, I couldn't tell it what UID/GID to use for the mounts, so I ended up having to use Vagrant Ubuntu and now it's working.
from docker-galaxy-stable.
xref: #71
from docker-galaxy-stable.
@lparsons can you check if #71 is of any help for you?
from docker-galaxy-stable.
@bgruening I've been having the same problem and unfortunately #71 doesn't solve it for me.
I believe the root cause is Docker and boot2docker/boot2docker#581 . It seems that it is not possible to change the owner of the /export directories from within the VM (or a container running in the VM) and so postgres refuses to start. If I ssh into the container (see Problem #3) I can see the directories properly mounted under /export, but they are all owned by docker:staff and any attempt to sudo chown
them results in a "permission denied" error.
Another (almost) related problem is that the /export/galaxy-central/database/*
directory hierarchy is not being created which causes other odd "permission denied" errors when Galaxy tries to start. The relevant stack trace is:
Traceback (most recent call last):
File "lib/galaxy/webapps/galaxy/buildapp.py", line 64, in paste_app_factory
app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs )
File "lib/galaxy/app.py", line 38, in __init__
self.config.check()
File "lib/galaxy/config.py", line 618, in check
self._ensure_directory( path )
File "lib/galaxy/config.py", line 599, in _ensure_directory
raise ConfigurationError( "Unable to create missing directory: %s\n%s" % ( path, e ) )
ConfigurationError: Unable to create missing directory: /export/galaxy-central/database/files
[Errno 13] Permission denied: '/export/galaxy-central/database'
Everything else under /export is being populated correctly so I don't know what is special about /export/galaxy-central/database. If I create the directories before running Docker at least I get to the "Galaxy won't start because Postgres isn't running" problem. Otherwise Galaxy won't start due to the missing /export/galaxy-central/database/*
directories.
from docker-galaxy-stable.
@ksuderman which UID/GID does your postgres user have? Are you running :dev
or :15.07
? If not can you try one of these new versions?
from docker-galaxy-stable.
Sorry @bgruening, I should have mentioned that I have been basing my images on galaxy-stable:dev. I just tried 15.07 image and the problems are still there. The postgres user is 1550:1550 and the galaxy user is 1450:1450 (see below).
However, I have made progress.
To "fix" (hack around) the "permission denied" problems I cloned the docker-galaxy-stable repo and changed the GALAXY_UID and GALAXY_GID to 1000 (the docker UID in VirtualBox). This solves the problem of Galaxy not being able to create files in /export/galaxy-central/database/* directories.
I had assumed Postgres wasn't starting due to permissions problems in the /export/postgresql directory, and that is sort of correct. I can start postgres with service postgresql start
but it seems supervisord doesn't like the permissions on the /export directory. If I try to use the same command supvisord uses (from /etc/supervisor/conf.d/galaxy.conf) I get the following: (note postmaster refuses to run as the root user):
> sudo -u postgres /usr/lib/postgresql/9.3/bin/postmaster -D "/export/postgresql/9.3/main"
LOG: could not open configuration file "/export/postgresql/9.3/main/postgresql.conf": Permission denied
FATAL: configuration file "/export/postgresql/9.3/main/postgresql.conf" contains errors
If I try as the galaxy user (which has now has r/w access to /export) I get:
> sudo -u galaxy /usr/lib/postgresql/9.3/bin/postmaster -D "/export/postgresql/9.3/main"
FATAL: data directory "/export/postgresql/9.3/main" has group or world access
DETAIL: Permissions should be u=rwx (0700).
Due to the VirtualBox/Docker/OS X bugs it is not possible to change the permissions/owner of any files/directories in the shared volume. So my workaround is to simply add service postgresql start
to the top of the /usr/bin/startup
script.
With the above two changes I can now use shared volumes, however I don't think these "fixes" should be merged into docker-galaxy-stable. Another suggested work around is to remount /Users with NFS in the VBox VM, but I haven't explored that yet.
from docker-galaxy-stable.
@ksuderman this is a really great analysis. Thanks a bunch for sharing. I suppose these bugs are known to the Docker guys? I hope they fix it soon. Let me know if I can support you.
What we can try is to figure out if the container is running under VBox/OS-X and squeeze in your workaround - only in this case? What do you think?
@lparsons any opinion here?
Sorry that I can not help much as I'm missing the a proper test environment.
from docker-galaxy-stable.
Keith, in my case, on Windows, I originally tried to share everything with
the host. First I gave up on having the Postgres share persist in Windows
(due to the same issue you ran into with inability to change permissions)
and opted to use a semi-permanent Linux VM to handle that persistence. Then
I eventually gave up on having the galaxy-central directories shared with
the host as well because of symbolic link shenanigans. You might consider
using a Vagrant VM like me to set up a semi-permanent Linux VM for your
Docker environment (it takes the place of boot2docker).
NFS would fix the postgres problems for you most likely on OSX, but Winnfsd
would not fix the issue because everything maps to a single global UID/GID.
On Sun, Sep 6, 2015 at 6:41 PM, Björn Grüning [email protected]
wrote:
@ksuderman https://github.com/ksuderman this is a really great
analysis. Thanks a bunch for sharing. I suppose these bugs are known to the
Docker guys? I hope they fix it soon. Let me know if I can support you.What we can try is to figure out if the container is running under
VBox/OS-X and squeeze in your workaround - only in this case? What do you
think?@lparsons https://github.com/lparsons any opinion here?
Sorry that I can not help much as I'm missing the a proper test
environment.—
Reply to this email directly or view it on GitHub
#68 (comment)
.
from docker-galaxy-stable.
@bgruening Yes, this turns out to be a known problem in Docker (boot2docker actually) but it seems the root cause is the way VirtualBox does shared folders so I don't know how quickly a fix will be coming.
If anyone wants to test my hacks I've pushed an image to the Docker hub:
FROM ksuderman/galaxy-stable:osx
...
@chambm I will take look at Vagrant. Our use case right now is to develop something that can be used for course work and demos so I want to keep things as simple as possible for our users. Vagrant looks promising if it solves the permission problems.
from docker-galaxy-stable.
@bgruening I don't see an elegant solution (at least not near term). The idea of using the workaround in the VBox/OS-X case seems like a reasonable and practical solution if it works.
@chambm The Vagrant idea does indeed seem promising. My main goal is to be able to quickly setup clean Galaxy environments where I can test things like tool integrations and installations. Semi-permanent would be fine in those cases. Am I understanding correctly that you are creating a Vagrantfile
that creates a Linux VM, then vagrant ssh
in and then running the docker image with shared folder there?
from docker-galaxy-stable.
@lparsons Check out https://github.com/chambm/bingomics-galaxy/tree/master/vagrant - it's the Vagrantfile I use for developing and testing my lab's docker-galaxy flavor. It has a lot of stuff needed for a cluster-based configuration commented out since that's a bit resource intensive. But that is a possibility. There are some shell provisioning commands to prep a fresh VM and then it runs the docker image on vagrant up
. I log in with Putty rather than vagrant ssh, but if your vagrant ssh will work, that's even easier. My Bingomics docker image installs planemo, so it really makes tool development smooth (that being a relative term).
from docker-galaxy-stable.
Another work around, particularly if you just want some persistence during testing and development is to mount a directory from the VM. I.E.
> docker run -v /var/lib/galaxy:/export -d -p 8000:80 bgruening/galaxy-stable:dev
Then use docker-machine scp
to moves file in/out of the VM directory. As long as you don't delete the VM the /var/lib/galaxy directory will be persisted and available for multiple containers.
from docker-galaxy-stable.
@ksuderman @lparsons is there anything I can do here?
Can this be closed?
from docker-galaxy-stable.
I think you can close this. There is not much you can do other than document the problem and possible work arounds.
I've simply been mounting volumes I need to write to in the VirtualBox VM and then using docker-machine scp
or docker-machine ssh
to configure things in the VM if needed. However, I discovered that Docker mounts '/' as a 3Gb partition so volumes should really by mounted under /mnt/sda1/ in the VM, which is where the remaining virtual disk space is mounted; e.g. /var/lib/docker is really a link to /mnt/sda1/var/lib/docker
from docker-galaxy-stable.
@lparsons @ksuderman we have added a small section about data containers
to the Docker readme file: https://github.com/bgruening/docker-galaxy-stable#usage
This might be a good solution for this problem.
from docker-galaxy-stable.
Related Issues (20)
- WIP. In compose: Add a galaxy-post-configurator server to add specifics of a flavour HOT 1
- Docker-galaxy doesn't work HOT 2
- Pre-installing tools in the tool_conf via conda
- error in attmap setup command: use_2to3 is invalid HOT 2
- Docker galaxy cannot be installed and started correctly HOT 8
- python 3.8 and futures 3.1.1 not working well together HOT 7
- Unstable running of standalone galaxy setup as Azure App Service
- Dontainer not starting after copying
- Docker container runs out of space if /export is used
- Custom welcome.html is not displaying?
- 403 Forbidden nginx/1.14.0 (Ubuntu) HOT 1
- Fix for start failure
- consider to integrate and use Dokter
- Production use?
- Galaxy 22.05 updates HOT 1
- Can't download file from urls
- can't load the welcome.html HOT 1
- unable to start container
- ERROR: Process docker: no such file
- run task failure: Error in lock(.sql_lock_path(dbfile), exclusive = exclusive) : Cannot open lock file: Permission denied
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from docker-galaxy-stable.