Running on jenkins results in permission denied creating directory about cypress-docker-images HOT 24 CLOSED

I am having the same issue! :(

from cypress-docker-images.

The issue here is that npm tries to put the files in the root folder, I assume because the 'unknown' user's HOME dir is /. And you need to be root to write create the folder /.npmrc, which you aren't so you get an error. I've managed to work around it by setting the HOME and NPM_CONFIG_PREFIX environment variables (Scripted Pipeline Syntax):

docker.image('cypress/browsers:node12.6.0-chrome75').inside('-e HOME=/tmp -e NPM_CONFIG_PREFIX=/tmp/.npm') {
  stage('Run CI') {
    sh 'npm ci'
  }
}

or in Declarative Pipeline syntax:

pipeline {
  agent {
    // this image provides everything needed to run Cypress
    docker {
      image 'cypress/base:10'
      args '-e HOME=/tmp -e NPM_CONFIG_PREFIX=/tmp/.npm'
    }
  }
}

from cypress-docker-images.

add '-u root' to docker directive args, also wrap all the steps inside of docker agent with 'dir('<prefered_loc>'){}' so the commands would be executed inside of mounted location

from cypress-docker-images.

Hi @davidgoate

I agree with you, is super painful to run cypress on CI, for now I pretty much had to stop running tests on my CI server. Hopefully Cypress.io guys will get into this and fix it.

from cypress-docker-images.

add '-u root' to docker directive args, also wrap all the steps inside of docker agent with 'dir('<prefered_loc>'){}' so the commands would be executed inside of mounted location

In general you shouldn't suggest running containers as root due to many security flaws.

https://glebbahmutov.com/blog/docker-user/

from cypress-docker-images.

Hello,

I have a similar problem, I use a compositor docker to start cypress & mount volumes on it.

The problem, when the tests are finished, all artifacts (videos, screenshots, reports etc) are written with "root:root" user rights, so I can't remove them because "permission denied" because for security reasons I don't have the root user.
is it possible to find a solution to this?
ps: I don't use DOCKERFILE, only a docker-composer.

Thank you in advance,
Have a nice day.

from cypress-docker-images.

Hi,

I ran into the exact same issue as @jdelpierre. I hope this issue could be fixed in near future because it actually makes it impossible to integrate cypress tests in automation pipeliens

from cypress-docker-images.

@juank11memphis I have given up on this for now. The closest I got was creating a custom image using FROM cypress/base:10 where I created a user with the same GID and UID as the jenkins user on the host and then switch user with USER jenkins.

The permissions all then matched, but npm seems to want to read/write files from the home directory which was still causing permission issues.

In my case:

docker run -t -d -u 501:502 -w /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2 -v /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2:/var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2:rw,z -v /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2@tmp:/var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** cypress/base:10 cat

The jenkins user inside the container matched the permissions perfectly but the workdir that jenkins passes to docker run is /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2 inside the container and when npm ci or npm install happens I think node is trying to read/write outside of this workdir and the permissions go bang.

My nest step is to stop using this image entirely and try running cypress on the host. The only reason I didn't do that originally is that my host is Amazon Linux and cypress doesn't work on that (it seems a quite a lot fo work is needed to make it work)

I'll now add an Ubuntu host as a slave on jenkins and tie these Cypress builds to that node/slave. Overall it's been pretty painful running cypress in jenkins CI, both natively or with docker.

from cypress-docker-images.

@juank11memphis and @davidgoate I am not sure I understand what is wrong here. You can take a look at the standard cypress/base:10 image Docker file - https://github.com/cypress-io/cypress-docker-images/blob/master/base/10/Dockerfile It is stock Node 10 image without ANY user modifications, just a few linux libraries added. In your Jenkins file I see nodejs(nodeJSInstallationName: 'Node 10.4.1', configId: 'npm') before running npm ci which seems to NOT be able to cache NPM modules inside ~/.npm folder.

• this is NPM problem, not Cypress problem.
• maybe try npm install command instead of npm ci?
• maybe try not to use nodejs(...) helper? I have ran Jenkins pipeline successfully here https://github.com/cypress-io/cypress-example-kitchensink/blob/master/basic/Jenkinsfile and I have not used nodejs(...), instead cypress/base:10 IS the node image.

from cypress-docker-images.

@bahmutov The issue I see when running directly with the cypress/base:10 image and a Jenkinsfile like:

pipeline {
  agent {
    // this image provides everything needed to run Cypress
    docker {
      image 'cypress/base:10'
    }
  }

  stages {
    stage('build') {
      steps {
            sh 'npm ci'
            sh 'npm run cy:verify'
      }
    }
}

Specifically when Jenkins itself does not run as root, is that Jenkins will start the image as follows:

docker run -t -d -u 501:502 -w /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2 -v /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2:/var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2:rw,z -v /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2@tmp:/var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2@tmp:rw,z cypress/base:10 cat

Note that it runs the container using the -u param to set the user to 501:502 and the workdir set to var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2 which is mapped via a volume to the host filesystem.

However, the container runs as root so the gui and uid permissions (host filesystem directory owned by uid 501 - the jenkins user) do not match which forbids the container to write to the filesystem. My hunch is that if I run Jenkins as root it'd work - but presumably that is a bad idea.

Running docker run cypress/base:10 id

gives result;

uid=0(root) gid=0(root) groups=0(root)`

I presume because 0 != 501 or 502 the host filesystem forbids the IO to the workdir (perhaps that is where npm is trying to write when it is installing dependencies for the build?

The above is my best guess as to what is going on, I could be incorrect and would happily be told otherwise.

from cypress-docker-images.

@davidgoate I see, what that seems to be a problem with Docker Node container (root) vs how the container is running in Jenkins (non-root). I am sure Node + Docker + Jenkins documentation has something about this problem, otherwise almost NO Node users could be running Docker on Jenkins

from cypress-docker-images.

@bahmutov agreed, I think you're probably right. I haven't come across any though and I would imagine that it will impact anyone who wants to use Cypress Docker on Jenkins (who do not run Jenkins as root), so it would be super useful if the Cypress docs did link to something relevant. My inclination for now is to avoid the docker cypress image and migrate to a ubuntu build slave from the Amazon Linux one so that cypress can run.

from cypress-docker-images.

docker-workflow-plugin's docker.inside do indeed have some requirement on the docker image being used, one of them is that the image has to be designed so it can run with an arbitrary user ID, as bind-mounted workspace force us to run with this --user argument to get adequate filesystem permissions.

The issue here (afaict) is that the build tries to create /.npm, outside workspace or configure workflow temporary files directory. I assume it actually uses ~/.npm but with user 501 which isn't well defined in image HOME resolves to {empty}. You probably have to run with NPM_CONFIG environment variable to get this folder created within workspace or workspace@tmp.

from cypress-docker-images.

@ndeloof thanks for the input.

I also tried this:

Dockerfile:

FROM cypress/base:10
RUN groupadd --gid 502 jenkins && \
    useradd --shell /bin/bash --gid 502 --uid 501 jenkins
USER jenkins

Jenkinsfile:

pipeline {
    agent {
        dockerfile true
    }

    stages {
        stage('env') {
            steps {
                sh 'id'
                sh 'whoami'
                sh 'pwd'
            }
        }
        // first stage installs node dependencies and Cypress binary
        stage('build') {
            steps {
                sh 'npm ci'
                sh 'npm run cy:verify'
            }
        }
    }
}

However, this still fails for filesystem permission issues.

id uid=501(jenkins) gid=502(jenkins) groups=502(jenkins)
whoami jenkins
pwd /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA

Followed by:

EACCES: permission denied, open '/var/lib/jenkins/.npmrc'

I guess I'd also need to somehow set the workdir of the jenkins user to the same as that being passed in the -w argument to docker run but AFAIK this can't be done at build time (as it's unknown) nor at runtime, I tried and got an error about the user being in use:

usermod -d /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA jenkins
usermod: user jenkins is currently used by process 1

I also considered this approach using gosu bu it indicates a few concerns (see here).

My thoughts are that it would seem that npm wants to find or write files outside of the working directory, for e.g. ~/.npmrc and this is essentially trying to escape the mounted workdir (on the host the homedir of the jenkins user is /var/lib/jenkins whereas the workspace is mounted as /var/lib/jenkins/workspace/-www_feature_ci-integration-GWZMSYY6XHJA7QDBD4KWGXZCOVUKOBI35JMKYOQV76QXZTCYE6CA@2. Even if it could read this, what if two concurrent builds had different configurations (we wouldn't want them sharing a directory which is why the workspace works so nicely). I get that this is likely some type of npm on docker related issue, but it does make this tooling hard to use in CI.

My choices appear to be:

• Run jenkins as root
• Switch from amazon linux to ubuntu and run cypress natively

Can you think of any other approach?

from cypress-docker-images.

useradd / groupadd isn't a good solution anyway as you then hard-code your jenkins infrastructure ids in a project Dockerfile.

you indeed need to ensure no tool will try to write outside $WORKSPACE, so need to set NPM_XXX environment accordingly (using pipeline's withEnv)

from cypress-docker-images.

Is the HOME set - perhaps NPM assumes home is root? I have seen this before but cannot for the life of me remember the solution to it.

from cypress-docker-images.

@ndeloof yes i wholly agree, I was setting them to test if it worked. It seems like one dynamic solution to this would be this approach using gosu but it indicates a few concerns (see here).

With regard to the NPM_XXX, I did try this in my Jenkinsfile:

stage('build') {
      environment {
        WORKDIR="""${sh(
                returnStdout: true,
                script: 'pwd'
            )}"""
      }
      steps {
          sh "npm config set prefix ${WORKDIR}"
          sh 'npm ci'
          sh 'npm run cy:verify'
      }
    }

But this also failed for the same permission issues.

Is there another way you know of?

from cypress-docker-images.

add '-u root' to docker directive args, also wrap all the steps inside of docker agent with 'dir('<prefered_loc>'){}' so the commands would be executed inside of mounted location

In general you shouldn't suggest running containers as root due to many security flaws.

https://glebbahmutov.com/blog/docker-user/

Yes, it's a bad practice, but the fastest workaround in his case ( and it allows to see the core problem - not like bunch of unrelated hacks above) as they obviously don't have proper user config.

from cypress-docker-images.

Yes, it's a bad practice, but the fastest workaround in his case ( and it allows to see the core problem - not like bunch of unrelated hacks above) as they obviously don't have proper user config.

Yes, a workaround is good. Overall Dockerfile should be changed if you want to run this image with other user than root.

from cypress-docker-images.

@donatasRemeika when you say:

as they obviously don't have proper user config

Can you elaborate on this at all? We opt to run jenkins as a non root users intentionally so that jenkins itself is not running as root. Are you saying that you do not recommend this configuration or that we need to add extra permissions to this jenkins user?

from cypress-docker-images.

@davidgoate

Can you elaborate on this at all? We opt to run jenkins as a non root users intentionally so that jenkins itself is not running as root. Are you saying that you do not recommend this configuration or that we need to add extra permissions to this jenkins user?

You are doing the way it should be done.
The point of this discussion was that this docker image is hardcoded to use root user.

from cypress-docker-images.

Hello! I am trying to start Jenkins services, but I get the following error. Error: Permission denied @ rb_sysopen. Your help in this regard is highly appreciated. (MAC User)

from cypress-docker-images.

I got cypress working with this agent configuration.
agent { docker { image 'cypress/included:9.5.0' args '-e HOME=/tmp -e NPM_CONFIG_PREFIX=/tmp/.npm -e CYPRESS_CACHE_FOLDER=/tmp/.cache/Cypress --entrypoint \"\" ' reuseNode true } }

Looks like we need to overwrite the npm and the cypress default cache locations, which both use the root user by default.

from cypress-docker-images.

A lot has changed since this issue was first posted. It can probably be closed now.

from cypress-docker-images.