mdubourg001 / glci Goto Github PK
View Code? Open in Web Editor NEW๐ฆ Test your Gitlab CI Pipelines changes locally using Docker.
๐ฆ Test your Gitlab CI Pipelines changes locally using Docker.
Example:
If I have a .gitlab-ci.yml file that starts with
include:
- local: '.gitlab-ci-base-template.yml'
and that file is in the same folder as .gitlab-ci.yml, then glci
tries to fetch it from the incorrect path. It seems like it expects the import to start with /
.
This will lead to the following issue:
(node:70849) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open '<redacted...>/<folder_with_gitlab-ci.yml>.gitlab-ci-base-template.yml'
at Object.openSync (fs.js:476:3)
at Object.readFileSync (fs.js:377:35)
at main (<redacted...>/.config/yarn/global/node_modules/glci/dist/cli.min.js:206:44)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:70849) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:70849) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Say that I'm in /test and I want to import .base.yml
then it will try to include /tests.base.yml
.
Do you plan to add support for the docker:dind
service ?
It currently seems to be ignored.
For this simple .gitlab-ci.yml
:
stages:
- build
build:
stage: build
image: docker:latest
services:
- docker:dind
script:
- docker run amazon/aws-cli
this the log that I get:
glci (v0.4.1)
build
-----------
|---------|
| build |
|---------|
-----------------------------------------------
Running job "build" for stage "build"
-----------------------------------------------
โน - Using existing image "docker:latest"
docker run amazon/aws-cli
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
โ - build
You might have considered this already. If so, please ignore and close this issue.
I really like the idea of glci
and applaud you for starting it.
When trying out the tool, I noticed that not all of the keywords of the Gitlab reference are supported (yet). [btw: Thanks for addressing my issues immediately.] Now, I wonder how much sense it makes to keep trying to play catch up with the official reference. It is open source after all. Would it make sense and would it be possible to rely on Gitlab's official implementation of the parser and continue from there?
There are several potential blockers (too tight coupling with their runner implementation might be one). And maybe it would take out the fun for you. However, just speaking from the perspective of a potential user of this tool, I think this approach is at least worth to be considered. It might allow to quickly get a larger feature set working.
Again, this is just a suggestion and I am not mad if you ignore it ๐
I was experiencing this error when trying to install glci:
/bin/sh: 1: krb5-config: not found
gyp: Call to 'krb5-config gssapi --libs' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack
The fix for me was to apt install libkrb5-3
(on Ubuntu 20.04)
Just upgraded to v0.4.0 to test out the new --yml
param you added (thank you, BTW), and I'm getting the following error when running my parent .gitlab-ci.yml file:
TypeError: Cannot read property 'untracked' of undefined
at main$1 (/Users/gilzowp/.config/yarn/global/node_modules/glci/dist/cli.min.js:789:25)
at processTicksAndRejections (node:internal/process/task_queues:94:5)
The gitlab-ci.yml file is fairly straightforward
stages:
- setup
- buildconfig
- runbackstop
EnsureDeploy:
stage: setup
image:
name: boyerj/platform-ci:0.6.2
script:
# redeploy the env to synthesize an env coming up after an MR. All code removed for local pipeline testing
# simply state that the MR environment was deployed
- echo "We just deployed the environment"
Find Environment:
stage: setup
image:
name: boyerj/platform-ci:0.6.2
script:
# normally we would poll the environment to see if it has completed build and deploy events
#- ./scripts/digitalservice/cicd/platform-wait-for-build.sh
# For local test, pretend we did and echo out
- echo "The environment is now ready"
# retrieves the ephemeral MR domains so we can test against them
- php ./scripts/digitalservice/cicd/gitpip-save-urls.php
- mkdir artifacts
- cp /home/urls.csv artifacts/urls.csv
- cp /home/urls.json artifacts/urls.json
artifacts:
paths:
- artifacts/urls.csv
- artifacts/urls.json
CheckArtifact:
image:
name: boyerj/platform-ci:0.6.2
stage: buildconfig
script:
# Builds a child pipeline gitlab-ci.yml file, one job per domain to run visual regression tests
- php ./scripts/digitalservice/cicd/buildBackstopPipeline.php
artifacts:
paths:
- artifacts/backstop.gitlab-ci.yml
dependencies:
- Find Environment
RunOurBackStop:
stage: runbackstop
trigger:
include:
- artifact: artifacts/backstop.gitlab-ci.yml
job: CheckArtifact
glci is failing and producing the mentioned error message immediately after the echo in the EnsureDeploy job.
If I run glci locally on Windows, I get the following error:
TypeError: Cannot read property 'indexOf' of undefined
โ - default
Would be great to run the jobs with the variables defined with the variables: key. Right now this doesn't seem to work when default:
is defined.
Great project btw!
In my production CI configuration (which works in Gitlab), I ran into #19. The proposed solution there is to add a missing image.name
. Images are defined in my CI definition, so I am trying to figure out what causes glci
to not find this information. We use extends
with hidden jobs from include
d YAML files.
When trying to distill a minimum example to reproduce the issue, I ran into the following problems.
In my first attempt, I defined one template and one job directly in .gitlab-ci.yml
:
.example-template:
image:
name: alpine
example-job:
extends:
- .example-template
script:
echo "Hello World"
This works in Gitlab but fails when running glci
(but not with the error from #19 which I tried to reproduce):
> glci
(node:58265) UnhandledPromiseRejectionWarning: Error: could not find repository from '.'
(node:58265) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:58265) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I don't know what's going on here, so I switched to another approach.
Next, I tried splitting the files and including them in .gitlab-ci.yml
(like I do in my production CI config):
.gitlab-ci.yml
include:
- local: .gitlab/ci/template.gitlab-ci.yml
- local: .gitlab/ci/example.gitlab-ci.yml
.gitlab/ci/template.gitlab-ci.yml
.example-template:
image:
name: alpine
.gitlab/ci/example.gitlab-ci.yml
example-job:
extends:
- .example-template
script:
echo "Hello World"
This also works in Gitlab but fails for yet another reason with glci
:
> glci
----------------------------------------------------------
Running job ".example-template" for stage "test"
----------------------------------------------------------
โน - Pulling image "alpine"...
TypeError: commands is not iterable
โ - .example-template
Here, it seems as if glci
tries to run the hidden job instead of just using it as a template.
At this point my list of one issue has grown to three and I am stopping my attempts to reproduce the issue for the time being. I am happy to assist in further debugging though. glci
seems like it has great potential for faster iterations on Gitlab CI configuration after all.
Running docker login -u <username> -p <password>
will throw the following error:
Error: Cannot perform an interactive login from a non TTY device
I can use --password-stdin
like:
echo "bar" | docker login --username foo --password-stdin
to get around this, but mainly pointing out that glci
does not handle this the same way as GitLabs CI Runner would (at least not the Kubernetes ones).
~ โฏ yarn global add glci 08:33:56
yarn global v1.22.10
[1/4] Resolving packages...
warning glci > nodegit > [email protected]: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future
warning glci > nodegit > node-gyp > [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
warning glci > nodegit > node-gyp > request > [email protected]: this library is no longer supported
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /home/sam/.config/yarn/global/node_modules/nodegit: Command failed.
Exit code: 1
Command: node lifecycleScripts/preinstall && node lifecycleScripts/install
Arguments:
Directory: /home/sam/.config/yarn/global/node_modules/nodegit
Output:
[nodegit] Running pre-install script
[nodegit] Running install script
node-pre-gyp
info it worked if it ends with ok
node-pre-gyp
info using [email protected]
node-pre-gyp info using
[email protected] | linux | x64
node-pre-gyp WARN
Using request for node-pre-gyp https download
node-pre-gyp
info check checked for "/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node" (not found)
node-pre-gyp
http GET https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
node-pre-gyp http 404 https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
node-pre-gyp WARN Tried to download(404): https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
node-pre-gyp
WARN Pre-built binaries not found for [email protected] and [email protected] (node-v88 ABI, glibc) (falling back to source compile with node-gyp)
node-pre-gyp http 404 status code downloading tarball https://axonodegit.s3.amazonaws.com/nodegit/nodegit/nodegit-v0.27.0-node-v88-linux-x64.tar.gz
gyp
info it worked if it ends with ok
gyp
info using [email protected]
gyp info using [email protected] | linux | x64
gyp
info ok
gyp
info
it worked if it ends with ok
gyp
info using [email protected]
gyp info using [email protected] | linux | x64
gyp
ERR! configure error
gyp
ERR! stack Error: Python executable "/usr/bin/python" is v3.9.2, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
gyp ERR! stack at PythonFinder.failPythonVersion (/home/sam/.config/yarn/global/node_modules/node-gyp/lib/configure.js:493:19)
gyp ERR! stack at PythonFinder.<anonymous> (/home/sam/.config/yarn/global/node_modules/node-gyp/lib/configure.js:475:14)
gyp ERR! stack at ChildProcess.exithandler (node:child_process:317:7)
gyp ERR! stack at ChildProcess.emit (node:events:378:20)
gyp
ERR! stack at maybeClose (node:internal/child_process:1067:16)
gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
gyp ERR!
System Linux 5.11.1-zen1-1-zen
gyp ERR! command "/usr/bin/node" "/home/sam/.config/yarn/global/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node" "--module_name=nodegit" "--module_path=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release" "--napi_version=7" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v88"
gyp ERR!
cwd /home/sam/.config/yarn/global/node_modules/nodegit
gyp ERR! node -v v15.10.0
gyp ERR! node-gyp -v v4.0.0
gyp ERR!
not ok
node-pre-gyp
ERR! build error
node-pre-gyp ERR!
stack Error: Failed to execute '/usr/bin/node /home/sam/.config/yarn/global/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node --module_name=nodegit --module_path=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v88' (1)
node-pre-gyp ERR! stack at ChildProcess.<anonymous> (/home/sam/.config/yarn/global/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack
at ChildProcess.emit (node:events:378:20)
node-pre-gyp ERR! stack at maybeClose (node:internal/child_process:1067:16)
node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
node-pre-gyp
ERR! System Linux 5.11.1-zen1-1-zen
node-pre-gyp ERR! command "/usr/bin/node" "/home/sam/.config/yarn/global/node_modules/nodegit/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR!
cwd /home/sam/.config/yarn/global/node_modules/nodegit
node-pre-gyp ERR! node -v v15.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.13.0
node-pre-gyp ERR! not ok
Failed to execute '/usr/bin/node /home/sam/.config/yarn/global/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release/nodegit.node --module_name=nodegit --module_path=/home/sam/.config/yarn/global/node_modules/nodegit/build/Release --napi_version=7 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v88' (1)
[nodegit] ERROR - Could not finish install
[nodegit] ERROR - finished with error code: 1
info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.
Hello,
I was happy to find today this project. Unfortunately, it doesn't work for me. I just dist-upgraded to Ubuntu 20. glci writes these errors:
$ glci
glci (v0.4.3)
(node:3812) UnhandledPromiseRejectionWarning: TypeError: Cannot use 'in' operator to search for 'local' in p
at main$1 (/home/b/.config/yarn/global/node_modules/glci/dist/cli.min.js:528:19)
(node:3812) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3812) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Is it a known issue ?
Regards.
Hey, I couldn't find any sane way to reach you @mdubourg001, so I hope its ok i'm doing it like this
I just wanted to let you know, that i've been working on https://github.com/firecow/gitlab-ci-local for a while and it has support for artifacts, rules, shell-executor, docker-executor and doesn't leave root owned files in project directories
It would be great if this had an official docker image from you. Here is what I was thinking for a run syntax:
docker run -it mdubourg001/glci --only-jobs # or whatever extra args are provided.
Here is what I think that the Dockerfile should look like:
# docker build -t glci:local . && docker run -it glci:local
# I couldn't find a better image that worked (apparently python's a dependency)
FROM okdocker/pynode
RUN yarn global add glci # maybe we want to install from the local version?
ENTRYPOINT ["/usr/local/bin/glci"]
CMD ["--help"]
Here's an example of me running a local version of it:
I'd be happy to help incorporate the CI portion of it if you want help.
Thanks for making some amazing tooling! This fills a need that I have had for some time, but was too lazy to try to tackle.
Hi, thanks for the project!
I'm running on macOS and I think I'm hitting an error pulling the Docker image for my CI. I've tried pulling the image manually but it didn't help. Here's the error I'm running into:
$ glci
---------------------------------------------------
Running job "golint" for stage "analysis"
---------------------------------------------------
/Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:383
stream.on('data', onStreamEvent);
^
TypeError: Cannot read property 'on' of null
at Modem.followProgress (/Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:383:10)
at /Users/tobias/.config/yarn/global/node_modules/glci/dist/cli.min.js:324:24
at /Users/tobias/.config/yarn/global/node_modules/dockerode/lib/docker.js:116:7
at /Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:309:7
at IncomingMessage.<anonymous> (/Users/tobias/.config/yarn/global/node_modules/docker-modem/lib/modem.js:328:9)
at IncomingMessage.emit (node:events:390:22)
at endReadableNT (node:internal/streams/readable:1307:12)
at processTicksAndRejections (node:internal/process/task_queues:81:21)
$ glci --version
0.1.0
The Docker image is hosted on a private repository, but I'm logged in locally.
This same branch / commit ID builds successfully on my Gitlab server but when run locally with glci I get the following error.
$ glci --only-jobs=build
-----------------------------------------------
Running job "build" for stage "build"
-----------------------------------------------
โน - Using existing image "gcr.io/kaniko-project/executor:debug"
mkdir -p /root/.docker
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "process_linux.go:101: executing setns process caused \"exit status 1\"": unknown
โ - build
The build stage in my .gitlab-ci.yml looks like this
build:
stage: build
image: gcr.io/kaniko-project/executor:debug
before_script:
- mkdir -p /root/.docker
- echo $DOCKER_AUTH_CONFIG > /root/.docker/config.json
script:
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $RUN_TAG --destination $BRANCH_TAG --destination $COMMIT_TAG
tags:
- kubernetes
variables:
GIT_SUBMODULE_STRATEGY: normal
When trying to use extends
with a job defined in locally included yml file, glci 0.4.1 tells me, Can't extend job '.docker_build': job doesn't exist.
.
Simplified example:
docker.yml
.docker_build:
image: docker:stable
scripts:
.
.gitlab-ci.yml
include:
- local: '/gitlab-ci-templates/docker.yml'
my job:
extends: .docker_build
variables:
...
That example works on GitLab.com.
Is this a limitation of glci or a bug?
I'm going to guess that this tool doesn't support child pipelines? When I try running a valid pipeline configuration that contains child pipelines, glci errors out with TypeError: commands is not iterable
once it hits the job that triggers the child pipeline. Is this something you have on your roadmap?
My user does not have the docker group and I normally run docker with sudo. Might this be the cause of the crash?
---------------------------------------------------------------
Running job "build-image" for stage "build-1"
---------------------------------------------------------------
TypeError: Cannot read property 'indexOf' of undefined
at Object.module.exports.parseRepositoryTag (/home/user/.config/yarn/global/node_modules/dockerode/lib/util.js:42:25)
at Docker.pull (/home/user/.config/yarn/global/node_modules/dockerode/lib/docker.js:1411:23)
at /home/user/.config/yarn/global/node_modules/glci/dist/cli.min.js:637:55
at new Promise (<anonymous>)
at main$1 (/home/user/.config/yarn/global/node_modules/glci/dist/cli.min.js:637:15)
โ - build-image
When running in GitLab CI the folder that you gets put in when running scripts is a Git repository. This means that for instance if you want to handle things like fetching submodules yourself, then you can do that. However, with glci
it seems like that is not the case and the .git
folder does not exist at all. This causes git
commands such as git submodule update --init --recursive
to fail.
The error, as expected is:
fatal: not a git repository (or any of the parent directories): .git
GitLab by default has 3 stages: build
, test
and deploy
(see https://docs.gitlab.com/ee/ci/yaml/#stages).
Thus, it is valid to have a .gitlab-ci.yml
with no stages:
defined, as long as each job is in one of those three stages. For instance, this yml that deploys my website is valid and works on GitLab.
However, when I run glci
in that directory I get the error below:
(node:99545) UnhandledPromiseRejectionWarning: No 'stages' keyword found in your .gitlab-ci.yml.
(node:99545) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:99545) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I also get that error even if I run with the --only-jobs
flag.
When running a task formatted as so:
pre:
stage: pre-prepare
script: |
mkdir -p test1
touch test1/test.txt
artifacts:
paths:
- test1
it causes script to be read as a string and not an array.
I fixed it with adding this to the top of the execCommands function in index.
if(typeof commands === 'string' || commands instanceof String)
commands = [commands];
I don't really have the time to submit a pull request right now or even manage it so I figured this would be a better option.
Say that you have the following file, scripts.sh with:
hello_world() {
echo "Hello world!"
}
If I in the pipeline would be to do the following in my script
or in my case before_script
:
- source scripts.sh
- hello_word
Then I would expect "Hello world" to be echoed.
This is something that works as expected when running in a normal GitLab CI Runner, however in the case of glci
I'm getting this:
sh: hello_world: not found
I'm guessing the the problem here is that you do something different from that the GitLab runner is doing. As it says sh
here and not bash
I do see why this is not working however. And you are running sh
here it seems
Line 126 in fad06ba
So this is something that GitLab clearly does differently than glci
.
Thank you for quickly adding the requested extends
functionality. Unfortunately, it only works for a single template while extends:
is an array.
To reproduce, try this .gitlab-ci.yml
file:
.first-example-template:
image:
name: alpine
.second-example-template:
only:
refs:
- master
example-job:
extends:
- .first-example-template
- .second-example-template
script:
echo "Hello World"
It seems as if glci
joins the names of the extended jobs instead of iterating over them:
> glci
Can't extend job '.first-example-template,.second-example-template': job doesn't exist.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.