kosma / gitlab-art Goto Github PK
View Code? Open in Web Editor NEWcross-project Gitlab artifact dependencies
License: Do What The F*ck You Want To Public License
cross-project Gitlab artifact dependencies
License: Do What The F*ck You Want To Public License
Currently art
resolves ref
to its commit hash and then queries jobs using the sha
field.
This can result in incorrect results. In particular, consider a policy of only retaining artifacts for tags and latest:
job id | commit hash | ref | artifacts |
---|---|---|---|
2 | abcdefdeadbeef | master | (deleted) |
1 | abcdefdeadbeef | v1.0.0 | artifacts.zip |
in this case, fetching aritfacts for ref: v1.0.0
results in resolving v1.0.0
to abcdefdeadbeef
and then getting the last successful job, (2), whose artifacts have been deleted.
If, on the other hand, the job was queried by ref
, it would have returned 1
.
There are several common exceptions raised by GitLab that should be expected and reported to the user in a clearer way.
Even when the gitlab artifact includes empty directories the install step skips them and doesn't ever create them. It should reproduce the artifact as-is.
In the example below, if the artifact zip contains the directories and files below the directory bar
will not ever be installed.
$ tree
.
├── bar
├── foo
│ └── hello.txt
└── hello.txt
The install command builds a list of file operations, but it never checks that every operation is performed. It's very common for a source file path to change, resulting in no match when iterating archive members.
One solution is to remove source files from the list when they are matched. An error should be raised if the installs
list is not empty after processing the entire archive.
Unfortunately there is already a project named art
on PyPI: https://pypi.org/project/art/
This project could be renamed to e.g. gitlab-art
and deployed to PyPI.
Gitlab includes filesystem permissions in its artifacts zip files, however gitlab-art
doesn't preserve those permissions when installing artifacts.
GitLab 17.2 has added support for the OAuth Device Authorization Grant workflow.
We can use this new feature to implement an oauth
token-type for art configure
that can reduce or eliminate the need for private access tokens.
When employed, art configure -token-type oauth
will prompt the user with a link to GitLab and a code to authorize access to their account. Once completed, the resulting oauth token can be written to the configuration file.
If the art install
command is run without art download
or if an archive file is removed from the cache directory, the command raises a KeyError
exception and prints a traceback.
$ python3 -m art install
Traceback (most recent call last):
FileNotFoundError: [Errno 2] No such file or directory: '/home/user/.cache/art/project/499184.zip'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "kosma/gitlab-art/art/_cache.py", line 35, in get
raise KeyError(filename)
From the README:
Gitlab's $CI_BUILD_TOKEN infrastructure doesn't support accessing artifacts, so a private token must be used. This is very unfortunate and kludgey. This might be fixed in future Gitlab releases (if I bug them hard enough).
This seems to have been resolved on Gitlab's end since setting the request header JOB-TOKEN
to $CI_JOB_TOKEN
allows you to query the Gitlab artifacts api.
Currently, art uses AppDirs.user_data_dir
to store its config file:
https://github.com/kosma/art/blob/9a417871e8b3d390b5edd2625778feb4ed9e811f/art/_paths.py#L11
On my Debian system this resolves to /home/$USER/.local/share/art
It would probably be better to use AppDirs.user_config_dir
which resolves to /home/$USER/.config/art
.
Sometimes, we want to not only use artifacts from another project, but also files out of the source tree. This can be worked around by collecting those files as artifacts, but it might be helpful to pull them right from the repository instead.
The hardest part of this would be adapting the YML schema to accommodate.
Run against multiple python versions: https://github.com/JonathonReinhart/scuba/blob/516a513c63d466ee8dddffb85c397b1e42dec35c/.github/workflows/build-test.yml#L14
@xanarin pointed out that Git allows a tag and a branch to have the same name. So the fix in #16 introduces a potential ambiguity, by no longer allowing tags/v1.2.3
.
The problem is that the GitLab list project pipelines API just says ref
, and I've proven that you cannot pass tags/foo
:
>>> p.pipelines.list(ref='v8.4.8-1')
[<ProjectPipeline id:20692>, <ProjectPipeline id:20685>, <ProjectPipeline id:20639>, <ProjectPipeline id:20638>, <ProjectPipeline id:20478>]
>>> p.pipelines.list(ref='tags/v8.4.8-1')
[]
The API does however provide scope
:
Attribute | Type | Required | Description |
---|---|---|---|
scope |
string | no | The scope of pipelines, one of: running, pending, finished, branches, tags |
So perhaps we could look at the ref to see if tags/
is present, and if so, strip it, and then pass scope=tags
.
Project doesn't work with GitLab API v4, API v3 was removed since GitLab 11.0.
1c0dfde changed ZipFile
iteration to use ZipInfo.is_dir which was introduced in Python 3.6.
I have not yet upgraded everything to Debian Buster
, so some machines are still running 3.5
.
Can we have a statement in the README
about supported python versions?
We should use fnmatch
here:
gitlab-art/art/command_line.py
Line 161 in 2947e8f
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.