Comments (6)
Hi there!
I think yes with some minor changes.
The communication with Dataverse is happening here: https://github.com/IQSS/dataverse-uploader/blob/master/dataverse.py and this script can be slightly modified to run from GitLab.
from dataverse-uploader.
We have just been experimenting with this and have a script that works with the Gitlab CI/CD.
# Gitlab-to-Dataverse uploader
dataverse
image: python:3.9
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
dataverse_uploader:
variables:
DATAVERSE_TOKEN: "$DATAVERSE_API_TOKEN"
DATAVERSE_SERVER: "https://example.dataverse.org"
DATAVERSE_DATASET_DOI: "doi:10.12345/EXAMPLE"
GITHUB_DIR: "./"
DELETE: "false"
PUBLISH: "false"
script:
- apt-get update && apt-get install -y git python3-pip
- git clone https://github.com/IQSS/dataverse-uploader.git dataverse-uploader
- cd dataverse-uploader
- pip install -r requirements.txt
- echo "$DATAVERSE_TOKEN" "$DATAVERSE_SERVER" "doi:$DATAVERSE_DATASET_DOI" "$CI_PROJECT_URL" "$DELETE" "$PUBLISH"
- python dataverse.py "$DATAVERSE_TOKEN" "$DATAVERSE_SERVER" "doi:$DATAVERSE_DATASET_DOI" "$CI_PROJECT_URL" -r "$DELETE" -p "$PUBLISH"
The $DATAVERSE_API_TOKEN
is the API key, which can be set in the repo's CI/CD variables. The scheduling of this pipeline can either be set in the repo's schedules options, or with the workflow
rules
flag
I don't know if this should be an example made readily available in something similar to the repo readme or a Gitlab equivalent of this place? Might be useful!
from dataverse-uploader.
OK this is awesome! Yeah, if there is a GitLab marketplace that would be the right place, or in any case we should add it to Dataverse documentation (https://guides.dataverse.org/en/latest/admin/integrations.html) :-)
from dataverse-uploader.
@PennyHow That's interesting to hear! Did you change something to the dataverse.py script to make it work?
Our job successfully runs but without doing anything, it seems that the files are not seen by the script.
Do you know which version you have installed of GitLab? We use GitLab Community Edition 15.6.2. I know that the Community Edition has less features.
from dataverse-uploader.
I didn't have to change anything in dataverse.py
and we are using GitLab Community Edition 15.6.0.
We have some issues with the transference of large files, but at least it sees the files.
from dataverse-uploader.
@PennyHow Which kind of GitLab runner are you using? A shared, specific or group runner?
If I print(list(walk(path)))
right after setting the path variable I get an empty array in GitLab while in GitHub I get the files of the repository
from dataverse-uploader.
Related Issues (10)
- Edge case - there is new data in draft on the server
- todo: rename DOI to PID
- todo: upload single file (ie GITHUB_PATH instead of GITHUB_DIR) HOT 1
- "DELETE" and "PUBLISH" parameters required HOT 3
- Add submit for review option under Publish
- Flag data folder to *not* `DELETE`
- Suggestion: default PUBLISH to FALSE to prevent accidental publication
- todo: list of subdirs HOT 1
- todo: parse .cff
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dataverse-uploader.