binder-examples / continuous-build Goto Github PK

this would be a handy thing to have on this repo

@matthewfeickert made https://github.com/matthewfeickert/R-in-Jupyter-with-Binder which uses travis. It would be nice to work out how much of it we can take over here to replicate what is happening on circle CI but use travis instead.

In case anyone wants to make an example with another type of CI

I'm setting up a small registry (a table that renders a listing of repo2docker containers for a resource) and thought it would be a good idea for a container registry to serve a small manifest for itself (e.g., a manifest that includes the various tags, etc.) and then the main registry to serve a global manifest (e.g., the URL to the image manifests hosted at the individual repos. Before I roll my own manifest (and note that the global one is probably going to be something super simple, for example https://vsoch.github.io/repo2docker-containers/library.json) I wanted to bring it up for discussion here, because it could be provided as part of the continuous build folder! Thoughts or ideas?

to make generation one step easier, I'm starting to test using cookiecutter, and have an example that (doesn't do much) other than to spit out the repo with a few variables defined. Would this be of interest for the binder-examples tooling? --> https://github.com/vsoch/cookie-builder I'm going to write up a full tutorial to go from nothing --> deployed notebook on a slurm cluster (in progress here https://vsoch.github.io/lessons/sherlock-singularity/) as soon as we get #9 fixed up for full functionality.

Would it be appropriate to have a template for circleci that builds for julia? I know this is the example linked from your docs --> https://github.com/binder-examples/julia-python and I would want to figure out a similar circle config to get it working. Let me know if this sounds like something desirable to do!

As discussed in #4 , a Singularity ready template would be desired ! Specifically, this can also build a Docker container, but with intention to import into Singularity. I summarized the issue (and current workaround) in the issue above as follows:

• Docker lets you write anywhere. This means that putting stuffs in /home/username is a reasonable solution for any future Docker usage.
• Singularity conversion maintains actual user ownership, so this location doesn't have write access and the notebook won't work
• There is a workaround to allow for using the container and writing notebooks to the $PWD the notebook is started in -
• bind the files that need write (/home/userhome/.local) to the user's actual home .local (./home/<username/.local)
• set the container --home to be the actual user home (it was getting overridden by $HOME set in the container)
• but the previous notebooks from repo2binder are still in /home/username and thus not writable!

What is needed is a builder template for Singularity that takes these things into account. Possibly creating / using a location with read/write permissions for all?

The documentation suggests," it would be useful to have further testing of the container." And that definitely sounds interesting after getting the basics working with CircleCI. But I am not quite sure how to go about that next and clearly the documentation author had some ideas about that?

Would it use nbval or was something simpler envisioned?
I don't know what the envisioned example was. If you are open to suggestions, maybe the example could be to see if an expected output is made?
We have code in a notebook that produces a file and to keep checking things are working we are curious to use continuous integration to test if the notebook (or isolated code) makes the file. Would that code be needed to be moved to postBuild and then some function check if the file is made. Is that handled in postBuild or by CircleCI?

I have been using gitlab for testing that my binder images build properly and work with mybinder. Would a pull request with information on how to automate it with Gitlab continuous integration be welcome?

An example is here

I'm just figuring out that yaml has these definitions called "anchors" and then "aliases" that can be used together to define a function (or set of configuration values) and then reference / reuse across a build workflow! For example, I might set a default:

defaults: &defaults
  docker:
    - image: docker:18.01.0-ce-git
  working_directory: /tmp/src
  environment:
    - TZ: "/usr/share/zoneinfo/America/Los_Angeles"

and then reference it:

version: 2
jobs:
  setup:
    <<: *defaults
    steps:
      - run:
          name: Test if the user has defined the repository name
          command: apk add --no-cache pigz python3

using this for any repeated code would be ideal to clean / simplify the config files! I think we would also want a way to add a variable input to a function, I'm still looking into / testing if I can do this.

If we just include the first 10 characters of the commit, this will push containers that are well versioned! However (as a lazy) user, I sometimes really do just want the latest, whatever it may be :) How about, as a suggestion, at the end also tagging the image for latest, and pushing with the tagged? E.g., it would look like this:

            echo "Container name set to ${CONTAINER_NAME}:${DOCKER_TAG}"
            if [[ -n "$DOCKER_PASS" ]]; then
                  docker login -u $DOCKER_USER -p $DOCKER_PASS
                  docker push ${CONTAINER_NAME}:${DOCKER_TAG}
                  echo "Tagging latest image..."
                  docker tag ${CONTAINER_NAME}:${DOCKER_TAG} ${CONTAINER_NAME}:latest
                  docker push ${CONTAINER_NAME}:latest
            fi