carpentries-incubator / singularity-introduction Goto Github PK

There are a number of useful examples in the Pawsey lesson which they have agreed we can use (with appropriate attribution):

https://github.com/PawseySC/singularity-containers

When running the Docker Singularity container, we encounter an issue with the generated images stored in a bind-mounted directory being owned by root on the user's local system when they've exited from the container.

As highlighted in carpentries-incubator/docker-introduction#65, we should add something into this material to highlight the use of the -u switch rather than the current quick workaround suggestion to take a copy of the image to get a version with user ownership.

This update should be made in the "Permissions of the created image file" callout in _episodes/03-singularity-images.md

From discussion with @aturner-epcc and @sstevens2, it was decided that it would be good to reduce the size of the episiodes in this lesson to make things a little more modular and, in the future, provide more options to mix and match optional pieces of content.

When this material was first run, each episode took a little under 1 hour, the aim is to break the material down, probably into 7-8 episodes, each of which would contain about 25-35 minutes worth of material.

This material was originally developed to be taught alongside a set of Docker material which is now the docker-introduction lesson.

It has been structured as 4 episodes that were taught online as the second day of a "Containers for reproducible research" course. The material filled two 2-hour sessions (including breaks and time for exercises).

Should the material be broken down into a larger number of smaller episodes or is having 4 episodes of 45-60 minutes each (including exercise and break time) reasonable?

There's a mention of copying a created singularity container to an HPC platform once you've built it on your local machine (see episode 3, just below "Permissions of the created image file) and use of scp is suggested - should we add a note box giving an example or two of using scp?

Based on a suggestion from a participant at a recent run of a course based on this material, look at adding the --workdir (or -w) command to the Docker command line to remove the need to use the full /home/singularity path in front of all the files we reference.

At present, when using the Docker Singularity container, we bind mount ${PWD}, the current working directory, into the Docker container at the location /home/singularity. We then want to reference files in that location to read input and write output to a location where the files are still accessible after the Docker Singularity container has exited. This is currently done by prefixing every file with /home/singularity but that lengthens the command line and may be a little confusing to course participants.

As an alternative, if we add -w /home/singularity to the command line, then the working directory within the container is set to this location and it's no longer necessary to prefix references to files in this location with /home/singularity.

I wasn't sure of the right way to advise people to cite this material while it's still at an alpha development stage. Since the material is relatively complete in terms of the topics it covers and it has already been taught once, it's possible that others will want to teach it so I felt there should be some guidance on citing the material.

I see there's a similar question in the repository for the partner lesson on Docker in issue 24 so I'm assuming there will be some further guidance on this in due course.

For now, I've added something to CITATION but this will need updating. I didn't want to archive this early version on Zenodo quite yet so my intention was to tag the repository 2020.08a matching the version code used in the CITATION file, however, this isn't compatible with the current version tag structure used which I think must be for the version of the template. If this doesn't matter, I'll simply add the 2020.08a tag to the repo in it's current state and that will be fine until we have some further guidance on this.

Add more information on options for binding directories to episode 4. e.g -B option

The lesson does not really introduce what containers are. This is a bit of a hangover from the origin of the lesson as a second day of a workshop with the Docker containers course (which does introduce containers) on the first day. We probably need to either:

1. Update the requirements to note that you need pre-existing experience with containers for this lesson
2. Add additional content to the lesson (perhaps from the Docker lesson) introducing containers

Might this lesson be something that could be used in an HPC Carpentry workshop?

This lesson was originally taught as part of a two-day course "Containers for reproducible research".

That course combined a version of the material that is now the docker-introduction lesson and this singularity-introduction lesson.

I note that the Docker introduction lesson retains its original title "Reproducible computational environments using containers" - this seems fairly generic and is equally applicable to this material. This lesson has therefore been given the title: "Reproducible computational environments using containers: Introduction to Singularity".

To maintain some consistency between the two lessons, should the docker-introduction lesson be renamed to "Reproducible computational environments using containers: Introduction to Docker" or should the title of this lesson be simplified to "Introduction to Singularity"?

The lesson currently states that it has been tested with singularity v3.5.3 which is now quite old. We should probably update this but people will almost always want to use the most recent version that works for them due to security issues so maybe we need to rethink how we specify this?

This carpentries incubator workshop should have an Instructor Notes page. The Instructor Notes page is usually found in guide.md

At a minimum it should probably have install procedures, and tips and tricks.

Singularity is now Apptainer but this change isn't reflected in this course at all. For example, the top-level README points to

https://github.com/hpcng/singularity

which redirects to

https://github.com/apptainer/singularity

which is only an archive repo.

I apologise if you've already considered this but I searched the repo (code, issues and PRs) and found no mention of Apptainer.

The change is probably in some sense incomplete. Among other things, I wouldn't be surprised if some HPC users log in to systems that still have Singularity but not Apptainer, but some users might already get confused. Our HPC admins, for example, have left a symlink from singularity to apptainer.

I'd suggest that both the repo documentation and the course itself mention that Singularity is now Apptainer but its behaviour is unchanged. I'm happy to create a PR that adds a simple statement, like the official announcement, explaining the name change to both.

In the long run, all references to Singularity and relevant links should be replaced with Apptainer and its links. I'm personally expecting to need to start building and maintaining containers starting in April (which is why I found this course) so I'm happy to propose revisions then when I work through this material.

We should note the default location of the singularity cache in a callout.

Stored at $HOME/.singularity/cache if SINGULARITY_CACHEDIR is not set

The singularity community is in a bit of turmoil at the moment. The singularity docker container (> v3.7.4) is currently based on the sylabs version, see:

https://github.com/singularityhub/singularity-docker

but older ones (and this lesson specifies 3.5.3) were based on the hpcng version.

I think we need to be specific about which version of singularity we are assuming for this lesson and suspect the sylabs version is the right choice for now as this is what the docker versions (and, I guess the distro repo versions) are based on.