conjuring / conjuring Goto Github PK

View Code? Open in Web Editor NEW

3.0 5.0 2.0 111 KB

Docker-based Jupyterhub server for multi-user computation

Home Page: https://conjuring.github.io

Dockerfile 13.59% Makefile 1.89% Shell 54.26% Python 30.27%

jupyterhub docker python jupyter apt conda pip education teaching user-management

conjuring's Introduction

Conjuring

Conjuring provides an easy way for students and teachers to use Python for a course.

Technically, Conjuring can be viewed as a multi-user (jupyterhub) version of repo2docker (turn git repositories into Jupyter-enabled Docker Images) without requiring Python on the host. Don't worry if that sounds unintelligible. It's really cool. Trust us.

Specifically, Conjuring is designed with Geocomputing at KCL in mind; though is general enough to be used for any course.

We assume Conjuring will run on a single machine on a private network. Students (clients) on the same network will connect to the machine via their web browsers to access their Python workspace. The Python web interface is called Jupyter.

JupyterHub is used to manage multiple student accounts; each with their own isolated workspace.

JupyterHub recommends "The Littlest JupyterHub (TLJH)" or "Zero to JupyterHub (Z2JH)" to install, neither of which are appropriate in our case. We thus manually install it ourselves. It would be painful to provide installation instructions for all operating systems. Instead, we provide a docker container (which can run on any supported OS). The container itself runs Ubuntu 18.04.

Installation

Prerequisites

A machine which is accessible by students (e.g. via Ethernet or even a WiFi hotspot)
Docker CE (Community Edition)
docker-compose

Instructions

Download and if necessary customise (see below)
Run docker-compose up --build -d

The Conjuring JupyterHub machine should be built and accessible via a browser on http://localhost:8989.

To shut down, run docker-compose down. Student home directories will persist in the custom/home/ folder with the correct access permissions.

Customisation

Configuration files are all found within the custom/ directory.

Define packages to apt install
- Modify apt.txt
Define packages to conda install
- Modify environment.yml
Define packages to pip install
- Modify environment.yml or requirements.txt
Define alternative environments (kernels)
- Create environment-<name>.yml files
Define student usernames and passwords
- Modify users.csv (first row is a header and is ignored)
Define files which should be copied to each student's workspace
- Add files to home_default/
Define read-only files which should be shared for all students
- Add files to shared/
Change the base image to something other than ubuntu:18.04
- Modify base.Dockerfile

Auto-boot

A physical server can be configured to automatically start conjuring upon bootup, start a hotspot (or connect to a network), and monitor for external USB drives with additional configuration.

Use the autoboot.sh script for this purpose
- Run crontab -e and add @reboot cd /path/to/conjuring && ./autoboot.sh
- Run sudo apt-get install git openssh-server make sudo
- Run sudo visudo and add the line %sudo ALL=(ALL) NOPASSWD:ALL
- Enable Settings > Details > Users > Automatic Login

Notes:

The path monitored for a configuration folder is /media/*/*/conjuring/custom
The default network which is connected to is called Hotspot
- This can be set up in Settings > Wi-Fi > (settings icon) > Turn On Wi-Fi Hotspot..., then running (in a terminal) nm-connection-editor to rename the connection to Hotspot and edit the password
- Alternatively, choose a different default (e.g. external) network name by modifying autoboot.sh
- The NUC itself and all clients need to be on the same network. This means that the NUC doesn't have to act as a Wi-Fi hotspot if there's a pre-existing Wi-Fi network which everyone can connect to

FAQ

How does it all work?

If you are not familiar with docker, it may seem quite complicated. This overview (combined with the glossary below) might help.

docker-compose reads docker-compose.yml (and if it exits, docker-compose.override.yml) in order to make the following happen:

docker downloads the latest version of the ubuntu:18.04 image
docker follows the instructions in custom/base.Dockerfile to (re)build an image called conjuring/conjuring:base (based on ubuntu:18.04)
docker follows the instructions in the first half of Dockerfile to (re)build an image called conjuring/conjuring:core (based on conjuring/conjuring:base)
docker follows the instructions in the second half of Dockerfile to (re)build an image called conjuring/conjuring:latest (based on conjuring/conjuring:core)
- this references files from the src/ and custom/ directories to create users and Jupyter environments
docker creates a container called conjuring (based on conjuring/conjuring:latest) which also does the following:
- links custom/home/ to conjuring:/home/
- links custom/home_default/ (read-only)
- populates the container user home directories (conjuring:/home/*)
  - links custom/shared/ (read-only)
- starts a JupyterHub server accessible on the host at http://localhost:8989

All builds are "cached", i.e. unchanged lines from *Dockerfile will be not actually be re-run; saving time and bandwidth. Note that a line which references a changed file (from src/ or custom/) also counts as a changed line.

Run docker system prune to clear unused cache.

Glossary

Base image: the starting point for building a Docker image
- analogous to a clean OS (in this case ubuntu:18.04)
Layer: a single (cached) build step
- usually represented by a single line in a Dockerfile (e.g. apt-get install git)
Image: a sequence of layers (applied on top of a base image)
- analogous to a clean OS with things set up as specified in custom/ (in this case tagged conjuring/conjuring:latest)
Container: a sandboxed workspace derived from an image
- analogous to a running virtual machine (in this case named conjuring)
- easily stoppable, restartable, and disposable
- can be thought of as end-user-created layers which would never be formally part of a redistributable image
- can share files, network connections, and devices with the host computer

Images are pulled or built. Containers are created from them:

Pull: typically refers to downloading an image from the internet (which someone else built)
- usually only required when there is no source code available to allow for building locally (e.g. ubuntu:18.04)
Build: typically refers to pulling a base image, then building all the layers necessary to form an image
- usually once-off
Create: typically refers to making a container from an image
- often recreated for a semi-clean slate - especially if data is shared with the host computer so that no data is lost on disposal

Software

Python: a programming language designed to be very human-readable
Jupyter: an IDE (integrated development environment -- i.e. glorified text editor) which runs in a web browser
JupyterHub: a tool to manage multiple Jupyter servers for multiple users
git: a code version control tool
Docker: a virtual machine replacement tool
- allows running e.g. isolated Ubuntu Linux JupyterHub containers on any host operating system

Websites

GitHub: a website which hosts many git repositories

conjuring's People

Contributors

Stargazers

Watchers

Forkers

darribas jreades

conjuring's Issues

org

organisations (rather than my current @casperdcl user account) would allow better control (e.g. assigning & managing collaborator teams)

contact GitHub re:@conjuring
- success!
- invite everyone to @conjuring/maintainers
- when ready to make public, transfer this repo
reserve https://hub.docker.com/u/conjuring
~~maybe use https://github.com/search?p=1&q=KCL+king%27s+college+london&type=Users~~

live reconfig

Auto-detect when custom/ has changed in such a way that re-building the container isn't necessary.

Maybe more effort than it's worth.

user data persistence

custom/home:/home
ensure correct permissions
- including with changing UIDs
ensure docker can (re)build
document

User account generation bug?

I'm trying to build a new Conjuring project and, as far as I can tell, something strange is going on when the users.csv file is processed. The users.csv file has been created by my create_users.py script:
users.txt

Here's the relevant output from the build:

Step 24/31 : RUN groupadd -g ${GROUP_ID} conjuring
 ---> Running in 40e1b1cb6ad2
Removing intermediate container 40e1b1cb6ad2
 ---> 034caa53faa1
Step 25/31 : RUN useradd -D -s /bin/bash -N
 ---> Running in e688a2f064c0
Username,Password
geocomp1,passwdfoo1
geocomp2,passwdfoo2
geocomp3,passwdfoo3
geocomp4,passwdfoo4
geocomp5,passwdfoo5
geocomp6,passwdfoo6
geocomp7,passwdfoo7
geocomp8,passwdfoo8
geocomp9,passwdfoo9
geocomp10,passwdfoo10
geocomp11,passwdfoo11
Removing intermediate container e688a2f064c0
 ---> 423e54089b4d
Step 26/31 : COPY src/csv2useradd.sh custom/users.csv /opt/
 ---> ac1607c77b3e
Username,Password
geocomp1,passwdfoo1
geocomp2,passwdfoo2
geocomp3,passwdfoo3
geocomp4,passwdfoo4
geocomp5,passwdfoo5
geocomp6,passwdfoo6
geocomp7,passwdfoo7
geocomp8,passwdfoo8
geocomp9,passwdfoo9
geocomp10,passwdfoo10
geocomp11,passwdfoo11
Step 27/31 : RUN chmod 400 /opt/users.csv  # keep it secret from container users
 ---> Running in 5379b06813b9
Removing intermediate container 5379b06813b9

I cannot get any of the accounts listed above to work when the container starts, and there are obviously far fewer accounts listed above than should have been created. There are, however, 44 accounts in custom/home/.

Weird Client Network issue

I linked 3 pedestals to a soul funnel and logged out trying to log in I am unable to this is the error I get
https://hastebin.com/fagiyavazi.apache
Having someone else breaking the items allowed me to log in fine.

Conjuring 1.0.7

SSL & security

probably best to drop due to self-signed certificate warnings in browsers
document how to enable

Error replacing base image

I swap ubuntu:18.04 for darribas/gds:3.0 as the base image on the configurable base.Dockerfile and run docker-compose up --build -d. It works well until step 6, see below error log.

I'm not sure how to on top of my head, but if the base image has a working JupyterHub installed, all the reinstalling, etc. should be skipped (it's basically ready to go as it's built on top of the official jupyter Docker stack. Any suggestions?

Step 6/12 : RUN wget -q https://repo.continuum.io/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O /tmp/miniconda.sh   && echo 'e1045ee415162f944b6aebfe560b8fee */tmp/miniconda.sh' | md5sum -c -   && bash /tmp/miniconda.sh -f -b -p /opt/conda   && rm /tmp/miniconda.sh   && /opt/conda/bin/conda update --all -y -c conda-forge   && /opt/conda/bin/conda install -y -c conda-forge     sqlalchemy tornado jinja2 traitlets requests pip pycurl nodejs configurable-http-proxy   && /opt/conda/bin/pip install -U pip   && /opt/conda/bin/conda install -y -c conda-forge notebook jupyterlab   && /opt/conda/bin/conda clean -a -y
 ---> Running in 6019f92a1b3e
/tmp/miniconda.sh: OK
PREFIX=/opt/conda
reinstalling: python-3.7.0-hc3d631a_0 ...
Python 3.7.0
reinstalling: ca-certificates-2018.03.07-0 ...
reinstalling: conda-env-2.6.0-1 ...
reinstalling: libgcc-ng-8.2.0-hdf63c60_1 ...
reinstalling: libstdcxx-ng-8.2.0-hdf63c60_1 ...
reinstalling: libffi-3.2.1-hd88cf55_4 ...
reinstalling: ncurses-6.1-hf484d3e_0 ...
reinstalling: openssl-1.0.2p-h14c3975_0 ...
reinstalling: xz-5.2.4-h14c3975_4 ...
reinstalling: yaml-0.1.7-had09818_2 ...
reinstalling: zlib-1.2.11-ha838bed_2 ...
reinstalling: libedit-3.1.20170329-h6b74fdf_2 ...
reinstalling: readline-7.0-h7b6447c_5 ...
reinstalling: tk-8.6.8-hbc83047_0 ...
reinstalling: sqlite-3.24.0-h84994c4_0 ...
reinstalling: asn1crypto-0.24.0-py37_0 ...
reinstalling: certifi-2018.8.24-py37_1 ...
reinstalling: chardet-3.0.4-py37_1 ...
reinstalling: idna-2.7-py37_0 ...
reinstalling: pycosat-0.6.3-py37h14c3975_0 ...
reinstalling: pycparser-2.18-py37_1 ...
reinstalling: pysocks-1.6.8-py37_0 ...
reinstalling: ruamel_yaml-0.15.46-py37h14c3975_0 ...
reinstalling: six-1.11.0-py37_1 ...
reinstalling: cffi-1.11.5-py37he75722e_1 ...
reinstalling: setuptools-40.2.0-py37_0 ...
reinstalling: cryptography-2.3.1-py37hc365091_0 ...
reinstalling: wheel-0.31.1-py37_0 ...
reinstalling: pip-10.0.1-py37_0 ...
reinstalling: pyopenssl-18.0.0-py37_0 ...
reinstalling: urllib3-1.23-py37_0 ...
reinstalling: requests-2.19.1-py37_0 ...
reinstalling: conda-4.5.11-py37_0 ...
unlinking: asn1crypto-0.24.0-py37_1003
unlinking: ca-certificates-2019.6.16-hecc5488_0
unlinking: certifi-2019.6.16-py37_1
unlinking: cffi-1.12.3-py37h8022711_0
unlinking: chardet-3.0.4-py37_1003
unlinking: conda-4.7.10-py37_0
unlinking: cryptography-2.7-py37h72c5cf5_0
unlinking: idna-2.8-py37_1000
unlinking: libedit-3.1.20170329-hf8c457e_1001
unlinking: libffi-3.2.1-he1b5a44_1006
unlinking: libgcc-ng-9.1.0-hdf63c60_0
unlinking: libstdcxx-ng-9.1.0-hdf63c60_0
unlinking: ncurses-6.1-hf484d3e_1002
unlinking: openssl-1.1.1c-h516909a_0
unlinking: pip-19.2.1-py37_0
unlinking: pycosat-0.6.3-py37h14c3975_1001
unlinking: pycparser-2.19-py37_1
unlinking: pyopenssl-19.0.0-py37_0
unlinking: pysocks-1.7.0-py37_0
unlinking: python-3.7.3-h33d41f4_1
unlinking: readline-8.0-hf8c457e_0
unlinking: requests-2.22.0-py37_1
unlinking: ruamel_yaml-0.15.71-py37h14c3975_1000
unlinking: setuptools-41.0.1-py37_0
unlinking: six-1.12.0-py37_1000
unlinking: sqlite-3.29.0-hcee41ef_0
unlinking: tk-8.6.9-hed695b0_1002
unlinking: urllib3-1.25.3-py37_0
unlinking: wheel-0.33.4-py37_0
unlinking: xz-5.2.4-h14c3975_1001
unlinking: yaml-0.1.7-h14c3975_1001
unlinking: zlib-1.2.11-h516909a_1005
installation finished.

CustomValidationError: Parameter channel_priority = 'strict' declared in <<merged>> is invalid.
The value 'strict' cannot be boolified.

ERROR: Service 'core' failed to build: The command '/bin/sh -c wget -q https://repo.continuum.io/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O /tmp/miniconda.sh   && echo 'e1045ee415162f944b6aebfe560b8fee */tmp/miniconda.sh' | md5sum -c -   && bash /tmp/miniconda.sh -f -b -p /opt/conda   && rm /tmp/miniconda.sh   && /opt/conda/bin/conda update --all -y -c conda-forge   && /opt/conda/bin/conda install -y -c conda-forge     sqlalchemy tornado jinja2 traitlets requests pip pycurl nodejs configurable-http-proxy   && /opt/conda/bin/pip install -U pip   && /opt/conda/bin/conda install -y -c conda-forge notebook jupyterlab   && /opt/conda/bin/conda clean -a -y' returned a non-zero code: 1
(base) dh073146:conjuring dani$

Option to launch LabApp instead of NotebookApp

It would be great to either change the default or give users the option to change the default app to launch when they log in. Currently it's the NotebookApp, but I'd vote for Jupyter Lab since it's now on 1.0. I make that change on gds_env with:

https://github.com/darribas/gds_env/blob/master/gds_py/Dockerfile#L77

RUN echo "c.NotebookApp.default_url = '/lab'" \
 >> /home/$NB_USER/.jupyter/jupyter_notebook_config.py

I imagine this should be set up on every user's workspace or before the userspaces are created?

T1

Opening a thread for discussion of issues specifically relating to the T1 machines.

document ways to customise

simplest: copy conjuring and modify custom/ in-place
advanced: fork conjuring, git rm -rf custom/, git sumbodule add <repo> custom
anything in-between

Trying to compose...

I've cloned the repo and tried to run docker-compose up -d, to which I get:

ERROR: Service 'conjuring' failed to build: pull access denied for casperdcl/conjuring, repository does not exist or may require 'docker login'

This may well be a 'doh!' moment for me in about 5 minutes, but I'm playing the newb here. ;-)

jon

Build issue `conda.sh`

Next step fail:

Step 7/14 : RUN /conda.sh install -c conda-forge -q -y pip  && /conda.sh path_exec pip install --no-cache-dir -U pip
 ---> Running in d68b2a00d100
/usr/bin/env: ‘bash\r’: No such file or directory
ERROR: Service 'core' failed to build: The command '/bin/sh -c /conda.sh install -c conda-forge -q -y pip  && /conda.sh path_exec pip install --no-cache-dir -U pip' returned a non-zero code: 127

cli API

Hey @casperdcl, I've drafted up a note describing a workflow that relies mostly on what we have already, but requires couple of additions (cli interface and the custom.json option). The note is on the wiki, here:

https://github.com/conjuring/conjuring/wiki/CLI-workflow

How hard do you think it'd be to implement what is not available already?

Secure VNC/Remote Access

I've got x11vnc running but am having trouble encrypting traffic -- although I thought I'd followed both the X11VNC guide to generate a key pair and (after that didn't work) the appropriate OpenSSL guide to generate a key pair I am still getting a warning that I can't connect to VNC securely. I basically think that it would be a good idea to make it so that the machine can be manipulated remotely as soon as you know what IP address it's on. The issue with forwarding X11 over SSH (which I tried) is that you can't run some of the applets that do things like manage network connections, and actually need the full desktop to do that (so you need VNC). Currently the machine is running on the IP posted to [conjuring.reades.com] and it seems (somewhat scarily) like any user on the Kings Wifi connection can link to it (I'd expected much more port filtering and aggressive policing of subnets). I also think that we should be running SSH and VNC on non-standard ports: that's easy to set up once and then not have to deal with it again.

separate user & shared dirs

home_default:ro -> /home/user:rw
shared -> /shared:ro
decide on sync options
- one way sync, skip if existing is newer
- one way sync, skip if existing
- future: auto merge?
- future: let user decide?
document

Ideas

TODO

Links

blog
kingsgeocomp@GH
kingsgeocomp/code-camp
- entry-level repo that we ask our students to tackle over the summer before they start Geocomputation
kingsgeocomp/geocomputation
- Year 2, Term 1 intro module and has a setup.yml file for configuring a conda environment
kingsgeocomp/spatial-analysis
- Year 2, Term 2 module
jreades/urb-studies-predicting-gentrification
- about the limit of what I’d expect to demo on the system, so it might be a good test
darribas/gds_env
- attempt at compiling everything we usually need for geographic data science and put it into a container. In my mind, this is what would form the basis of the hub (since it pulls from the official Jupyter stack, the hub is enabled already, so it needs very little overhead to get it up and running. And the less reinventing we do, I think the better.
darribas/gds18
- At Liverpool I teach this course that encapsulates a lot of what I usually do in workshops and other teaching engagements. After four years, we’ve got the install relatively smooth on campus, but it’s still a native install on Windows which is not great. I’d love to teach it relying on say the gds_env, but apparently it’s very hard to deploy Docker on campus managed machine
- you can see a bit more of the background for the course on this recent paper 10.21105/jose.00042
darribas/gds_ufmg19
- one of my most recent workshops. I ran that out of binder (link in there) and out of Azure Notebooks (project link). Since it’s not clear how much longer Microsoft will keep it free, that’s an additional motivation Conjuring
jupyter/nbgrader
- setting and marking assignments
jupyter/repo2docker
- create a jupyter-enabled docker image from a repository (probably too general)
NVIDIA/nvidia-docker#568
- nvidia driver with docker-compose
ipython-contrib/jupyter_contrib_nbextensions
- useful Jupyter customisation options
jupyter-widgets/ipywidgets
- widgets provider
jupyter-widgets/ipyleaflet
- interactive maps in widgets
jupyterhub/nbgitpuller
- one-way sync (remote->local) git repos, auto-resolving conflicts
kidig/rpi-jupyter-lab
- "slow at best" but nice proof-of-concept
rsbivand/ectqg19-workshop
- workshop materials for early Sept 2019

non-expert version

clear installation instructions
clearly documented config options

Wi-Fi config

The NUC (probably) cannot connect to a Wi-Fi network (for internet) and serve a hotspot simultaneously. This is an issue when running in hotspot mode (especially when updating from a USB drive).

Work-arounds:

Modify autoboot.sh to use a reliable external hotspot instead (e.g. tethered phone)
Modify autoboot.sh to use an external network while updating, then switch back to hotspot mode when running

Additionally, it would be nice to have a config file (e.g. wifi.yml) for this:

OS of choice

Given we're about to receive the kit soon, what OS should we run on them? Requirement #1 is that it can run docker natively. That's about any linux distribution. But, beyond that, what features would we require?

Ease of use
Small memory footprint
Energy efficient (?)
Anything else?

Given those, which options do we have? A few I can think of but, by no means, not extensive:

Ubuntu LTS
Ubuntu LTS server-edition

Just opening this to kick off the discussion...

sudo user permissions

dev framework

contributing guidelines
github issues & pull request guidelines
continuous integration for testing (c4a9c12)
automatically deploy on dockerhub (https://hub.docker.com/u/conjuring, related to #18)
automatically deploy on GitHub packages
release history

package management

environment.yml
requirements.txt
apt.txt

USB config

support py2 vs py3 vs both

py2 is scheduled for deprecation in 2020 but perhaps it should be an option?

additionally (perhaps too much) support both py2 and 3 so that users can choose between kernels in their notebooks?

Build issue `is not valid yet`

Pretty sure this is not conjuring only problem, but it comes up on build...

Step 5/15 : RUN apt-get -yqq update && apt-get -yqq upgrade && apt-get -yqq install   wget git bzip2   && apt-get purge && apt-get clean && rm -rf /var/lib/apt/lists/*
 ---> Running in 5c5799730244
E: Release file for http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease is not valid yet (invalid for another 16h 36min 28s). Updates for this repository will not be applied.
E: Release file for http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease is not valid yet (invalid for another 16h 37min 4s). Updates for this repository will not be applied.
E: Release file for https://cloud.r-project.org/bin/linux/ubuntu/bionic-cran35/InRelease is not valid yet (invalid for another 9h 15min 24s). Updates for this repository will not be applied.
E: Release file for http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease is not valid yet (invalid for another 16h 37min 56s). Updates for this repository will not be applied.
ERROR: Service 'core' failed to build: The command '/bin/sh -c apt-get -yqq update && apt-get -yqq upgrade && apt-get -yqq install   wget git bzip2   && apt-get purge && apt-get clean && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

power-user version

Not much documentation needed.
Document customisation by modifying:

base.Dockerfile
docker-compose.yml

binary or pip install

a binary conjuring package:

may be installable via
- pip
  - would automatically install docker-compose
  - would require pip on the host
  - may not require git on the host
- a single downloadable script/url
  - may not require git on the host
can print docker install instructions/link
make use of a local custom.y(a)ml/json.config file and custom config directory
make use of a remote config (i.e. url of github repo)?
Overall would make this project even more similar to jupyter/repo2docker (albeit of course a faster multi-user version)

example use

running conjuring would automatically:

option 1: (current) manual git repo with optional submodules

git clone conjuring
cd conjuring

#### optional: convert to submodule
cd custom
git init
git add *
git commit -m "initial commit"
cd ..
mv custom orig_custom
git rm -rf custom
mv orig_custom custom
git submodule add ./custom custom
git commit -m "make custom/ a submodule"
#### end optional

# TODO: add users/packages to custom/ based on optional arguments to `conjuring init`?

docker-compose up --build -d

option 2: hidden "installed" git repo with executable on PATH, searches CWD for `custom`

CWD="${PWD}"
# make sure conjuring is up-to-date in ~/.conjuring
[ -d ~/.conjuring ] || git clone conjuring ~/.conjuring
cd ~/.conjuring
git reset --hard
git pull
# ensure $CWD/custom exists
[ -d "$CWD"/custom ] || mv custom "$CWD"/
rm -rf custom
ln -s "$CWD"/custom
docker-compose up --build -d
cd "$CWD"

installation could also be automated in a single script:

bash

no additional dependencies on linux
may require e.g. git-bash on windows (easy to install)

curl -o ~/.local/bin/conjuring https://conjuring.github.io/conjuring.sh

pip

requires python on the host, sort of defeating the point on conjuring
or via pip

pip install conjuring

option 3: docker image to do things

no additional dependencies
properly cross-platform
docker "conjuring-utils" image creates config (e.g. #37's python scripts can be run in the container)
the generated config can then be used to create and run the actual "conjuring" image

option X: my musings

I think this is overall the best option

The end user will:

install git
- sudo apt-get install git curl (linux)
- brew install git curl (mac)
- https://git-scm.com/download (win, includes git-bash and curl etc.)
(win) open git-bash terminal
curl -o conjuring https://raw.githubusercontent.com/conjuring/conjuring/master/src/conjuring.sh
- or e.g. curl -o ~/.local/bin/conjuring https://conjuring.github.io/conjuring.sh
./conjuring

The contents of conjuring are below.

cross-platform
would support (future/separate issue) config generation (e.g. #37 user generation) using more shell scripts

#!/usr/bin/env bash
set -e

CWD="${PWD}"
echo "Ensuring conjuring is up-to-date in ~/.conjuring"
[ -d ~/.conjuring ] || git clone https://github.com/conjuring/conjuring ~/.conjuring
pushd ~/.conjuring
git reset --hard
git pull
#files="$(ls -d */) $(ls *.yml Dockerfile)"
files=""  # blank for all
echo "Installing conjuring ($files) in $CWD"
echo -n "Overwrite all, overwrite older, keep existing, or quit [a/o/E/q]? "
read overwrite
case "$overwrite" in
  [aA])
    git archive --format=tar HEAD $files | (cd "$CWD" && tar -x -f - )
    ;;
  [oO])
    git archive --format=tar HEAD $files | (cd "$CWD" && tar -x -f - --keep-newer-files)
    ;;
  [eE]|"")
    git archive --format=tar HEAD $files | (cd "$CWD" && tar -x -f - --skip-old-files)
    ;;
  [qQ])
    exit 0
    ;;
  *)
    echo "Unknown option; aborting"
    exit 1
    ;;
esac
popd

docker-compose up --build -d

support multiple environments/kernels

probably define in environment*.yml
- kernel name precedence
  1. name: key in *.yml
  2. *.yml filename (environment-name/environment_name/environmentname)
  3. base
maybe support requirements-<envname>.txt
document
would also fix #17

custom install script

insert bash scripts at one or more points in the Dockerfile which make it possible to e.g. git pull ... && make install without touching any Dockerfiles

entrypoint user creation

Currently users are defined and created in the Dockerfile.

If creation is deferred to the entrypoint.sh, then per-user duplication of custom/home_default is avoided.

user management

users.csv

instructions/script option (on bootup)

WiFi hotspot
- nmcli connection up Hotspot
sshserver
- sudo ssh server start
docker container with mounted shared folder(s)
- docker-compose up ...
automate on boot (cron?)
- provide a crontab-editing script (probably best to just document)?
document
- echo @reboot cd /path/to/conjuring && ./autoboot.sh
- create new hotspot: Settings > Wi-Fi > (settings icon) > Turn On Wi-Fi Hotspot...
- rename to Hotspot and edit password: nm-connection-editor
- echo '@reboot cd /path/to/conjuring && ./autoboot.sh' | crontab -
- currently official documentation is in README.md but parts of https://github.com/casperdcl/conjuring/wiki/Install-Intel-NUC8i7BEH could be mostly movable to the README.md file, or docs website (#25)

todo

accidentally opened

managing jupyter user envs

probably all currently supported but needs documentation:

enabling jupyter (lab) extensions system-wide and per-user
setting default kernel
per-user package installation
possibly related to #12
from #30

Connect to WiFi on Boot

Perhaps I've broken something (the NUC no longer seems to log in by default, which may be linked to my attempts to configure VNC/X11VNC) but now the machine doesn't connect to eduroam when powering on. This in turn means that you end up needing a keyboard and monitor in order to get the machine online and running happily. Apologies if I managed to break this!

image tag re-building and documentation

From #14: document modifying tags in compose files for multi-image systems

fix changing custom/base.Dockerfile and then running docker-compose up -d won't rebuild
- suggest docker-compose up --build -d instead?
  - caveat: will rebuild everything if ubuntu:18.04 changes (would require (--)pull)
document tags in Dockerfiles and docker-compose files
- document modifying tags
mention occasional docker system prune or equivalent to clean up old dangling images

logo

maybe design an appropriate logo

Docs website

Once we have a testable version, it'd be good to set up a documentation website (that can live as static html on a docs folder on master and served up through GH Pages). That'd make it easier for people to get a sense of what the project does and how to get started.

@casperdcl EDIT:

currently official documentation is in README.md but parts of https://github.com/casperdcl/conjuring/wiki/Install-Intel-NUC8i7BEH could be mostly movable to the README.md file, or the proposed docs website

Explore mem, cpu, and user limits

It might be good add a config file to limit user resources:

https://zero-to-jupyterhub.readthedocs.io/en/latest/user-resources.html

I'm thinking particularly with the small machines, we'd rather shut one user down and kill their kernel rather than they shut the whole thing down.