Git Product home page Git Product logo

ucphhpc / nbi-jupyter-docker-stacks Goto Github PK

View Code? Open in Web Editor NEW
7.0 5.0 7.0 8.95 MB

A collection of docker jupyterlab/notebook images used at the University of Copenhagen

License: MIT License

Shell 0.07% Jupyter Notebook 93.51% Python 3.26% Makefile 0.14% Cuda 0.02% C 0.07% R 0.20% C++ 0.04% Jinja 2.69% Julia 0.01%
docker docker-images jupyterlab jupyter-notebook jupyter jupyterlab-docker

nbi-jupyter-docker-stacks's Introduction

nbi-jupyter-docker-stacks

A collection of docker jupyter notebook images used at UCPH.

Deployed notebooks are published at DockerHub.

Getting Started

Before any of the images can be built, the Dockerfiles for each Notebook has to be generated first. The reason for this is that by default each notebook defined by a Jinja2 template.

This entire process can also be simplified by simply executing make in the root directory, which will create an initial Python virtual environment and use the current architecture.yml file setup to generate the Dockerfiles and the associated GOCD configuration:

make

Alternatively, you can also setup the environment manually with the following steps.

To start with, you should ensure that the nessesary libraries are installed to generate the Notebooks, we recommend that you create a virtual environment and install the associated requiremnets:

username@hostname:~/jupyter/nbi-jupyter-docker-stacks$ python3 -m virtualenv venv
username@hostname:~/jupyter/nbi-jupyter-docker-stacks$ source venv/bin/activate
(venv) username@hostname:~/jupyter/nbi-jupyter-docker-stacks$ pip install -r requirements.txt

Afterwards, you should now be able to use the init-notebooks.py script to generate template and render the Jinja2 template files:

(venv) username@hostname:~/jupyter/nbi-jupyter-docker-stacks$ python3 init-notebooks.py
Generated the file: hpc-ocean-notebook/Dockerfile.latest
Generated the file: hpc-ocean-notebook/Dockerfile.20.02.7
Generated the file: hpc-ocean-notebook/Dockerfile.21.08.0
Generated the file: ocean-notebook/Dockerfile.latest
Generated the file: geo-notebook/Dockerfile.latest
Generated the file: bio-notebook/Dockerfile.latest
Generated the file: bio-bigdata-notebook/Dockerfile.latest
Generated the file: bio-bsa-notebook/Dockerfile.latest
Generated the file: sme-notebook/Dockerfile.latest
Generated the file: jwst-notebook/Dockerfile.latest
Generated a new GOCD config in: ...jupyter/nbi-jupyter-docker-stacks/1.gocd.yml
Generated a new Makefile in: .../jupyter/nbi-jupyter-docker-stacks/Makefile

In addition to generating the Dockerfiles, the init-notebooks.py script also generates a GOCD configuration and an up-to-date Makefile configuration.

The GOCD 1.gocd.yml configuration file is used to automatically build and test the defined notebook pipelines as part of the UCPH CI/CD infrastructure.

The Notebooks and the associated GOCD configuration file are generated based on the definitions specified in the architecture.yml file.

Building

After the repository has been configured with the init-notebooks.py script as specified in the Getting Started section, every notebook is ready to be built.

To build a particular notebook pass the directory name that contains the Dockerfile to make:

make build/base-notebook TAG=latest

This will start the build procedure, the output will be a container image that is tagged with the word "latest".

To change this, override the TAG variable when calling a make target, e.g.:

make build/base-notebook TAG=edge

However, the Makefile expects that a Dockerfile is available that has the specified TAG as a postfix. For instance, when using TAG=edge for the base-notebook, the Makefile requires that there is a base-notebook/Dockerfile.edge file.

Testing

The Makefile also provides targets for test and push options. For example, first test the edge tagged image, then build an official latest tagged image and push it to the preset OWNER variable's dockerhub repository:

make test/base-notebook TAG=latest
make build/base-notebook TAG=latest
make push/base-notebook TAG=latest

To evaluate the entire stack, a target to 'build-all'/'test-all' is also available i.e.:

make build-all
make test-all

Contributing

To contribute either a change to an existing Notebook or introduce a new Notebook is fairly "easy".

Making a change

A change can be introduced to one of the current Notebooks, by altering the selected Notebook setup in its associated directory.

For instance, if a change is to be introduced to the datascience-notebook, the change must be applied either to the datascience-notebook/Dockerfile.j2 Jinja template file, its associated Conda environment file datascience-notebook/environment.yml, or the Python datascience-notebook/requirements.txt file.

Therefore, to introduce a new Python package to the datascience-notebook, the package should be added to the datascience-notebook/requirements.txt file. Hereafter, the change can be tested by following the steps outlined in the Getting Started, Building, and Testing sections above.

In addition, if the change involves introducing a new feature, such as a new Python package, it is prudent that the introduction also ensures that the new package is tested as part of the Notebook image. To accomplish this, a Jupyter Notebook test file should be added, that validates that the added package functions as expected in the Notebooks associated tests directory. For example, if the numpy package was introduced into the datascience-notebook, a test Notebook such as the datascience-notebook/tests/numpy.ipynb should be included.

Introducing a Notebook

To introduce a new Notebook, we recommend that the following approach is adopted to ensure a succesfull introduction. Firstly, the architecture.yml file should be updated to include the new Notebook. The Notebook should be defined as follows in the architecture file:

architecture:
        <notebook-name>:
                <notebook-image-tag>
                        parent:
                                owner: <parent-image-owner>
                                image: <parent-image-name>
                                tag: <parent-image-tag>
                                pipeline_dependent: <boolean>
                        parameters:
                                <dockerfile-template-variable-name>: <dockerfile-template-variable-value>

Secondly, a directory with the name of the new Notebook should be created in the root directory.

Thirdly, create a Jinja Dockerfile template inside the new directory that defines the default build for the new Notebook. Inspiration here can be gathered from one of the existing Notebooks. Importantly though, is that the template uses the following beginning and end:

FROM {{ parent }}
...

# Ensure that container Runs as
USER $NB_UID

Where the parent declaration must be defined in the architecture.yml file, and is extracted by the init-notebooks.py script when the subsequent Dockerfiles are generated. Finally, the new Notebook directory should included a tests directory that must contain a test_notebook.py, test.sh, and a requirements.txt file. Examples of these files can be found in the root res/tests directory.

Security

Any security related questions/issues/inquries should be directed at [email protected]

nbi-jupyter-docker-stacks's People

Contributors

dependabot[bot] avatar emilstevnsborg avatar enryh avatar gbrammer avatar guidov avatar jonasbardino avatar madsbk avatar patchofscotland avatar rasmunk avatar sebastianprehn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

nbi-jupyter-docker-stacks's Issues

JWST notebook photoutils.patch error

photoutils.patch error

When building the dev build of the jwst-notebook, the build fails with the following relevant error code:

7.642     from .base import BaseDifferential, BaseRepresentation, BaseRepresentationOrDifferential
7.642   File "/opt/conda/envs/python3/lib/python3.10/site-packages/astropy/coordinates/representation/base.py", line 12, in <module>
7.642     from astropy.coordinates.angles import Angle
7.642   File "/opt/conda/envs/python3/lib/python3.10/site-packages/astropy/coordinates/angles/__init__.py", line 11, in <module>
7.642     from .utils import *
7.642   File "/opt/conda/envs/python3/lib/python3.10/site-packages/astropy/coordinates/angles/utils.py", line 21, in <module>
7.642     from astropy.coordinates.representation import (
7.642 ImportError: cannot import name 'SphericalRepresentation' from partially initialized module 'astropy.coordinates.representation' (most likely due to a circular import) (/opt/conda/envs/python3/lib/python3.10/site-packages/astropy/coordinates/representation/__init__.py)

starting on this line in the code

RUN mamba run -n python3 python /tmp/photutils.patch \

@gbrammer You initially created this Notebook with the file photoutils.patch and I wanted to ask if you had any idea for a fix to this, as we would like to be able to use the new JupyterLab 4.0.11 introduced in eac4bce for its security fixes.

Possible solutions

When looking at photoutils install, it recommends conda install -c conda-forge photutils which could be a potential solution, unless the patch does something specific and is not just legacy at this point?

CERN ROOT notebook

The data analysis framework ROOT from CERN have been requested as a feature for our notebook interface.

The package itself is very large and has a fairly complex installation, which lends itself to creating its own notebook.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.