This project contains nix expressions for the syslab courses at HTWG Konstanz. The expression are written in the nix language for the package manager with the same name.

This README aims to be a mixture of

• nix tutorial
• presentation for the design decisions
• labshell installation usage documentation

• 1. Syslab Nix Expressions
  • 1.1. Overview
  • 1.2. Requirements
  • 1.3. Repository Overview
  • 1.4. Usage and Design
    • 1.4.1. The labshell derivation
    • 1.4.2. Update Handling
    • 1.4.3. The script pkgs/labshell/src/labshell.sh
    • 1.4.4. Logical modes and choice of the shell flavor
      • 1.4.4.1. 'Interactive' Mode
        • 1.4.4.1.1. Zero arguments passed
        • 1.4.4.1.2. One Argument passed
      • 1.4.4.2. Shell mode
        • 1.4.4.2.1. Passed via environment variable LABSHELL_FLAVOR
        • 1.4.4.2.2. Invocation as #! (sharp-bang) Interpreter
      • 1.4.4.3. The mkShellDerivation(.nix) function
    • 1.4.5. Shell derivations (shells/default.nix) - Labshell flavors
  • 1.5. Installation
    • 1.5.1. The labshell application
    • 1.5.2. The labshell_${flavor} wrappers
  • 1.6. Development
    • 1.6.1. Clone Repository
    • 1.6.2. Test changes to a shell derivation
    • 1.6.3. Tests
    • 1.6.4. Contribution Policy

In order to make use of this project you need to have nix and its many utilities installed locally. After the installation you should have these tools in your PATH:

• nix-prefetch-url - Used to download and store files in the nix store
• nix-instantiate - instantiates expressions to derivations
• nix-build - Build derivations without installing them to the environment
• nix-env - Evaluates expressions to derivations and installs these into your current shell environment You can test this with

  nix-env -i hello (will work only as root on the HTWG Syslab Containers for now)

• nix-shell - launches a new shell environment based on the derivations built by nix-instantiate

The following is a simplified tree layout for the files in this repository and their purpose:

.
├── default.nix (the nix entry point, exposes packages and shell flavors)
├── README.md (this file)
├── ci (scripts for integration tests)
│   ├── complete.sh
│   ├── install.sh
│   ├── source.sh
│   └── test.sh
├── pkgs (package definitions)
│   ├── configured
│   │   └── (...)
│   ├── labshell (the package provides the wrapper scripts)
│   │   ├── default.nix
│   │   └── src
│   │       └── labshell.sh (the script that executes nix-shell)
│   └── overrides
│       └── default.nix (some nix configuration overrides)
└── shells
    ├── default.nix (the definitions for the shell flavors)
    └── mkShellDerivation.nix (the function used to build shell flavors)

The main components are the nix expressions themselves, and the labshell.sh script source code. The latter has an installable nix package in this repository.

The labshell derivation that is written in pkgs/labshell/default.nix allows to install a shell script that wraps the execution of pkgs/labshell/src/labshell.sh. It is used to set default values for environment variable that alter the runtime behavior of labshell.sh.

To understand how this project works, it is very useful to read the nix manual section about writing nix expressions along with this README. Especially important is to understand how mkDerivation works to build an installable package, and how the installation procedure can be adjusted. If in doubt, contact the IRC channel #nixos at FreeNode and feel free to ping steveeJ.

The output path of the derivation contains only one binary:

/nix/store/<HASH>-labshell/
└── bin
    └── labshell

On a local build this file looks like this, built from a local copy of the repository with nix-build -A labshell --arg labshellExpressionsUpdateFromLocal true

#! /nix/store/53h800j8kgpj0a349f7wxa5hgkj1vby2-bash-4.4-p12/bin/bash -e
export LABSHELL_EXPRESSIONS_LOCAL="${LABSHELL_EXPRESSION_LOCAL:-/home/steveej/src/htwg-syslab/nix-expressions}"
export LABSHELL_EXPRESSIONS_REMOTE_URL="${LABSHELL_EXPRESSIONS_REMOTE_URL:-/home/steveej/src/htwg-syslab/nix-expressions}"
exec "/home/steveej/src/htwg-syslab/nix-expressions/pkgs/labshell/src/labshell.sh"  "${extraFlagsArray[@]}" "$@"

This wrapper is generated in the installation step of the labshell nix derivation.

When the wrapper is run, a new shell is spawned and PATH and other environment variables are altered so that no utilities from the host are accessible. This is done by using nix-shell's --pure argument.

Updates only take place if LABSHELL_UPDATE is not 0.

In the above example the URL for updates is a local directory, which is very practical for development. On production installations this URL will download the archive from a online repository.

This is where the hard work is done to figure out which of nix-* tools needs to be invoked at which time. The main job of the script is to set up the invocation parameters for the nix-shell with the shell flavor that can be passed to, as described in the usage section.

This section gives an idea of the supported features.

The labshell application currently supports two logical modes: interactive and shell. These modes are not explicitly specified but inferred from the way the labshell script is invoked.

This mode is to launch a shell in which the user works interactively.

If labshell is run without any arguments it will use the base flavor and start an interactive shell with it.

In case there is exactly one arguments passed, it will be interpreted as the flavor.

labshell FLAVOR

In 'shell' mode, the invocation of the script behaves the same as invoking bash, just that the shell is run with the environment defined by LABSHELL_FLAVOR.

In this mode, labshell behaves like a shell which allows it to be placed in the SHELL environment variable. In fact, if you spawn a labshell you will see something like this:

steveej@steveej-laptop ✓ »base@2« ~/src/htwg-syslab/nix-expressions
$ echo $SHELL
/nix/store/5zc5ljyp55k0533fp4z46kx7x02pyz2s-labshell_base/bin/labshell_base

This variable is used by many utilities like tmux or vim, which enables them to access the tools that are defined in the shell environments.

In this case, the invocation looks like this

[LABSHELL_FLAVOR=flavor...] labshell [ARGUMENTS PASSED TO BASH ...]`

The labshell script is designed to be used as a script interpreter, which enables the users to write scripts that can be invoked in batch jobs.

The generic syntax of a shell script looks like this:

#/usr/bin/env labshell
#!LABSHELL_OPTION1=VALUE1
#!LABSHELL_OPTION2=VALUE2
#!  (...)
#!/path/to/real/interpreter argument
commands for the real interpreter
(...)

Valid LABSHELL_OPTIONS are currently:

• LABSHELL_FLAVOR=flavor - has the same effect as setting the LABSHELL_FLAVOR=flavor environment variable, or arg1 in the interactive mode

It is best to prepend /usr/bin/env instead of using an unreliable hardcoded path.

#!/usr/bin/env labshell
#!LABSHELL_FLAVOR=code
#!/bin/sh -xe
echo I'm verbose and running with the ${LABSHELL_FLAVOR} flavor.

This nix expression represents a function that emits an installable derivation, that can also be used to instantiate a nix-shell environment.

Some of the cornerstones of this derivation:

• The list of packages declared by the buildInputs attribute of packages will be available in the environment, which is called a flavor within the context of this project.

• The shellHook string are bash commands that are run just before the shell is spawned.

  It can be used to set environment variables or perform other initialization tasks.

The flavors effectively define an environment construct that consists a list of packages (-> buildInputs) and a string (-> shellHooks)

The dependencies and strings are organized by use-case and laboratory requirements. The file should be self explanatory but roughly the following is true:

• all flavors are built with mkShellDerivation
• base and code collections for generic tasks and text editing (does not include language specific code completion tools)
• admin tools
• every programming language has it's own package collection for tools and compilers
• every course has its own shell derivation

The flavors that are ultimately available for installation are exposed in the default.nix file by inheriting these from shells/default.nix, which was just discussed.

(...)
  inherit ( callPackage ./shells { } )
    shell_base
    shell_admin
    (...)
    ;
(...)
;

1. Install labshell Package from the Repository on the target machine.

   REV=sj-improve-labshell-script \
       nix-env -iA labshell \
       --argstr labshellExpressionsRemoteRev ${REV} \
       -f https://github.com/htwg-syslab/nix-expressions/archive/${REV}.tar.gz

   nix-env will download and unpack the tar archive and automatically read its default.nix file, and start evaluating the expression pkgs.labshell, which is a derivation that installs a labshell wrapper script in the current PATH. The example uses a development branch.

   For the HTWG Syslab Container infrastructure, this is accomplished via ansible-playbook (convenience link, requires permission)

2. On the target machine you can now see that labshell is installed. In fact, the script in PATH is a wrapper to the labshell wrapper :-)

   $ type labshell
   labshell is /home/sjunker/.nix-profile/bin/labshell
   $ cat $(type -P labshell)
   #! /nix/store/hi4j75r312lsjhpdln9p8blyixs59hbs-bash-4.4-p12/bin/bash -e
   exec "/nix/store/28pbgmrly69j0yvjxmrgighva0nl5jd3-sj-improve-labshell-script.tar.gz/pkgs/labshell/src/labshell.sh"  "${extraFlagsArray[@]}" "$@"

   This wrapper will probably be extexnded with default environment variables.

The installation is analogue to the labshell wrapper script. The following example uses a local repository for bootstrapping.

steveej@steveej-laptop ✗ ~/src/htwg-syslab/nix-expressions
$ nix-env -iA shell_base -f .
replacing old ‘shell_base’
installing ‘shell_base’
these derivations will be built:
  /nix/store/7j538si1ggq90c1vzpdz9pf7ivicmwyv-shell_base.drv
building path(s) ‘/nix/store/37fa5qrpbrqy9p905kd48bd299wyk0d4-shell_base’
installing
building path(s) ‘/nix/store/m02ly71nmxxhcinkmds9ij6xlylgpx6v-user-environment’
created 1749 symlinks in user environment

steveej@steveej-laptop ✓ ~/src/htwg-syslab/nix-expressions
$ cat $(type -P labshell_base)
#! /nix/store/hi4j75r312lsjhpdln9p8blyixs59hbs-bash-4.4-p12/bin/bash -e
export LABSHELL_EXPRESSIONS_LOCAL="/home/steveej/src/htwg-syslab/nix-expressions"
export LABSHELL_EXPRESSIONS_REMOTE_URL="https://github.com/htwg-syslab/nix-expressions/archive/master.tar.gz"
export LABSHELL_MODE="shell"
export LABSHELL_UPDATE="0"
export LABSHELL_FLAVOR="base"
unset LABSHELL_EXPRESSIONS_REMOTE_URL
exec "/home/steveej/src/htwg-syslab/nix-expressions/pkgs/labshell/src/labshell.sh"  "${extraFlagsArray[@]}" "$@"

For development the repository needs to be cloned locally.

$ git clone [email protected]:htwg-syslab/nix-expressions.git
$ pushd nix-expressions

The following example assumes $PWD is the path of the git repository.

1. Make your desired changes E.g., change the shell derivations and its dependencies:

   $EDITOR shells/default.nix

   Read the comments in the file :-) If you add a new shell, you also need to add the derivation name in the default.nix in the repository root.

2. Try out your changes locally

   1. Use a local build of the labshell wrapper:

      steveej@steveej-laptop ✓ ~/src/htwg-syslab/nix-expressions
      $ nix-build -A labshell --arg labshellExpressionsUpdateFromLocal true
      /nix/store/a8893zppmlb75lmir7czfr3mkb2r0qla-labshell
      steveej@steveej-laptop ✓ ~/src/htwg-syslab/nix-expressions
      $ ./result/bin/labshell
      Spawning shell with settings:
          flavor: 'base'
          #!-Interpreter: 0
      Please wait...
      Using expressions on filesystem /home/steveej/src/htwg-syslab/nix-expressions
      Environment initialized!
      steveej@steveej-laptop ✓ »base@2« ~/src/htwg-syslab/nix-expressions
      $

   2. Use result/bin/labshell to test changes to the local repository

• If nix-env --install ... works for you, simply run: ./ci/complete.sh
• Document how to run it with nix-build result.

The master branch is protected, hence all changes must go through Pull-Requests which require status checks.