spack / spack-tutorial Goto Github PK

Standalone Spack Tutorial Repository

License: Other

Python 47.88% Makefile 10.51% Shell 39.80% Dockerfile 1.68% CSS 0.13%

spack-tutorial's Introduction

Spack Tutorial

Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.

This repository houses Spack's hands-on tutorial, which is a subset of Spack's full documentation (or you can run spack help or spack help --all).

This tutorial covers basic to advanced usage, packaging, developer features, and large HPC deployments. You can do all of the exercises on your own laptop using a Docker container. Feel free to use these materials to teach users at your organization about Spack.

Updating the tutorial

Create a new branch named for the event/milestone that corresponds to the new version you want to create.
Upload screen shot of first slide (244px wide, .png) to images directory following existing file-naming convention.
Upload PDF of slide deck to slides directory following existing file-naming convention.
Update index.rst with event name and date; full citation; and file paths for image and PDF.
Update this README (lines 3 and 7) with link to new version's URL.
Build docs locally.
Push changes to GitHub and active new tag/version on Read the Docs.
Build new version on Read the Docs.

Updating the tutorial container

The Spack tutorial container is automatically built from repository by this GitHub action. The latest version is available at

ghcr.io/spack/tutorial:latest

and is rebuilt on a schedule. It can also be triggered manually.

The tutorial image builds on top of the container image that runs in Spack CI, which is built in a different repository at spack/gitlab-runners

Automatically generating command ouputs

The tutorial rst files include output from Spack commands. This process is automated, and it is recommended not to run commands manually.

Note: as a preliminary step, check your terminal width. All current outputs are generated on a fixed terminal width 94; deviating from that can cause unnecessarily large diffs:

$ tput cols
94

To regenerate the outputs, run:

make -C outputs -j <N>

This runs each outputs/<section>.sh script in parallel in a container, and collects outputs in outputs/raw/*. When all complete succesfully, the outputs are post-processed and put in outputs/.

In case you want to restrict to particular sections, or if you need to modify the container executable and flags, specify those as variables in outputs/Make.user:

sections := basics scripting
DOCKER := sudo docker

make will regenerate the relevant outputs when outputs/<section>.sh files are modified.
To start from scratch, run make clean
make run-<tab> can also be used to regenerate a particular section, but notice it will only create raw outputs.

License

Spack is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.

All new contributions must be made under both the MIT and Apache-2.0 licenses.

See LICENSE-MIT, LICENSE-APACHE, COPYRIGHT, and NOTICE for details.

SPDX-License-Identifier: (Apache-2.0 OR MIT)

LLNL-CODE-811652

spack-tutorial's People

Contributors

Stargazers

Watchers

spack-tutorial's Issues

Package Creation tutorial has too much beginner stuff

The Package Creation tutorial has a long section on setting SPACK_ROOT, PATH, and EDITOR, sourcing the setup script, etc. At this point in the tutorial, we've done this a million times, no need to specify this again. In fact, things like EDITOR should be moved to earlier points in the tutorial.

install netcdf ^mpich section needs an overhaul

Hi,

thanks for building this tutorial, I found an issue in the Extra query parameters section. This does not work for me with the current releases/v0.15 branch.

There is no netcdf package, so this should probably be netcdf-c
Installing netcdf-c results in the following error which I cannot figure out:

==> Installing netcdf-c
==> No binary for netcdf-c found: installing from source
==> Error: AttributeError: 'list' object has no attribute 'directories'

/shared/spack/lib/spack/spack/build_environment.py:862, in child_process:
        859            tb_string = traceback.format_exc()
        860
        861            # build up some context from the offending package so we can
  >>    862            # show that, too.
        863            package_context = get_package_context(tb)
        864
        865            build_log = None

Update advanced packaging section

The advanced packaging section still refer to:

def setup_environment(self, build_env, run_env):
    pass

etc. which have been deprecated.

documentation console script are not readable

The console log of the documentation does not provide the good information. Instead there is the path of the needed files.
https://github.com/spack/spack-tutorial/blob/main/tutorial_packaging.rst

Make sc19 tutorial branch + RTD version

Add dot to the tutorial image for SC19

In the "Basic Installation" part of the tutorial we show an example that should generate a pdf with the spec's DAG:

$ spack graph --dot trilinos | dot -Tpdf trilinos_graph.pdf
bash: dot: command not found

but it currently fails since dot in not installed.

Dry runs of SC19 tutorial sections

@becker33 and @scheibelp to do dry runs of tutorial sections sometime later this week.

Make last AMI

Advanced packaging needs updating wrt setup_environment

As pointed out in spack/spack#23500, the advanced packaging guide is outdated. It is referring to setup_environment, which is no longer being used since spack/spack#11115 was merged.

There are now two methods: setup_build_environment and setup_run_environment.

Make project for SC19 tutorial

Update slides for SC19 Tutorial

Create new slides
Add to tutorial repo

Finish up Spack Stacks Tutorial

At the end of the Spack Stacks Tutorial, the ls views/default command shows no results, but since we installed more packages than expected, there actually is output. Update this with the correct output.

Envirionments: zlib-ng version vs zlib version

In the mpi-hello.c program in the environments section, we need to print both ZLIB_VERSION and ZLIBNG_VERSION.

Hash in basics tutorial not updated

https://github.com/spack/spack-tutorial/blame/main/tutorial_basics.rst#L166 says the hash of zlib is /2f2, but https://github.com/spack/spack-tutorial/blob/main/outputs/basics/tcl-zlib-hash.out uses /zqd. Needs to be updated.

Fix highlighting in dev workflows tutorial

Catch up script

Many users come to the tutorial later in the day, especially at the halfway point. Or they mess something up in their environment or fall behind. It would be good to have a script that they can source to catch up to the current point in the tutorial. Especially checking out the right branch, gpg trusting, and clearing erroneous config files.

Finalize section assignments for SC19

We settled on the following section breakdown:

Morning

Intro (slides) @tgamblin
Basic installation @becker33

Break

Afternoon

Break

Basic tutorial instructions don't work

Actually, the below is a re-enactment. In reality I started reading the docs, said "That's not gonna work", tried it, got the failure I expected, cleaned up and started from scratch so that I could provide a clean bug (is that an oxymoron?)

There's something missing in the prose for the basic tutorial, it appears to be running as a regular user (prompt is $ ) but there's no issue with creating things in /mirror).

Is this actually happening in a Docker image that already has a '/mirror'?

Is '/mirror' an Ubuntu-ism?

While we're at it, why are we monkeying with mirrors as the 4th spack command in a basic tutorial. Presumably there's a good reason (e.g., we're working in an image (Docker, AMI, ???) that contains a bunch of prebuilt stuff to save us time), but we should state it.

If there's a bunch of prebuilt stuff, is there a script that those of us playing along at home can run to "catch up"?

In I'll touch up docs in exchange for a clue about what's supposed to be happening?

I followed the Basic Instructions Tutorial instructions, except that instead of setting up

an AWS instance running Ubuntu 18.04

I set up a Digital Ocean Droplet running "CentOS 7.6 x64" from their prebuilt images.

Steps to reproduce the issue

git clone https://github.com/spack/spack
cd spack/
. share/spack/setup-env.sh
spack install zlib
spack mirror add tutorial /mirror
spack gpg trust /mirror/public.key

Error Message

[centos@centos-s-6vcpu-16gb-sfo2-01 spack]$ spack gpg trust /mirror/public.key
gpg: keyring `/home/centos/spack/opt/spack/gpg/secring.gpg' created
gpg: keyring `/home/centos/spack/opt/spack/gpg/pubring.gpg' created
gpg: can't open `/mirror/public.key': No such file or directory
gpg: Total number processed: 0
==> Error: Command exited with status 2:
'gpg2' '--import' '/mirror/public.key'
[centos@centos-s-6vcpu-16gb-sfo2-01 spack]$

Spack installation fails when using recommended tutorial Docker image

I'm new to spack, and when I try to follow the tutorial, the installation fails when I use the recommended Docker container. The documentation recommends running the container like so:

 docker pull ghcr.io/spack/tutorial:sc23
 docker run -it ghcr.io/spack/tutorial:sc23

I'm on an Apple Silicon Mac, so I've got to specify a working platform:

 docker pull --platform=linux/amd64 ghcr.io/spack/tutorial:sc23
 docker run -it --platform=linux/amd64 ghcr.io/spack/tutorial:sc23

From there, after cloning the spack repo into the container with git clone --depth=100 --branch=releases/v0.21.0 https://github.com/spack/spack.git ~/spack the setup-env.sh script fails to run:

$ . spack/share/spack/setup-env.sh 
==> Error: SPACK_ROOT must point to spack's prefix when using qemu-x_
Run this with the correct prefix before sourcing setup-env.sh:
    export SPACK_ROOT=</path/to/spack>

This message only appears once; future runs of the setup-env.sh script produce no output. I've found that the message persists even when I set the SPACK_ROOT environment variable to either /home/spack/spack/bin/spack or /home/spack/spack/bin

I've been able to set up spack on EC2 instances outside of a container environment, but I'd like to run through the tutorial and experimenting with the tool locally in a container, as recommended.

First mention of "spack install netlib-scalapack ^openmpi ^netlib-lapack" does not state not to run command.

As I go along this tutorial (which is awesome!), I copy-paste most of the commands that I encounter to work along inside my own spack installation. When I copy-paste this line "spack install netlib-scalapack ^openmpi ^netlib-lapack" from here it fails with an error. After some digging I found out that this is the expected behavior and should only be encountered later in the tutorial.

I guess a note saying "don't copy-paste these commands here" right in front of this would help:
$ spack install netlib-scalapack ^openmpi ^openblas
$ spack install netlib-scalapack ^mpich ^openblas
$ spack install netlib-scalapack ^openmpi ^netlib-lapack
$ spack install netlib-scalapack ^mpich ^netlib-lapack
$ spack install py-scipy ^openblas

Redo output for first 4 tutorial sections to match ubuntu18 container

Redo output in the tutorial for these sections:

Basic installation
Configuration
Package creation
Modules

config section: make more explicit that we remove old packages config

DESCRIPTION

The last command in https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html failed because we have defined zlib as an external package

  zlib:
    paths:
      [email protected]%[email protected]: /usr
    buildable: false

ERROR

spack1@ip-172-31-62-20:~$ spack -d install --no-cache --verbose --overwrite --yes-to-all zlib
==> [2020-11-09-17:10:27.166673] Imported install from built-in commands
==> [2020-11-09-17:10:27.167533] Reading config file /home/spack1/spack/etc/spack/defaults/config.yaml
==> [2020-11-09-17:10:27.178392] Reading config file /etc/spack/config.yaml
==> [2020-11-09-17:10:27.179304] Reading config file /home/spack1/.spack/config.yaml
==> [2020-11-09-17:10:27.180458] Imported install from built-in commands
==> [2020-11-09-17:10:27.181849] Reading config file /home/spack1/spack/etc/spack/defaults/repos.yaml
==> [2020-11-09-17:10:27.218556] Reading config file /home/spack1/spack/etc/spack/defaults/packages.yaml
==> [2020-11-09-17:10:27.231927] Reading config file /etc/spack/packages.yaml
==> [2020-11-09-17:10:27.233263] Reading config file /home/spack1/.spack/packages.yaml
==> [2020-11-09-17:10:27.245759] Reading config file /home/spack1/.spack/linux/compilers.yaml
==> [2020-11-09-17:10:27.268682] DATABASE LOCK TIMEOUT: 3s
==> [2020-11-09-17:10:27.268733] PACKAGE LOCK TIMEOUT: No timeout
==> [2020-11-09-17:10:27.282552] TEMPORARY DIRECTORY CREATED [/tmp/tmpr7kh_doq]
Traceback (most recent call last):
  File "/usr/lib/python3.6/shutil.py", line 550, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/usr' -> '/tmp/tmpr7kh_doq/usr'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/spack1/spack/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/home/spack1/spack/lib/spack/spack/main.py", line 762, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/spack1/spack/lib/spack/spack/main.py", line 490, in _invoke_command
    return_val = command(parser, args)
  File "/home/spack1/spack/lib/spack/spack/cmd/install.py", line 390, in install
    with fs.replace_directory_transaction(concrete.prefix):
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/spack1/spack/lib/spack/llnl/util/filesystem.py", line 639, in replace_directory_transaction
    shutil.move(src=directory_name, dst=tmp_dir)
  File "/usr/lib/python3.6/shutil.py", line 562, in move
    rmtree(src)
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'service.sh'

SOLUTION

I removed the zlib external package section and then rerun command. Either we skip the last two section or explicitly document that user must remove the zlib section

spack1@ip-172-31-62-20:~/spack$ spack --debug install --no-cache --verbose --overwrite --yes-to-all zlib
==> [2020-11-09-17:17:55.575690] Imported install from built-in commands
==> [2020-11-09-17:17:55.576605] Reading config file /home/spack1/spack/etc/spack/defaults/config.yaml
==> [2020-11-09-17:17:55.587789] Reading config file /etc/spack/config.yaml
==> [2020-11-09-17:17:55.588792] Reading config file /home/spack1/.spack/config.yaml
==> [2020-11-09-17:17:55.589997] Imported install from built-in commands
==> [2020-11-09-17:17:55.591546] Reading config file /home/spack1/spack/etc/spack/defaults/repos.yaml
==> [2020-11-09-17:17:55.629553] Reading config file /home/spack1/spack/etc/spack/defaults/packages.yaml
==> [2020-11-09-17:17:55.643406] Reading config file /etc/spack/packages.yaml
==> [2020-11-09-17:17:55.644831] Reading config file /home/spack1/.spack/packages.yaml
==> [2020-11-09-17:17:55.656147] Reading config file /home/spack1/.spack/linux/compilers.yaml
==> [2020-11-09-17:17:55.677500] '/usr/bin/clang-6.0' '--version'
==> [2020-11-09-17:17:55.695873] DATABASE LOCK TIMEOUT: 3s
==> [2020-11-09-17:17:55.695947] PACKAGE LOCK TIMEOUT: No timeout
==> [2020-11-09-17:17:55.709946] '/usr/bin/clang-6.0' '--version'
==> [2020-11-09-17:17:55.722778] Initializing the build queue for zlib
==> [2020-11-09-17:17:55.733230] Acquiring a write lock on zlib with timeout 1e-09
==> [2020-11-09-17:17:55.734804] Creating stage lock spack-stage-zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5
==> [2020-11-09-17:17:55.735376] Installing zlib
==> [2020-11-09-17:17:55.781223] [TOOLCHAIN] MATCH AppleClang
==> [2020-11-09-17:17:55.781342] [TOOLCHAIN] MATCH Clang
==> [2020-11-09-17:17:55.781377] [TOOLCHAIN] MATCH Gcc
==> [2020-11-09-17:17:55.781419] [TOOLCHAINS] {'Gcc', 'AppleClang', 'Clang'}
==> [2020-11-09-17:17:55.781526] Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check [email protected] for further details]
==> [2020-11-09-17:17:55.789775] A dependency has updated CPATH, this may lead pkg-config to assume that the package is part of the system includes and omit it when invoked with '--cflags'.
==> [2020-11-09-17:17:55.820214] '/usr/bin/clang++-6.0' '-g' '-v' '/tmp/spack-implicit-link-info3vto7sia/main.c' '-o' '/tmp/spack-implicit-link-info3vto7sia/output'
==> [2020-11-09-17:17:55.883695] parsing implicit link info
==> [2020-11-09-17:17:55.883958] linker line:  "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/spack-implicit-link-info3vto7sia/output /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crt1.o /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crti.o /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/crtbegin.o -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../.. -L/usr/lib/llvm-6.0/bin/../lib -L/lib -L/usr/lib /tmp/main-671aa9.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/crtend.o /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crtn.o
==> [2020-11-09-17:17:55.884018] linkdir: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
==> [2020-11-09-17:17:55.884047] linkdir: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu
==> [2020-11-09-17:17:55.884074] linkdir: /lib/x86_64-linux-gnu
==> [2020-11-09-17:17:55.884096] linkdir: /lib/../lib64
==> [2020-11-09-17:17:55.884118] linkdir: /usr/lib/x86_64-linux-gnu
==> [2020-11-09-17:17:55.884139] linkdir: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../..
==> [2020-11-09-17:17:55.884160] linkdir: /usr/lib/llvm-6.0/bin/../lib
==> [2020-11-09-17:17:55.884181] linkdir: /lib
==> [2020-11-09-17:17:55.884202] linkdir: /usr/lib
==> [2020-11-09-17:17:55.884231] found raw link dirs: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0, /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu, /lib/x86_64-linux-gnu, /lib/../lib64, /usr/lib/x86_64-linux-gnu, /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../.., /usr/lib/llvm-6.0/bin/../lib, /lib, /usr/lib
==> [2020-11-09-17:17:55.884313] found link dirs: /usr/lib/gcc/x86_64-linux-gnu/7.5.0, /usr/lib/x86_64-linux-gnu, /lib/x86_64-linux-gnu, /lib64, /usr/lib, /usr/lib/llvm-6.0/lib, /lib
==> [2020-11-09-17:17:55.886703] Reading config file /home/spack1/spack/etc/spack/defaults/mirrors.yaml
==> [2020-11-09-17:17:55.888823] Reading config file /home/spack1/.spack/mirrors.yaml
==> [2020-11-09-17:17:55.894753] Using cached archive: /home/spack1/spack/var/spack/cache/_source-cache/archive/c3/c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1.tar.gz
==> [2020-11-09-17:17:55.896811] Staging archive: /tmp/spack1/spack-stage/spack-stage-zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/zlib-1.2.11.tar.gz
==> [2020-11-09-17:17:55.897339] '/bin/tar' '-xf' '/tmp/spack1/spack-stage/spack-stage-zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/zlib-1.2.11.tar.gz'
==> [2020-11-09-17:17:55.916710] Created stage in /tmp/spack1/spack-stage/spack-stage-zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5
==> [2020-11-09-17:17:55.917057] No patches needed for zlib
==> [2020-11-09-17:17:55.917110] zlib: Building zlib [Package]
==> [2020-11-09-17:17:55.922335] zlib: Executing phase: 'install'
==> [2020-11-09-17:17:55.923507] './configure' '--prefix=/home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5' '--static'
Building static library libz.a version 1.2.11 with /home/spack1/spack/lib/spack/env/clang/clang.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
==> [2020-11-09-17:17:56.363420] 'make' '-j2'
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o example.o test/example.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o adler32.o adler32.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o crc32.o crc32.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o deflate.o deflate.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o infback.o infback.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inffast.o inffast.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inflate.o inflate.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o inftrees.o inftrees.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o trees.o trees.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o zutil.o zutil.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o compress.o compress.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o uncompr.o uncompr.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzclose.o gzclose.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzlib.o gzlib.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzread.o gzread.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN  -c -o gzwrite.o gzwrite.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o minigzip.o test/minigzip.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o example64.o test/example.c
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -D_FILE_OFFSET_BITS=64 -c -o minigzip64.o test/minigzip.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o 
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example example.o -L. libz.a
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip minigzip.o -L. libz.a
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o example64 example64.o -L. libz.a
/home/spack1/spack/lib/spack/env/clang/clang -fPIC -O2 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -o minigzip64 minigzip64.o -L. libz.a
==> [2020-11-09-17:17:57.548490] 'make' '-j2' 'install'
rm -f /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/lib/libz.a
cp libz.a /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/lib
chmod 644 /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/lib/libz.a
rm -f /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/share/man/man3/zlib.3
cp zlib.3 /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/share/man/man3
chmod 644 /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/share/man/man3/zlib.3
rm -f /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/lib/pkgconfig/zlib.pc
cp zlib.pc /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/lib/pkgconfig
chmod 644 /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/lib/pkgconfig/zlib.pc
rm -f /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/include/zlib.h /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/include/zconf.h
cp zlib.h zconf.h /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/include
chmod 644 /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/include/zlib.h /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/include/zconf.h
==> [2020-11-09-17:17:57.571431] FileNotFoundError: [Errno 2] No such file or directory: '/home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/.spack/repos'
==> [2020-11-09-17:17:57.571658] Installing /tmp/spack1/spack-stage/spack-stage-zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/spack-build-out.txt to /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/.spack/spack-build-out.txt
==> [2020-11-09-17:17:57.571897] Installing /tmp/spack1/spack-stage/spack-stage-zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/spack-build-env.txt to /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/.spack/spack-build-env.txt
==> [2020-11-09-17:17:57.573536] Installing /home/spack1/spack/var/spack/repos/builtin/packages/zlib/w_patch.patch to /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/.spack/repos/builtin/packages/zlib
==> [2020-11-09-17:17:57.573716] Installing /home/spack1/spack/var/spack/repos/builtin/packages/zlib/package.py to /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5/.spack/repos/builtin/packages/zlib
==> [2020-11-09-17:17:57.574381] Reading config file /home/spack1/spack/etc/spack/defaults/modules.yaml
==> [2020-11-09-17:17:57.582663] Reading config file /home/spack1/spack/etc/spack/defaults/linux/modules.yaml
==> [2020-11-09-17:17:57.587735] 	WRITE: [email protected]%[email protected]+optimize+pic~shared arch=linux-ubuntu18.04-x86_64/ycir5rv [/home/spack1/spack/share/spack/modules/linux-ubuntu18.04-x86_64/zlib-1.2.11-clang-6.0.0-gfortran-ycir5rv]
==> [2020-11-09-17:17:57.662292] Writing manifest file: No manifest from binary
==> [2020-11-09-17:17:57.669584] zlib: Successfully installed zlib
  Fetch: 0.01s.  Build: 1.78s.  Total: 1.78s.
[+] /home/spack1/spack/opt/spack/linux-ubuntu18.04-x86_64/clang-6.0.0-gfortran/zlib-1.2.11-ycir5rv5toxncarcd3doaaxhjqegpgp5
==> [2020-11-09-17:17:57.687463] Flagging zlib as installed
==> [2020-11-09-17:17:57.687570] Downgrading to a read lock on zlib with timeout 1e-09

Add note about `spack help --spec` to Basics tutorial

The spec syntax we cover can be confusing for new users, and it's good to know that we have a command just to describe this syntax. We should end the tutorial with a note about spack help --spec.

Use %compiler@major instead of major.minor.patch

spack install zlib %gcc@10 looks better than spack install zlib %[email protected] and is possible in spack 0.20, so let's use that where possible

Environments section missing reconcretize

After the spack remove trilinos command, we need to reconcretize to show that trilinos and its unique dependencies are no longer installed in that environment, before we go to show that they are still present in the other environment.

Doc tests: switch to GitHub Actions

The main Spack repo has ported all of its tests to GitHub Actions. We should do the same for our tutorial docs. We should also add a badge for the test to make sure they remain building properly.

blacklist C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, LIBRARY_PATH example update

after blacklist the variables and regenerate module command and you run module show gcc-8.3.0-gcc-7.5.0-6hbkzol you will see the output of module show has these paths. see https://spack-tutorial.readthedocs.io/en/latest/tutorial_modules.html#filter-unwanted-modifications-to-the-environment

$ module show gcc-8.3.0-gcc-7.5.0-6hbkzol
--------------------------------------------------------------------------------------------------------------------------------------------
   /home/spack/spack/share/spack/modules/linux-ubuntu18.04-x86_64/gcc-8.3.0-gcc-7.5.0-6hbkzol:
--------------------------------------------------------------------------------------------------------------------------------------------
whatis("The GNU Compiler Collection includes front ends for C, C++, Objective-C, Fortran, Ada, and Go, as well as libraries for these languages. ")
prepend_path("PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin")
prepend_path("MANPATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/share/man")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib")
prepend_path("LD_LIBRARY_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/lib64")
prepend_path("INCLUDE","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/include")
prepend_path("CMAKE_PREFIX_PATH","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/")
setenv("CC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gcc")
setenv("CXX","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/g++")
setenv("FC","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
setenv("F77","/home/spack/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/gcc-8.3.0-6hbkzolzshktgws6mz3f4s23v6sbkgnl/bin/gfortran")
help([[The GNU Compiler Collection includes front ends for C, C++, Objective-C,
Fortran, Ada, and Go, as well as libraries for these languages.
]])

Lmod hierarchical view under modern Cray environments

I'm looking for either an example or a full blown tutorial showing how to use the Cray PrgEnv to compile products and then expose the products through a hierarchical view.

I wish for the Cray environment to be left untouched. The user should only have to load some modules with a custom name to navigate the hierarchical view. the module exposing the hierarchy, should only change the module path, and not modify the environment in any other way unless exceptional.

i.e:

Hypre was compiled with the PrgEnv-[cray|amd|gcc], this will produce 3 products which I which to expose, never at the same time (mutual exclusion) and only when a particular module is loaded.

$ ml products-cce
$ ml load hypre
$ module li
------------------------- /cce/15.0.0/Core --------------------------
   hypre/2.25.0

-------------------------------------- /Core ---------------------------------------
   products-cce/xxx    products-gcc/xxx     products-rocmcc/xxx    
$ ml products-gcc
<swap modules and automatically reload hypre (under the gcc Core, typical lmod stuff>

buildcache/mirror tutorial

Hello,

It would be helpful for me if there is a tutorial on buildcache/mirror to accelerate software stack build in spack environment. Looking at the current docs at https://spack.readthedocs.io/en/latest/binary_caches.html and https://spack.readthedocs.io/en/latest/mirrors.html it is not clear how the workflow looks like when building software in one place and then having it installed in a new prefix.

Auto-generate config tutorial output

We should create scripts to generate the output of all commands for the config tutorial section.

Also, because we now disable reuse in this section, the exact binary needed for cmake does not exist in the cache. We should fix this during cache creation and remove the --reuse flags I had to add.

Ensure trilinos installed before environments section

During the RADIUSS tutorial, @tldahlgren was forced to ad-lib more than usual because an environment with trilinos was taking a long time to install.

Trilinos is also installed during the basics section in two different configurations.

When we next redo these sections, we should ensure that the trilinos that we install in the basics section (and don't uninstall, we uninstall one of them) is the same one used in the environments section.

Print handouts and IPs

Replace HDF5 with a quicker-to-build package

In the developer workflows tutorial, we build hdf5~mpi, but it still takes a while. Replace it with something that:

builds quickly
has at least one installed dependency already in the build cache
doesn't add any new build cache requirements