bdusell / singularity-tutorial Goto Github PK

I tried using the version-1.def file but ran into problem with the apt-get install step complaining that it couldn't find python3-pip.

After digging around I noticed that the /etc/apt/sources.list file in the ubuntu:18.04 image had only 1 line which meant that it wasn't including the full set of sources ... and couldn't find python3-pip.

To fix this I took the sources.list from here: https://gist.githubusercontent.com/h0bbel/4b28ede18d65c3527b11b12fa36aa8d1/raw/314419c944ce401039c7def964a3e06324db1128/sources.list

copied it to the local folder and updated the version-1.def file as:

Bootstrap: library
From: ubuntu:18.04

%files
    ## Overwrite sources.list to include full ubuntu sources
    ## Source file from: https://gist.githubusercontent.com/h0bbel/4b28ede18d65c3527b11b12fa36aa8d1/raw/314419c944ce401039c7def964a3e06324db1128/sources.list
    sources.list /etc/apt/sources.list

%post
    # Downloads the latest package lists (important).
    apt-get update -y
    # Runs apt-get while ensuring that there are no user prompts that would
    # cause the build process to hang.
    # python3-tk is required by matplotlib.
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        python3 \
        python3-tk \
        python3-pip
    # Reduce the size of the image by deleting the package lists we downloaded,
    # which are useless now.
    rm -rf /var/lib/apt/lists/*
    # Install Python modules.
    pip3 install torch numpy matplotlib

    raise RequiredDependencyException(msg)
__main__.RequiredDependencyException:

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html



----------------------------------------

Command "/usr/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-build-p6gvu23t/pillow/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-sfd7t3jb-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-p6gvu23t/pillow/
FATAL: While performing build: while running engine: exit status 1
dave@dave:~/singularity-tutorial/examples/language-model$ sudo apt install libjpeg8-dev zlib1g-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libjpeg8-dev is already the newest version (8c-2ubuntu8).
libjpeg8-dev set to manually installed.
zlib1g-dev is already the newest version (1:1.2.11.dfsg-2ubuntu1.3).

After installation, I was trying to run the following command:
cd examples/xor
sudo singularity build version-1.sif version-1.def

But I got: While performing build: conveyor failed to get: get did not succeed: 500
architecture not specified

Solution: Problem solved. I guess I was using an old version singularity 3.2. After I updated to 3.9, problem solved.

hey @bdusell ! This post is incredibly detailed and useful! I'm worried that it will get lost in the gazillions of repos on GitHub, and I want to put it somewhere that others can find/search easily in the future. Would you be open to posting it on Ask Cyberinfrastructure?

http://ask.cyberinfrastructure.org

It's akin to Stack Overflow, but for HPC! This means that you are allowed to ask and answer your own questions :) You could easily write a generic question (e.g., "How do I build a container on HPC?") and then give a brief summary that links here.