See the latest version of our docs for a quick overview of this platform for advanced AI/ML workflows in digital twin applications.

If you are a developer, please refer to the developers installation guide.

Requirements:

• Linux or macOS environment. Windows was never tested.

Depending on your environment, there are different ways to select a specific python version.

If you are working on a laptop or on a simple on-prem setup, you could consider using pyenv. See the installation instructions. If you are using pyenv, make sure to read this.

In HPC systems it is more popular to load dependencies using Environment Modules or Lmod. If you don't know what modules to load, contact the system administrator to learn how to select the proper modules.

Commands to execute every time before installing or activating the python virtual environment for PyTorch:

• Juelich Supercomputer (JSC):

  ml --force purge
  ml Stages/2024 GCC OpenMPI CUDA/12 cuDNN MPI-settings/CUDA
  ml Python CMake HDF5 PnetCDF libaio mpi4py

• Vega supercomputer:

  ml --force purge
  ml Python CMake/3.24.3-GCCcore-11.3.0 mpi4py OpenMPI CUDA/11.7
  ml GCCcore/11.3.0 NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0 cuDNN

Commands to execute every time before installing or activating the python virtual environment for TensorFlow:

• Juelich Supercomputer (JSC):

  ml --force purge
  ml Stages/2024 GCC/12.3.0 OpenMPI CUDA/12 MPI-settings/CUDA
  ml Python/3.11 HDF5 PnetCDF libaio mpi4py CMake cuDNN/8.9.5.29-CUDA-12

• Vega supercomputer:

  ml --force purge
  ml Python CMake/3.24.3-GCCcore-11.3.0 mpi4py OpenMPI CUDA/11.7
  ml GCCcore/11.3.0 NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0 cuDNN

Install itwinai and its dependencies using the following command, and follow the instructions:

# First, load the required environment modules, if on an HPC

# Second, create a python virtual environment and activate it
$ python -m venv ENV_NAME
$ source ENV_NAME/bin/activate

# Install itwinai inside the environment
(ENV_NAME) $ export ML_FRAMEWORK="pytorch" # or "tensorflow"
(ENV_NAME) $ curl -fsSL https://github.com/interTwin-eu/itwinai/raw/main/env-files/itwinai-installer.sh | bash

The ML_FRAMEWORK environment variable controls whether you are installing itwinai for PyTorch or TensorFlow.

If you are contributing to this repository, please continue below for more advanced instructions.

git clone [email protected]:interTwin-eu/itwinai.git

You can create the Python virtual environments using our predefined Makefile targets.

Makefile targets for environment installation:

• Juelich Supercomputer (JSC): torch-gpu-jsc
• Vega supercomputer: torch-env-vega
• In any other cases, when CUDA is available: torch-env
• In any other cases, when CUDA NOT is available (CPU-only installation): torch-env-cpu

For instance, on a laptop with a CUDA-compatible GPU you can use:

make torch-env 

When not on an HPC system, you can activate the python environment directly with:

source .venv-pytorch/bin/activate

Otherwise, if you are on an HPC system, please refer to this section explaining how to load the required environment modules before the python environment.

Makefile targets for environment installation:

• Juelich Supercomputer (JSC): tf-gpu-jsc
• Vega supercomputer: tf-env-vega
• In any other case, when CUDA is available: tensorflow-env
• In any other case, when CUDA NOT is available (CPU-only installation): tensorflow-env-cpu

For instance, on a laptop with a CUDA-compatible GPU you can use:

make tensorflow-env

When not on an HPC system, you can activate the python environment directly with:

source .venv-tf/bin/activate

Otherwise, if you are on an HPC system, please refer to this section explaining how to load the required environment modules before the python environment.

Usually, HPC systems organize their software in modules which need to be imported by the users every time they open a new shell, before activating a Python virtual environment.

Below you can find some examples on how to load the correct environment modules on the HPC systems we are currently working with.

Commands to be executed before activating the python virtual environment:

• Juelich Supercomputer (JSC):

  ml --force purge
  ml Stages/2024 GCC OpenMPI CUDA/12 cuDNN MPI-settings/CUDA
  ml Python CMake HDF5 PnetCDF libaio mpi4py

• Vega supercomputer:

  ml --force purge
  ml Python CMake/3.24.3-GCCcore-11.3.0 mpi4py OpenMPI CUDA/11.7
  ml GCCcore/11.3.0 NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0 cuDNN

• When not on an HPC: do nothing.

For instance, on JSC you can activate the PyTorch virtual environment in this way:

# Load environment modules
ml --force purge
ml Stages/2024 GCC OpenMPI CUDA/12 cuDNN MPI-settings/CUDA
ml Python CMake HDF5 PnetCDF libaio mpi4py

# Activate virtual env
source envAI_hdfml/bin/activate

Commands to be executed before activating the python virtual environment:

• Juelich Supercomputer (JSC):

  ml --force purge
  ml Stages/2024 GCC/12.3.0 OpenMPI CUDA/12 MPI-settings/CUDA
  ml Python/3.11 HDF5 PnetCDF libaio mpi4py CMake cuDNN/8.9.5.29-CUDA-12

• Vega supercomputer:

  ml --force purge
  ml Python CMake/3.24.3-GCCcore-11.3.0 mpi4py OpenMPI CUDA/11.7
  ml GCCcore/11.3.0 NCCL/2.12.12-GCCcore-11.3.0-CUDA-11.7.0 cuDNN

• When not on an HPC: do nothing.

For instance, on JSC you can activate the TensorFlow virtual environment in this way:

# Load environment modules
ml --force purge
ml Stages/2024 GCC/12.3.0 OpenMPI CUDA/12 MPI-settings/CUDA
ml Python/3.11 HDF5 PnetCDF libaio mpi4py CMake cuDNN/8.9.5.29-CUDA-12

# Activate virtual env
source envAItf_hdfml/bin/activate

Do this only if you are a developer wanting to test your code with pytest.

First, you need to create virtual environments both for torch and tensorflow, following the instructions above, depending on the system that you are using (e.g., JSC).

To select the name of the torch and tf environments in which the tests will be executed you can set the following environment variables. If these env variables are not set, the testing suite will assume that the PyTorch environment is under .venv-pytorch and the TensorFlow environment is under .venv-tf.

export TORCH_ENV="my_torch_env"
export TF_ENV="my_tf_env"

Functional tests (marked with pytest.mark.functional) will be executed under /tmp/pytest location to guarantee isolation among tests.

To run functional tests use:

pytest -v tests/ -m "functional"

We provide some Makefile targets to run the whole test suite including unit, integration, and functional tests. Choose the right target depending on the system that you are using:

Makefile targets:

• Juelich Supercomputer (JSC): test-jsc
• In any other case: test

For instance, to run the test suite on your laptop user:

make test