Build a uenv

CSCS User Environments (uenv) provide scientific applications, libraries and tools on Alps.

User environments are SquashFS files that store a compressed directory tree containing all of the software, tools and other information like modules and libraries required to provide a rich environment.

Each environment contains a software stack, comprised of compilers, libraries, tools and scientific applications, built using Spack.

To learn more about uenv and how to run them, please refer to the official documentation by CSCS.

This recipe will focus on building a custom user environment either locally or via CSCS CI/CD. Either way, a user environment is built from a set of recipes (YAML files), that define the compilers and software packages to be installed, along with configuration of modules and environment scripts the environment will provide to users.

In this recipe, we will install a Python package (ppafm) that provides a library and scripts to perform scanning probe microscopy simulations. The full recipe can be found here.

Locally

The main piece of software required to build a user environment is Stackinator.

Getting Stackinator

Clone the repo

git clone https://github.com/eth-cscs/stackinator.git

Checkout to a released versions since the main branch includes features for Spack v1.0, and may break older recipes.
Terminal window
```
git checkout v4.0
```
Run the bootstrap and make the binary available in your PATH
Terminal window
```
./bootstrap.sh
export PATH="<stackinator-install-path>/bin:$PATH"
```

Prepare the build environment

Move to another folder (e.g., one folder above from where you cloned Stackinator’s repo) and clone the repository with the cluster configurations
Terminal window
```
git clone https://github.com/eth-cscs/alps-cluster-config
```
These are recipes (YAML Spack files) that define the toolchains to compile and install the software described in our recipe.
If you never built a uenv, it’s a good idea to have a build cache to speed up subsequent builds. First, create an empty folder (in your $SCRATCH directory)
Terminal window
```
mkdir -p $SCRATCH/uenv-cache
```

Use a Spack installation to create generate GPG keys that are used for signing and verifying the packages

# Create a  .keys directory accessible only to you
mkdir $SCRATCH/.keys
chmod 700 $SCRATCH/.keys

# Generate the key
spack gpg create <your-name> <your-email>
spack gpg export --secret $SCRATCH/.keys/spack-push-key.gpg
chmod 600 $SCRATCH/.keys/spack-push-key.gpg

Create a file to configure the build cache
cache-config.yaml
```
root: $SCRATCH/uenv-cache
key:  $SCRATCH/.keys/spack-push-key.gpg
```
Note the path of this config file as it will be needed by Stackinator when building the uenv.

Build the uenv

In a folder of your choice, clone the empa-spack repository that contains the recipe for the ppafm package
Terminal window
```
git clone https://github.com/empa-scientific-it/empa-spack.git
```
Request an allocation on a cluster’s computing node to avoid hogging the resources on the login nodes
Terminal window
```
srun -A <account> -n1 -t180 --pty bash
```

Run stack-config (provided by Stackinator) to prepare the build directory. Adjust all the paths according to the previous steps

stack-config --recipe /path/to/empa-spack/uenv/ppafm/v0.3.2/daint \
             --build /dev/shm/$USER/ppafm \
             --system /path/to/alps-cluster-config/daint \
             --cache /path/to/cache-config.yaml

Perform the Build
Terminal window
```
cd /dev/shm/$USER/ppafm
env --ignore-environment PATH=/usr/bin:/bin:`pwd`/spack/bin make modules store.squashfs -j32
```
The call to make is prepended by env to unset all environment variables and make the build reproducible.

Installing the software

The make command generates two versions of the software stack in the build path:

A store directory, the full Spack environment with everthing that’s been built and installed
A store.squashfs file, a compressed filesystem image that can be mounted on a cluster node

The simplest method to install the software stack is to copy the store directory to the desired location (preferably out of the $SCRATCH folder). For the recommended way of deploying uenvs on Alps, please refer to the deploying uenv documentation by CSCS.

The default installation directory is specified in the config.yaml file of our recipe. In this example, we’re using the default /user-environment directory, which should be used for deployment on Alps.

name: ppafm
spack:
    commit: releases/v0.23
    repo: https://github.com/spack/spack.git
store: /user-environment
description: Library and scripts to perform scanning probe microscopy simulations based on a CP2K calculation

Via the CSCS build service

CSCS provides a build service for uenvs that takes a uenv recipe as input and builds it using the same pipeline used for the officially supported uenvs.

uenv build <recipe> <label>

The label has the form name/version@system%uarch, where name is the uenv name, version is a version string, system is the CSCS cluster to build on (e.g., daint), and uarch is the micro-architecture (e.g., gh200). For example, to build the ppafm uenv from above on the GH200 nodes of Daint:

uenv build $SCRATCH/recipes/ppafm ppafm/v0.3.2@daint%gh200

The command returns a URL to a status page where the build progress can be followed. After a successful build, the uenv can be pulled using the name and the numeric tag from the status page:

uenv image pull service::ppafm/v0.3.2:<tag>

All uenvs built by the build service are pushed to the service namespace, where they can be accessed by all users logged in to CSCS. If, for whatever reason, a uenv cannot be made publicly available, do not use the build service; build it locally with Stackinator instead, as described above.

For more details, please refer to the official documentation by CSCS.