View source for Admin Guide Easybuild

[[Category:HPC-Admin]]
[[Category:HPC.NRW-Best-Practices]]


== Introduction ==

EasyBuild (EB) is a Python framework to automate the process of software installations and the creation of environment modules in an HPC environment. Most installations are hereby directly compiled from source to create architecture optimized and performant builds. The following article shortly describes the first steps of how to set up EasyBuild. The main focus, however, lies on the description of a sustainable directory structure and configuration helping to keep an overview of installed software while beeing able to provide frequent updates of new versions. A more complete guide on how to set up EasyBuild can be found at the official [https://easybuild.readthedocs.io/en/latest/Installation.html documentation].

== Setup ==

=== Assumptions ===

* /sw/easybuild/ is the main folder and accessible from all compute nodes
* /sw/easybuild/default will be the default path for your EB installation
* /sw/easybuild/stacks/skylake/2020a is a path for a specific CPU arch and software stack
* CentOS 7 (otherwise the list of dependencies for EB can vary)
* LMOD &gt;= v8.3.4 (the current versions from the OpenHPC repos are '''not''' sufficient)
* Python 3 (You can also use Python 2, but at this point you really shouldn’t)

=== Initial installation and configuration ===

<source lang="bash"># INSTALL SYSTEM DEPS
yum install epel-release
yum install python3 git gcc gcc-c++ libibverbs-devel patch openssl-devel 

# DOWNLOAD THE BOOTSTRAP EB SCRIPT AND BOOTSTRAP EB
curl -O https://raw.githubusercontent.com/easybuilders/easybuild-framework/develop/easybuild/scripts/bootstrap_eb.py
python3 bootstrap_eb.py /sw/easybuild/default

# UPDATE $MODULEPATH, AND LOAD THE EasyBuild MODULE
module use /sw/easybuild/default/modules/all # add this to your bashrc
module load EasyBuild

# CREATE DIRS AND A FIRST CONFIGFILE FOR YOUR CPU ARCHITECTURE
mkdir -p /sw/easybuild/configfiles # directory for your configs
mkdir -p /sw/easybuild/sources
mkdir -p /sw/easybuild/stacks/skylake/2020a

vim /sw/easybuild/configfiles/skylake-2020a.cf # Enter the values from the example below

# CREATE AN ALIAS IN YOUR BASHRC FOR EVERY SPECIFIC CONFIGFILE, E.G
alias ebsky-2020a='eb --configfile=/sw/easybuild/configfiles/skylake-2020a.cfg'

# YOU CAN LIST ALL AVAILABLE CONFIG OPTIONS WITH
eb -a
# OR YOU REDIRECT THE OUTPUT OF eb --confighelp TO GET AN ANNOTADED CONFIGFILE
eb --confighelp >> myconfig.cfg

# TO SHOW YOUR CURRENT EB CONFIGURATION USE
eb --show-config
ebsky-2020a --show-config

#NOTE: If you install software with EB and just use 'eb' you will install it into your default folder!</source>
=== Example usage of EasyBuild ===

<source lang="bash"># Basic usage
eb -h # short help
eb -H # complete list of all options

# Dry-run installation of the foss-2020a toolchain
ebsky-2020a foss-2020a.eb -r -D # -D == dry-run / -r == install deps
ebsky-2020a foss-2020a.eb -r -M # -M == show only missing dependencies

# Real installation with all deps
ebsky-2020a foss-2020a.eb -r</source>
== Folder structure ==

Inside of <code>/sw/easybuild/stacks/</code> , a folder for every supported CPU architecture is created (broadwell, skylake, etc.). These contain subfolders of the specific EB toolchain release ''dates'' (i.e. 2018b, 2019a etc.) where one finds the modules and installed software. In addition the folders <code>configfiles</code>, <code>hooks</code>, <code>sources</code> and <code>easybuild_repo</code> can be found here. The easybuild_repo folder is synced via a local git repository and contains a folder <code>archive</code> and <code>custom_easyconfigs</code>.

<pre>/sw
└── easybuild
    ├── configfiles
    ├── custom_easyconfigs
    ├── hooks   # --&gt; Python script which can alter the build process
    └── sources # --&gt; place to store all downloaded source-files; prevents downloading  a source multiple times
    ├── stacks
        ├── broadwell
        ├── skylake
            └── 2018b
            └── 2019a
                ├── modules
                └── software
            ├── ...
        ├── ...</pre>
== Further Configuration of Easybuild ==

=== Config files ===

For every toolchain release there exists a ''base configuration'' file inside the folder <code>configfiles</code>. Personal settings of different admins can be made in local config.cfg files in <code>~/.config/easybuild/</code>. Configfiles can be used via

<pre>eb --configfile=/PATH/TO/CONFIG.CFG</pre>
Files which are listed within <code>--configfile</code> are treated first.

Example of a basic ''skylake-2020a.cfg'':

<pre>[config]
prefix = /sw/easybuild/stacks/skylake/2020a
module-naming-scheme = HierarchicalMNS
sourcepath=/sw/easybuild/sources/
robot-paths=/sw/easybuild/custom_easyconfigs:%(DEFAULT_ROBOT_PATHS)s
group-writable-installdir=true</pre>
Using different config files for different architectures and toolchain releases helps to handle heterogeneous systems and to keep an overview of installed software.

Create a bash alias for every config file for ease of use.

<source lang="bash">alias ebsky-2020a='eb --configfile=/sw/easybuild/configfiles/skylake-2020a.cfg'</source>
=== Archive and self written Easyconfig files ===

After every successful build, the used ''Easyconfig'' file will be be archived in <code>stacks/$ARCH/$RELEASE/easybuild_repo/</code>. Self-written Easyconfig files can be stored in the folder <code>custom_easyconfigs</code> and will be considered when searching for software. A guide on how to write your own easyconfig file can be found [https://easybuild.readthedocs.io/en/latest/Writing_easyconfig_files.html here].

=== SLURM integration ===

SLURM can be used as a [https://easybuild.readthedocs.io/en/latest/Submitting_jobs.html job] backend to compile multiple programs at the same time. Dependencies will be resolved automatically and the order of running jobs is taken care of. Use the flags <code>--job</code> when you run eb. You can add the following to your config files. Modify appropriately.

<source lang="bash">job-backend=Slurm
job-cores=<NCORES>
job-max-jobs=<NJOBS></source>
=== Hooks ===

[https://easybuild.readthedocs.io/en/latest/Hooks.html Hooks] are small Python scripts which can directly influence the build process. These can be e.g. used to make site-specific adjustment to Easyconfig file without the need to create a completely new one each time.

An example hook script is given below, which adds some slurm specific configuration, adds flags to every OpenMPI build and points to an license file for intel software installations:

<source lang="python">import sys, os
from easybuild.tools.build_log import print_msg
from distutils.version import LooseVersion

def start_hook(*args, **opts):
    if "--job" in sys.argv:
        # Check if env var was set
        slurm_partition = os.getenv("SBATCH_PARTITION")
        if slurm_partition is not None:
            print_msg("[start-hook] SLURM_PARTITION ENV VAR set: %s."%slurm_partition)
        else:
            slurm_partition = "normal"

        slurm_mem_per_node = os.getenv("SBATCH_MEM_PER_NODE")
        if slurm_mem_per_node is not None:
            print_msg("[start-hook] SLURM_MEM_PER_NODE ENV VAR set: %s."%slurm_mem_per_node)
        else:
            slurm_mem_per_node = "36G"


        import easybuild.tools.job.slurm as slurm
        class slurm_job(slurm.SlurmJob):
            def __init__(self, *args, **opts):
                super(slurm_job, self).__init__(*args, **opts)
                self.job_specs['partition'] = slurm_partition
                self.job_specs['mem'] = slurm_mem_per_node
                self.job_specs['time'] = '12:00:00'
        slurm.SlurmJob = slurm_job
        print_msg("[start-hook] using partition << %s >> "%slurm_partition)

def pre_prepare_hook(self, *args, **kwargs):
    # SET PATH TO INTEL LICENSE FILE
    if self.name in ["icc", "ifort", "itac", "VTune"]:
        self.cfg['license_file'] = "/sw/licenses/USE_SERVER.lic"
        self.log.info("[pre-prepare hook] Setting path to license file: %s" % self.cfg['license_file'] )
        print_msg("Intel license file: %s" % self.cfg['license_file'])

def pre_configure_hook(self, *args, **kwargs):
    if self.name == 'OpenMPI':
        extra_opts = ""
        # Enable slurm and pmi support
        extra_opts += "--with-slurm --with-pmi"

        # Now add the options
        self.log.info("[pre-configure hook] Adding %s" % extra_opts)
        self.cfg.update('configopts', extra_opts)</source>
== Toolchains ==

About every 6 months EasyBuild releases new ''toolchains'' which combine a set of specific modules for compilers, MPI and numerical libraries (cf. [https://easybuild.readthedocs.io/en/latest/Common-toolchains.html common toolchains]). The two most common are: 
* intel + intelcuda –&gt; icc/ifort, iMPI, MKL 
* foss + fosscuda –&gt; gcc, OpenMPI, OpenBLAS, LAPACK, ScaLAPACK, FFTW

== Modules ==

Modules in this example are automatically created using the [https://easybuilders.github.io/easybuild/files/hust14_paper.pdf hierarchical module naming scheme]. There are other options (the default being the ''EasyBuildModuleNamingScheme'') available as well as the option to create your own site-specific module naming scheme. See this [https://github.com/easybuilders/easybuild/wiki/Using-a-custom-module-naming-scheme link].

=== Meta-modules ===

A meta-module in this case is a module which makes a different module path available to the user while taking care of the correct path for the current architecture. An example of such a module is given below. These modules can e.g. reside in folder /sw/easybuild/meta-module/palma. The ''software'' in this example case is called palma and the versions correspond the EB toolchain releases, i.e. 2019b.lua, 2020a.lua etc. The <code>CPU_ARCH</code> environment variable which is used here, is exported in a <code>modules.sh</code> script within <code>/etc/profile.d</code> via <code>CPU_ARCH=$(cat /sys/devices/cpu/caps/pmu_name)</code>.

<source lang="lua">help([==[

Description
===========
This is a meta module giving you access to the PALMA 2020a software stack. Software on PALMA is build using the
EasyBuild Python Framework.

Supported CPU Architectures: skylake

More information
================
 - PALMA: https://confluence.uni-muenster.de/display/HPC
 - EasyBuild: https://easybuild.readthedocs.io
]==])

whatis([==[This is a meta module giving you access to the PALMA 2020a software stack. Software on PALMA is build using
 the EasyBuild Python Framework.]==])
whatis([==[PALMA: https://confluence.uni-muenster.de/display/HPC]==])
whatis([==[EasyBuild: https://easybuild.readthedocs.io]==])


local version = "2020a"
local root = "/sw/easybuild/stacks"
local cpu_arch = os.getenv("CPU_ARCH")
local suffix = "/modules/all/Core"
local hostname = subprocess("hostname -s")

-- LmodMessage("cpu_arch = ", cpu_arch)
-- LmodMessage("hostname = ", hostname)

local cpu_arch = os.getenv("CPU_ARCH")
if (string.find(hostname, "^r13n[01-12].*")) then
    cpu_arch = cpu_arch .. "-IB"
end

-- ONLY SKYLAKE HAS THIS TOOLCHAIN AT THE MOMENT
if (cpu_arch ~= "skylake") then
    -- THROWS AN ERROR MESSAGE AND EXIT
    LmodError(version, "IS NOT YET AVAILABLE ON THE ", cpu_arch,  "ARCHITECTURE.")
end

conflict("palma")
prepend_path("MODULEPATH", pathJoin(root, cpu_arch, version, suffix))
-- add_property("lmod",)</source>
=== Using the correct modules path ===

We created a <code>modules.sh</code> file inside of <code>/etc/profile.d/</code> on the compute and login nodes instead of symlinking directly to <code>lmod/init/profile</code>. This deals with some issues Lmod encounters with slurm. In case of an interactive slurm session, it also has to reset <code>$MODULEPATH</code> before updating/reloading the currently loaded modules. Otherwise updating the modules will not unload the old path since we might be on a node with a different architecture than on the login node.

<source lang="bash">#!/bin/sh
 
# NOTE: In a slurm batch job (non-interactive, non-login) this file is not sourced.
#       Only the file where $BASH_ENV points to is sourced in this case.
 
# CPU ARCH ENV VARIABLE FOR META MODULES
if [ -f "/sys/devices/cpu/caps/pmu_name" ];then
    export CPU_ARCH=`cat /sys/devices/cpu/caps/pmu_name`
fi
 
# Special treatment if we are in an interactive slurm session
if [ ! -z "$SLURM_NODELIST" ];then
  echo "YOU ARE NOW IN AN INTERACTIVE SLURM JOB"
  # Reset module path
  export MODULEPATH=/sw/easybuild/meta-modules/
  # Update the currently loaded modules to account for architecture specific paths
  module update
  # re-source the bash init to gain module autocomplete functionality in an interactive session
  . /opt/lmod/lmod/init/bash >/dev/null
  return
fi
 
# Enable bash module support
. /opt/lmod/lmod/init/profile >/dev/null</source>
== Improvements ==

* There are still a lot of (low-level) modules for system libraries exposed to the users –&gt; A solution can be using [https://easybuild.readthedocs.io/en/latest/Manipulating_dependencies.html#installing-dependencies-as-hidden-modules-using-hide-deps hidden modules]
* Instead of using meta-modules, one can use bind mounts on every node, such that the architecture specific folder points to a generall easybuild install folder (symlinks are know to cause issues)
* An automated build process for all architectures
* ‘module spider’ output pointing to toolchain instead of compiler plus MPI library