High Performance Computing
This document will aid an HPC Administrator on building an environment which end-users will use when compiling and running MOOSE based applications. If you do not have administrative rights, you will not be able to complete these instructions! Please forward these instructions to your HPC Administrator.
Because this document assumes the reader is an administrator, most of the content herein are suggestions, and should not be blindly followed with 'copy/paste/enter' behavior (your HPC cluster is no doubt different than our cluster these instructions were based on).
If you are an end-user trying to follow these instructions, please seek help from one of your HPC Administrators. However, if you are the sole owner of the machine for which you currently operate on (a single workstation), you should follow one of our Manual Install (GCC) instructions instead.
Some sort of environmental module management software. Such as Environment Modules.
A working MPI wrapper (MPICH/OpenMPI/MVAPICH) which wraps to a C++11 compliant compiler.
Knowledge on Access Control Lists (ACLs) or other means to safeguard a directory from further tampering by others.
Please use a single solitary terminal session throughout and to the completion of these instructions.
Begin by creating an area for which to build:
export STACK_SRC=`mktemp -d /tmp/stack_src_temp.XXXXXX` cd $STACK_SRC
Set your umask
Some systems have a secure umask set. We need to adjust our umask so that when you write a file (
make install), it is readable by everyone:
Choose a base path
Export a base path variable which will be the home location for the compiler stack. All files related to libraries necessary to build MOOSE, will be stored in this location (choose carefully, as this location should be accessible from all nodes on your cluster):
Create and chown $PACKGES_DIR
History teaches us, that implicitly trusting scripts we download off the internet with root access, is a very bad idea. So let us create and chown the $PACKAGES_DIR directory before we install anything. That way, the things we do install can be done so without invoking
sudo mkdir -p $PACKAGES_DIR sudo chown -R <your user id> $PACKAGES_DIR
Verify that your umask settings are indeed set to 0022 before continuing:
$> umask 0022
Set Up Modules
Even if you are not using Modules, the following provides information on what environment variables are needed for MOOSE developement.
Create a MOOSE module:
mkdir -p $PACKAGES_DIR/modulefiles vi $PACKAGES_DIR/modulefiles/moose-dev-gcc
Add the following content to that file:
#%Module1.0##################################################################### ## ## MOOSE module ## set base_path INSERT PACKAGES_DIR HERE GCC MPI PATHS prepend-path PATH /GCC and MPI /bin prepend-path LD_LIBRARY_PATH /GCC and MPI /lib setenv CC mpicc setenv CXX mpicxx setenv F90 mpif90 setenv F77 mpif77 setenv FC mpif90 setenv PETSC_DIR $base_path/petsc/petsc-3.8.3/gcc-opt # Optional if miniconda is installed prepend-path PATH $base_path/miniconda/bin
INSERT PACKAGES_DIR HERE with whatever you had set for $PACKAGES_DIR (do not literally enter: $PACKAGES_DIR. As an example, if you left packages_dir as: /opt/moose, then that is what you would enter)
GCC and MPI paths with any additional information needed to make GCC/MPI work on your cluster.
To make the module available in your terminal session, export the following:
The above command should be added to the list of other global profiles (perhaps in /etc/profiles.d). That way, the above is performed as the user logs into the machine.
With the modulefile in place and the MODULEPATH variable set, see if our module is available for loading:
module load moose-dev-gcc
Verify that this module loads properly by attempting to echo $PETSC_DIR:
echo $PETSC_DIR /opt/moose/petsc/petsc-3.8.3/gcc-opt
While we are at it, verify that your MPI wrapper works by running a few commands (your results will vary, but they should return something):
which mpicc /opt/moose/mpich-3.2/gcc-7.3.0/bin/mpicc mpicc -show gcc -I/opt/moose/mpich-3.2/gcc-7.3.0/include -L/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,-rpath -Wl,/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,--enable-new-dtags -lmpi which gcc /opt/moose/gcc-7.3.0/bin/gcc
Leave this module loaded for the remainder of the instructions (PETSc requirements).
Download PETSc 3.8.3
cd $STACK_SRC curl -L -O http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.8.3.tar.gz tar -xf petsc-3.8.3.tar.gz -C .
Now we configure, build, and install it
cd $STACK_SRC/petsc-3.8.3 ./configure \ --prefix=$PACKAGES_DIR/petsc-3.8.3 \ --download-hypre=1 \ --with-ssl=0 \ --with-debugging=no \ --with-pic=1 \ --with-shared-libraries=1 \ --with-cc=mpicc \ --with-cxx=mpicxx \ --with-fc=mpif90 \ --download-fblaslapack=1 \ --download-metis=1 \ --download-parmetis=1 \ --download-superlu_dist=1 \ --download-mumps=1 \ --download-scalapack=1 \ --CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 \ --CFLAGS='-fPIC -fopenmp' \ --CXXFLAGS='-fPIC -fopenmp' \ --FFLAGS='-fPIC -fopenmp' \ --FCFLAGS='-fPIC -fopenmp' \ --F90FLAGS='-fPIC -fopenmp' \ --F77FLAGS='-fPIC -fopenmp' \ PETSC_DIR=`pwd`
Once configure is done, we build PETSc
make PETSC_DIR=$STACK_SRC/petsc-3.8.3 PETSC_ARCH=arch-linux2-c-opt all
Everything good so far? PETSc should be asking to run more make commands
make PETSC_DIR=$STACK_SRC/petsc-3.8.3 PETSC_ARCH=arch-linux2-c-opt install
And now after the install, we can run some built-in tests
make PETSC_DIR=$PACKAGES_DIR/petsc-3.8.3 PETSC_ARCH="" test
Running the tests should produce some output like the following:
[moose@centos-7 petsc-3.8.3]$ make PETSC_DIR=$PACKAGES_DIR/petsc-3.8.3 PETSC_ARCH="" test Running test examples to verify correct installation Using PETSC_DIR=/opt/moose/petsc-3.8.3 and PETSC_ARCH= C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process Completed test examples =========================================
Peacock (an optional MOOSE GUI frontend) uses many libraries. The easiest way to obtain these libraries, is to install miniconda, along with several miniconda/pip packages.
cd $STACK_SRC curl -L -O https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh sh Miniconda2-latest-Linux-x86_64.sh -b -p $PACKAGES_DIR/miniconda PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda config --set ssl_verify false PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda install -c idaholab python=2.7 coverage \ reportlab \ mako \ numpy \ scipy \ scikit-learn \ h5py \ hdf5 \ scikit-image \ requests \ vtk=7.1.0 \ pyyaml \ matplotlib \ pip \ lxml \ pyflakes \ pandas \ conda-build \ mock \ yaml \ pyqt \ swig --yes
Peacock (as well as the TestHarness sytem in MOOSE), does not work with Python3. Please chose Miniconda2 for Python 2.7 instead.
Next, we need to use
pip to install additional libraries not supplied by conda:
PATH=$PACKAGES_DIR/miniconda/bin:$PATH pip install --no-cache-dir pybtex livereload==2.5.1 daemonlite pylint==1.6.5 lxml pylatexenc anytree
Clean Up and Chown
Clean all the temporary stuff and change the ownership to root, so no further writes are possible:
rm -rf $STACK_SRC sudo chown -R root:root $PACKAGE_DIR
This concludes setting up the environment for MOOSE-based development. However you decide to instruct your users on enabling the above environment, each user will need to perform the instructions provided by the following link: Obtaining and Building MOOSE.