diff --git a/README.md b/README.md index 111a1a2e..b36c6678 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ ### SpackD : Deploying Software Stack with Spack -This is work-in-progress instructions for deploying softwrae stack with Spack. +This is work-in-progress instructions for deploying software stack with Spack. ###### CREDIT : Based on [spack-packagelist](https://github.com/epfl-scitas/spack-packagelist) @@ -78,17 +78,16 @@ These axis represent how the hierarchical modules (e.g. with LMOD) are exposed t * different python versions could be used for compiling package : python@2.7.16, python@3.6.5 -We define compiler toolchains in [packages/compiler-toolchains.yaml](packages/compiler-toolchains.yaml) as: +We define compiler toolchains in [packages/toolchains.yaml](packages/toolchains.yaml) as: ```yaml +# core toolchain typically represent system compilers core: architecture: - x86_64 - - x86_64_knl compiler: - gcc@4.8.5 - - gcc@4.8.5 mpi: None lapack: None python: None @@ -97,32 +96,28 @@ core: gnu-stable: architecture: - x86_64 - - x86_64 compiler: - gcc@6.4.0 - - gcc@6.4.0 mpi: - hpe-mpi@2.16 - - mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple lapack: - openblas@0.2.20 - - openblas@0.2.20 python: - python@3.6.5 - - python@2.7.15 ``` The `core` toolchain typically represent system compiler. This compiler is used only to bootstrap/install other compilers and some basic utility packages. And hence it doesn't provide any mpi, lapack or python packages. -The next toolchain `gnu-stable` represent default GNU compiler. We are going to provide two mpi libraries and two python versions. Note that the length of each axis should be same (and hence there are duplicate entries). +The next toolchain `gnu-stable` represent default GNU compiler. We are going to provide single version of HPE mpi library and python version. #### Packages Specifications -Once compiler toolchains are defined, we can define packages to build for each toolchain as: +Once compiler toolchains are defined, we can define packages to build for each toolchain. For example, we define compiler packages as: ``` +# list of packages to build packages: compilers: target_matrix: @@ -132,22 +127,16 @@ packages: - compiler specs: - gcc@8.2.0 - - gcc@7.3.0 - gcc@6.4.0 - - gcc@5.5.0 - intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2018.3 - - intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2017.7 - intel@18.0.3 - - intel@17.0.7 - - llvm@6.0.1 - - llvm@5.0.1 + - llvm@7.0.0 - pgi+network+nvidia+mpi@18.4 - - pgi+network+nvidia+mpi@17.10 ``` Here is brief summary of what's going on: -* `compilers` here is a just tag +* `compilers` under packages is just a tag * `target_matrix` indicates which compiler toolchain we are going to use (see Toolchain Specifications) * `requires` indicates which axis will be used while building the packages * `specs` indicates which packages we are going to build @@ -157,67 +146,41 @@ It would be more clear if we look at the package specs generated with `spackd` c To see which packages will be installed, for example, to install all compilers on the system: ``` -→ spackd --input packages/compiler-packages.yaml packages x86_64 - +→ spackd --input packages/compilers.yaml packages x86_64 gcc@8.2.0 %gcc@4.8.5 target=x86_64 -gcc@7.3.0 %gcc@4.8.5 target=x86_64 gcc@6.4.0 %gcc@4.8.5 target=x86_64 -gcc@5.5.0 %gcc@4.8.5 target=x86_64 intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2018.3 %gcc@4.8.5 target=x86_64 -intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2017.7 %gcc@4.8.5 target=x86_64 intel@18.0.3 %gcc@4.8.5 target=x86_64 -intel@17.0.7 %gcc@4.8.5 target=x86_64 -llvm@6.0.1 %gcc@4.8.5 target=x86_64 -llvm@5.0.1 %gcc@4.8.5 target=x86_64 +llvm@7.0.0 %gcc@4.8.5 target=x86_64 pgi+network+nvidia+mpi@18.4 %gcc@4.8.5 target=x86_64 -pgi+network+nvidia+mpi@17.10 %gcc@4.8.5 target=x86_64 ``` These specs can be the used to install packages using `spack install` command. This become more useful when multiple compilers and mpi libraries come into picture, for example, to install all parallel libraries on the system : ``` → spackd --input packages/parallel-libraries.yaml packages x86_64 - -hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %clang@6.0.1 target=x86_64 -omega-h@9.17.2 ^hpe-mpi@2.16 %clang@6.0.1 target=x86_64 -... -hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64 -omega-h@9.17.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64 -... +hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %intel@18.0.3 target=x86_64 hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %gcc@6.4.0 target=x86_64 -omega-h@9.17.2 ^hpe-mpi@2.16 %gcc@6.4.0 target=x86_64 -... -hdf5+mpi@1.10.2 ^mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple %gcc@6.4.0 target=x86_64 -omega-h@9.17.2 ^mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple %gcc@6.4.0 target=x86_64 -.... +omega-h@9.17.2 ^openblas@0.2.20 %gcc@6.4.0 target=x86_64 +petsc+int64+mpi@3.9.3 ^openblas@0.2.20 %gcc@6.4.0 target=x86_64 +trilinos@12.12.1 ^openblas@0.2.20 %gcc@6.4.0 target=x86_64 +omega-h@9.17.2 ^intel-mkl@2018.1.163 %intel@18.0.3 target=x86_64 +petsc+int64+mpi@3.9.3 ^intel-mkl@2018.1.163 %intel@18.0.3 target=x86_64 +trilinos@12.12.1 ^intel-mkl@2018.1.163 %intel@18.0.3 target=x86_64 ``` -Similarly, packages for `x86_64_knl` target can be generated with : - -``` - → spackd --input packages/parallel-libraries.yaml packages x86_64_knl - -hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl -omega-h@9.17.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl -petsc+int64+mpi@3.9.3 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl -trilinos+kokkos+teuchos@12.12.1 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl -``` - -The specs become complicated but they are complete considering architecture, compiler and mpi dependecies. - - #### Whole Software Stack Here is list of all packages (in order) that we will be installing : ``` -spackd --input packages/compiler-packages.yaml packages x86_64 --output compiler-packages.txt -spackd --input packages/build-dep-packages.yaml packages x86_64 --output buiild-dep-packages.txt +spackd --input packages/compilers.yaml packages x86_64 --output compilers.txt +spackd --input packages/system-tools.yaml packages x86_64 --output system-tools.txt spackd --input packages/serial-libraries.yaml packages x86_64 --output serial-libraries.txt spackd --input packages/python-packages.yaml packages x86_64 --output python-packages.txt spackd --input packages/parallel-libraries.yaml packages x86_64 --output parallel-libraries.txt +spackd --input packages/bbp-packages.yaml packages x86_64 --output bbp-packages.txt ``` -Similarly for target `x86_64_knl`. #### Todo : Deployment Workflow @@ -227,7 +190,7 @@ Here is how deployment workflow should look like : #### Todo : Jenkins Pipeline Workflow -See exisiting scripts in `scripts/` directory and `Jenkinsfile`. Those should be refactored for new workflow. +See existing scripts in `scripts/` directory and `Jenkinsfile`. Those should be refactored for new workflow. #### Other Commands @@ -238,7 +201,6 @@ x86_64 → spackd --input packages/parallel-libraries.yaml compilers x86_64 clang@6.0.1%gcc@4.8.5 target=x86_64 gcc@6.4.0%gcc@4.8.5 target=x86_64 -intel@18.0.1%gcc@4.8.5 target=x86_64 -intel@18.0.1-knl%gcc@4.8.5 target=x86_64 +intel@18.0.3%gcc@4.8.5 target=x86_64 pgi@18.4%gcc@4.8.5 target=x86_64 ``` diff --git a/check_specs.sh b/check_specs.sh new file mode 100755 index 00000000..0e0f39cf --- /dev/null +++ b/check_specs.sh @@ -0,0 +1,29 @@ +#!/bin/bash +set +e +source "setup_env.sh" + + +check_specs() { + package_list=$1 + to_be_installed=$(spack filter --not-installed $(cat $package_list)) + + if [[ -z "${to_be_installed}" ]] + then + echo "All specs already installed" + else + echo "spack spec -Il $to_be_installed" + spack spec -Il $to_be_installed + fi +} + + +cd $WORKSPACE/HOME_DIR/spack-deploy +for category in "${package_categories[@]}" +do + package_list=`basename $category .yaml`.txt + echo "Checking installed packages in category $category" + check_specs $package_list +done + + +echo -e "Spec concretisation check finished" diff --git a/configs/compilers/config.yaml b/configs/compilers/config.yaml deleted file mode 100644 index 1c9872c9..00000000 --- a/configs/compilers/config.yaml +++ /dev/null @@ -1,8 +0,0 @@ -config: - install_tree: $SOFTS_DIR_PATH/install - source_cache: $SOFTS_DIR_PATH/.cache - module_roots: - tcl: $SOFTS_DIR_PATH/modules/tcl - lmod: $SOFTS_DIR_PATH/modules/lmod - install_path_scheme: '${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH:6}' - build_jobs: 36 diff --git a/configs/config.yaml b/configs/config.yaml index df93eb96..9f793dca 100644 --- a/configs/config.yaml +++ b/configs/config.yaml @@ -1,12 +1,11 @@ config: - # Root of the installation tree - install_tree: /ssoft/spack/paien/v2/opt/spack/ - - # Root of the module files + install_tree: $SOFTS_DIR_PATH/install + source_cache: $SOFTS_DIR_PATH/.cache module_roots: - tcl: /ssoft/spack/paien/v2/share/spack/modules - lmod: /ssoft/spack/paien/v2/share/spack/lmod - dotkit: /ssoft/spack/paien/v2/share/spack/dotkit - + tcl: $SOFTS_DIR_PATH/modules/tcl + lmod: $SOFTS_DIR_PATH/modules/lmod build_stage: - - /tmp/paien.v2 \ No newline at end of file + - $tempdir + - $SOFTS_DIR_PATH/.stage + install_path_scheme: '${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH:10}' + build_jobs: 36 diff --git a/configs/packages.yaml b/configs/packages.yaml index d9767d52..4afa9e3f 100644 --- a/configs/packages.yaml +++ b/configs/packages.yaml @@ -1,112 +1,131 @@ packages: - - #### - # Common libraries that we delegate to the OS - #### - - openssl: - buildable: False - paths: - openssl@system: /usr/lib64 - - jdk: - buildable: False - paths: - jdk@8u141-b16: /usr/lib/jvm/jre - - #### - # Intel compilers and libraries - #### - - intel: - buildable: False - paths: - intel@18.0.2%gcc@4.8.5: /ssoft/spack/external/intel/2018.2/compilers_and_libraries_2018.2.199/linux - - intel-mpi: - buildable: False - paths: - intel-mpi@2018.2.199 %intel@18.0.2: /ssoft/spack/external/intel/2018.2/impi/2018.2.199 - - intel-mkl: - buildable: False - paths: - intel-mkl @2018.2.199 %intel@18.0.2: /ssoft/spack/external/intel/2018.2/compilers_and_libraries_2018.2.199/linux/mkl - intel-mkl threads=openmp @2018.2.199 %intel@18.0.2: /ssoft/spack/external/intel/2018.2/compilers_and_libraries_2018.2.199/linux/mkl - - #### - # Third party software - #### - - # TODO: Abaqus needs to be reinstalled with the intel@2018 - # TODO: first available window is beginning of July - # abaqus: - # buildable: False - # version: ['@6.14-1'] - # paths: - # abaqus@6.14-1%intel@17.0.2: /ssoft/spack/external/abaqus/6.14-1 - - adf: - buildable: False - paths: - adf@2017.111 %gcc@4.8.5: /ssoft/spack/external/adf/adf2017.111 - - ansys: - buildable: False - paths: - ansys@17.1%gcc@4.8.5: /ssoft/spack/external/ansys/17.1/v171 - - cfdplusplus: - buildable: False - paths: - cfdplusplus@16.1%gcc@4.8.5: /ssoft/spack/external/CFD++/2016.05 - - comsol: - buildable: False - paths: - comsol@5.3 %gcc@4.8.5: /ssoft/spack/external/comsol/5.3 - - fdtd: - buildable: False - paths: - fdtd@8.20.1703 %gcc@4.8.5: /ssoft/spack/external/fdtd/8.20.1703 - fdtd@8.19.1416-1 %gcc@4.8.5: /ssoft/spack/external/fdtd/8.19.1416-1 - # installed due to segfault with other versions INC0204566 - fdtd@8.12.527 %gcc@4.8.5: /ssoft/spack/external/fdtd/8.12.527 - - likwid: - buildable: False - version: ['@4.3.0'] - paths: - likwid@4.3.0 %gcc@4.8.5: /ssoft/spack/external/LIKWID/4.3.0 - - gaussian: - buildable: False - paths: - gaussian@g16-A.03 %gcc@4.8.5: /ssoft/spack/external/gaussian/g16-A.03 - - matlab: - buildable: False - paths: - matlab@R2018a %gcc@4.8.5: /ssoft/spack/external/MATLAB/R2018a - - maple: - buildable: False - paths: - maple@2017 %gcc@4.8.5: /ssoft/spack/external/Maple/2017 - - mathematica: - buildable: False - paths: - mathematica@9.0.1 %gcc@4.8.5: /ssoft/spack/external/Mathematica/9.0.1 - mathematica@11.1.1 %gcc@4.8.5: /ssoft/spack/external/Mathematica/11.1.1 - - smr: - buildable: False - paths: - smr@2017.06 %gcc@4.8.5: /ssoft/spack/external/SMR/2017.06 - - totalview: - buildable: False - paths: - totalview@2017.2.11 %gcc@4.8.5: /ssoft/spack/external/toolworks/totalview.2017.2.11 + autoconf: + paths: + autoconf@2.69: /usr + version: [2.69] + automake: + paths: + automake@1.13.4: /usr + version: [1.13.4] + bison: + paths: + bison@3.0.4: /usr + version: [3.0.4] + bzip2: + paths: + bzip2@1.0.6: /usr + version: [1.0.6] + cairo: + paths: + cairo@1.8.10: /usr + version: [1.8.10] + curl: + paths: + curl@7.29.0: /usr + version: [7.29.0] + flex: + paths: + flex@2.5.37: /usr + version: [2.5.37] + fontconfig: + paths: + fontconfig@2.10.95: /usr + version: [2.10.95] + glib: + paths: + glib@2.50.3: /usr + version: [2.50.3] + gtkplus: + paths: + gtkplus@2.24.31: /usr + version: [2.24.31] + hpe-mpi: + paths: + hpe-mpi@2.16: /opt/hpe/hpc/mpt/mpt-2.16 + version: [2.16] + libjpeg: + paths: + libjpeg@1.2.90: /usr + version: [1.2.90] + libgcrypt: + paths: + libgcrypt@1.5.3: /usr + version: [1.5.3] + libmng: + paths: + libmng@1.0.10: /usr + version: [1.0.10] + libtool: + paths: + libtool@2.4.2: /usr + version: [2.4.2] + libxml2: + paths: + libxml2@2.9.1: /usr + version: [2.9.1] + m4: + paths: + m4@1.4.16: /usr + version: [1.4.16] + ncurses: + paths: + ncurses@5.9: /usr + version: [5.9] + # can not be installed with python 3 + node-js: + paths: + node-js@6.12.2: /usr + version: [6.12.2] + openssl: + paths: + openssl@1.0.2k: /usr + version: [1.0.2k] + pango: + paths: + pango@1.40.4: /usr + version: [1.40.4] + pcre: + paths: + pcre@8.32+jit+utf: /usr + version: [8.32] + perl: + paths: + perl@5.16.3: /usr + version: [5.16.3] + petsc: + variants: +patchmpi64 + pkg-config: + paths: + pkg-config@0.27.1: /usr + version: [0.27.1] + slurm: + paths: + slurm@17.02: /usr + buildable: False + version: [17.02] + steps: + variants: +lapack+petsc+mpi + tar: + paths: + tar@1.26: /usr + version: [1.26] + #tcl: + # paths: + # tcl@8.5.13: /usr + # version: [8.5.13] + tk: + paths: + tk@8.5.13: /usr + version: [8.5.13] + trilinos: + variants: +kokkos+teuchos~amesos~hypre~superlu-dist~mumps~metis~suite-sparse + version: [xsdk-0.4.0-rc1] + xz: + paths: + xz@5.2: /usr + version: [5.2] + all: + compiler: [gcc@6.4.0, intel@18.0.1] + providers: + mpi: [hpe-mpi, intel-mpi] + scalapack: [intel-mkl, netlib-scalapack] diff --git a/configs/repos.yaml b/configs/repos.yaml deleted file mode 100644 index 4ac9b5e1..00000000 --- a/configs/repos.yaml +++ /dev/null @@ -1,2 +0,0 @@ -repos: -- /ssoft/spack/paien/scitas-repository/scitasexternal diff --git a/create_mirror.sh b/create_mirror.sh new file mode 100755 index 00000000..2bee95ba --- /dev/null +++ b/create_mirror.sh @@ -0,0 +1,33 @@ +#!/bin/bash +set -e +source "setup_env.sh" + + +create_mirror() { + package_list=$1 + to_be_installed=$(spack filter --not-installed $(cat $package_list)) + + if [[ -z "${to_be_installed}" ]] + then + echo "All specs already installed" + else + echo "Populating mirror" + spack mirror create -D -d ${SPACK_MIRROR_DIR} $to_be_installed + fi +} + + +cd $WORKSPACE/HOME_DIR/spack-deploy +for category in "${package_categories[@]}" +do + # skip compilers for now + if [[ $category = *"compilers"* ]]; then + continue + fi + + package_list=`basename $category .yaml`.txt + echo "Creating mirror for packages in category $category" + create_mirror $package_list +done + +echo -e "Mirror creation finished" diff --git a/install_specs.sh b/install_specs.sh new file mode 100755 index 00000000..121dcbfa --- /dev/null +++ b/install_specs.sh @@ -0,0 +1,91 @@ +#!/bin/bash +set -e +source "setup_env.sh" + +timestamp=`date +"%a-%d-%m-%Y-%H-%M"` + +configure_compilers() { + GCC_DIR=`spack location --install-dir gcc@6.4.0` + + if [ ! -d $GCC_DIR ]; then + echo "Error : gcc@6.4.0 not installed ?" + exit 1 + fi + + while read -r line; do + set +o nounset + spack load ${line} + set -o nounset + if [[ ${line} != *"intel-parallel-studio"* ]]; then + spack compiler find --scope=user + fi + + if [[ ${line} = *"intel"* ]]; then + # update intel modules to use gcc@6.4.0 in .cfg files + install_dir=$(spack location --install-dir ${line}) + for f in $(find ${install_dir} -name "icc.cfg" -o -name "icpc.cfg" -o -name "ifort.cfg"); do + if ! grep -q "${GCC_DIR}" $f; then + echo "-gcc-name=${GCC_DIR}/bin/gcc" >> ${f} + echo "-Xlinker -rpath=${GCC_DIR}/lib" >> ${f} + echo "-Xlinker -rpath=${GCC_DIR}/lib64" >> ${f} + echo "[CFG] Updated ${f} with newer GCC" + fi + done + elif [[ ${line} = *"pgi"* ]]; then + #update pgi modules for network installation (on uc2 nodes) + PGI_DIR=$(dirname $(which makelocalrc)) + makelocalrc ${PGI_DIR} -gcc ${GCC_DIR}/bin/gcc -gpp ${GCC_DIR}/bin/g++ -g77 ${GCC_DIR}/bin/gfortran -x -net + + #configure pgi network license + template=`find $PGI_DIR -name localrc* | tail -n 1` + for node in bbpv1 bbpv2 r2i3n0 r2i3n1 r2i3n2 r2i3n3 r2i3n4 r2i3n5 r2i3n6 + do + cp $template $PGI_DIR/localrc.$node || echo "Same file" + done + fi + spack unload ${line} + done + + sed -i 's#.*f\(77\|c\): null# f\1: /usr/bin/gfortran#' ${HOME}/.spack/compilers.yaml +} + + + +install_specs() { + package_list=$1 + category=$2 + to_be_installed=$(spack filter --not-installed $(cat $package_list)) + + if [[ -z "${to_be_installed}" ]] + then + echo "All specs already installed" + else + echo "Installing packages " + spack spec -Il ${to_be_installed} + spack install --log-format=junit --log-file=${category}.${timestamp}.xml ${to_be_installed} + #spack filter --not-installed $(cat $package_list) > todo.txt + #while read -r package + #do + # spack spec -Il ${package} + # spack install --log-format=junit --log-file=${category}.xml ${package} + #done < todo.txt + + if [[ $category = *"compilers"* ]]; then + echo "Configuring compilers " + configure_compilers <<< "${to_be_installed}" + fi + fi +} + +# unset mpi variables +unset `env | awk -F= '/^\w/ {print $1}' | egrep '(PMI|SLURM_)' | xargs` + +cd $WORKSPACE/HOME_DIR/spack-deploy +for category in "${package_categories[@]}" +do + package_list=`basename $category .yaml`.txt + echo "Installing packages in category $category" + install_specs $package_list $category +done + +echo -e "All specs installed" diff --git a/packages/bbp-packages.yaml b/packages/bbp-packages.yaml new file mode 100644 index 00000000..7d4e60d8 --- /dev/null +++ b/packages/bbp-packages.yaml @@ -0,0 +1,103 @@ +# Package configurations for BBP Software Stack + +axis: + - architecture + - compiler + - mpi + - lapack + - python + +combinations: !include packages/toolchains.yaml + +packages: + + #### circuit building #### + + gnu-stable-serial: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - python + specs: + - spykfunc@0.11.0 + + gnu-stable-parallel: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - mpi + specs: + - functionalizer@3.11.0 + - touchdetector@4.3.3 + + gnu-stable-parallel-python: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - mpi + - python + specs: + - parquet-converters@0.2 + + #### sub-cellular #### + + gnu-stable-lapack-python-parallel: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - lapack + - mpi + - python + specs: + - steps@3.3.0+petsc + + gnu-stable-lapack-parallel: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - mpi + - lapack + - python + specs: + - zee@develop + + gnu-stable-lapack-python: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - lapack + - python + specs: + - py-bluepymm@0.6.38 ^neuron~mpi + - py-bluepyopt@1.6.56 ^neuron~mpi + - py-efel@3.0.22 + + #### cellular #### + + gnu-intel-stable-python-parallel: + target_matrix: + - gnu-stable + - intel-stable + requires: + - architecture + - compiler + - mpi + - python + specs: + - neurodamus@plasticity + - neurodamus@master~coreneuron + - neurodamus@hippocampus~coreneuron + - neuron@2018-10 + - synapsetool@0.3.2 diff --git a/packages/compiler-packages.yaml b/packages/compilers.yaml similarity index 68% rename from packages/compiler-packages.yaml rename to packages/compilers.yaml index 3c5fe089..5c92bdc7 100644 --- a/packages/compiler-packages.yaml +++ b/packages/compilers.yaml @@ -1,4 +1,4 @@ -# Build all compilers packages +# Compiler Specs # axis that defines software deployment strategy axis: @@ -20,7 +20,7 @@ combinations: lapack: None python: None -# list of packages to build +# list of compilers to build packages: compilers: target_matrix: @@ -30,14 +30,8 @@ packages: - compiler specs: - gcc@8.2.0 - - gcc@7.3.0 - gcc@6.4.0 - - gcc@5.5.0 - intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2018.3 - - intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2017.7 - intel@18.0.3 - - intel@17.0.7 - - llvm@6.0.1 - - llvm@5.0.1 + - llvm@7.0.0 - pgi+network+nvidia+mpi@18.4 - - pgi+network+nvidia+mpi@17.10 diff --git a/packages/parallel-libraries.yaml b/packages/parallel-libraries.yaml index 16d396b8..761b8d27 100644 --- a/packages/parallel-libraries.yaml +++ b/packages/parallel-libraries.yaml @@ -1,6 +1,5 @@ -# Build all parallel libraries +# Packages and libraries require MPI -# axis that defines software deployment strategy axis: - architecture - compiler @@ -8,48 +7,56 @@ axis: - lapack - python -# toolchain combinations -combinations: !include packages/compiler-toolchains.yaml.in +combinations: !include packages/toolchains.yaml -parallel-packages: ¶llel-specs - specs: - - hdf5+mpi@1.10.2 - - omega-h@9.17.2 - - petsc+int64+mpi@3.9.3 - - trilinos+kokkos+teuchos~amesos~hypre~superlu-dist~mumps~metis~suite-sparse@12.12.1 - -# list of packages to build packages: - # with default gnu, intel toolchains - gnu-intel-stable-parallel: + + gnu-stable-parallel: target_matrix: - gnu-stable - - intel-stable - - intel-knl-stable requires: - architecture - compiler - mpi - <<: *parallel-specs - # with default pgi toolchain - pgi-stable-parallel: + specs: + - hdf5+mpi@1.10.3 + - highfive@1.6 + - matio@1.5.9 + - netcdf@4.6.1 + - omega-h@9.22.1 + - parmetis@4.0.3 + + gnu-stable-parallel-lapack: target_matrix: - - pgi-stable + - gnu-stable requires: - architecture - compiler - mpi - <<: *parallel-specs - blacklist: - - omega-h@9.17.2 - - petsc+int64+mpi@3.9.3 - - trilinos+kokkos+teuchos@12.12.1 - # with default llvm toolchain - llvm-stable-parallel: + - lapack + specs: + - superlu-dist@5.4.0 + - trilinos@xsdk-0.4.0-rc1 + + gnu-stable-parallel-lapack-python: target_matrix: - - llvm-stable + - gnu-stable + requires: + - architecture + - compiler + - lapack + - mpi + - python + specs: + - petsc+int64+mpi@3.10.1 + + gnu-intel-stable-parallel: + target_matrix: + - gnu-stable + - intel-stable requires: - architecture - compiler - mpi - <<: *parallel-specs + specs: + - neuron@2018-10 diff --git a/packages/python-packages.yaml b/packages/python-packages.yaml index a5278802..3df98bdc 100644 --- a/packages/python-packages.yaml +++ b/packages/python-packages.yaml @@ -1,4 +1,5 @@ -# axis that defines software deployment strategy +# Python packages + axis: - architecture - compiler @@ -6,98 +7,113 @@ axis: - lapack - python -# toolchain combinations -combinations: !include packages/compiler-toolchains.yaml.in - -serial-packages: &python-specs - specs: - - py-argcomplete - - py-argparse - - py-beautifulsoup4 - - py-bokeh - - py-click - - py-coverage - - py-cpuinfo - - py-csvkit - - py-cython - - py-cpuinfo - - py-dateutil - - py-elasticsearch - - py-elephant - - py-enum34 - - py-fastaindex - - py-flake8 - - py-future - - py-gcovr - - py-gnuplot - - py-h5py - - py-ipykernel - - py-ipython - - py-ipython-genutils - - py-ipywidgets - - py-ipython - - py-jinja2 - - py-jupyter-client - - py-jupyter-console - - py-jupyter-core - - py-jupyter-notebook - - py-lazy-property - - py-line-profiler - - py-lxml - - py-markdown - - py-matplotlib - - py-memory-profiler - - py-multiprocess - - py-neo - - py-networkx - - py-nose - - py-numba - - py-pandas - - py-perf - - py-pil - - py-pillow - - py-pip - - py-ply - - py-pynn - - py-pytest - - py-pyyaml - - py-regex - - py-setuptools - - py-shapely - - py-simplejson - - py-six - - py-slurm-pipeline - - py-sphinx - - py-sphinx-bootstrap-theme - - py-sphinx-rtd-theme - - py-symengine - - py-sympy - - py-toml - - py-virtualenv - - py-virtualenv-clone - - py-virtualenvwrapper - - py-wheel - - py-yapf - - py-zmq +combinations: !include packages/toolchains.yaml -# list of packages to build packages: - # with default gnu, intel and intel-knl toolchain - stable-serial: + + gnu-stable-serial: target_matrix: - gnu-stable - - intel-stable - - intel-knl-stable requires: - architecture - compiler - python - <<: *python-specs - stable-serial-lapack: + specs: + - py-argcomplete + - py-argparse + - py-babel + - py-bb5 + - py-beautifulsoup4 + - py-bokeh + - py-cffi + - py-click + - py-clustershell + - py-coverage + - py-cpuinfo + - py-csvkit + - py-cycler + - py-cython + - py-cpuinfo + - py-dateutil + - py-deap + - py-decorator + - py-docopt + - py-efel + - py-elasticsearch + - py-elephant + - py-fastaindex + - py-flake8 + - py-funcsigs + - py-future + - py-futures + - py-gcovr + - py-h5py~mpi + - py-hdfs + - py-ipykernel + - py-ipyparallel + - py-ipython + - py-ipython-genutils + - py-ipywidgets + - py-ipython + - py-jinja2 + - py-jprops + - py-jupyter-client + - py-jupyter-console + - py-jupyter-core + - py-jupyter-notebook + - py-lazy-property + - py-line-profiler + - py-lxml + - py-markdown + - py-matplotlib + - py-memory-profiler + - py-mpmath + - py-multiprocess + - py-neo + - py-networkx + - py-nose + - py-pandas + - py-perf + - py-pillow + - py-pip + - py-ply + - py-progress + - py-py4j + - py-pyarrow + - py-pynn + - py-pyspark + - py-pytest + - py-pyyaml + - py-regex + - py-requests + - py-seaborn + - py-setuptools + - py-setuptools-scm + - py-sh + - py-shapely + - py-simplejson + - py-six + - py-slurm-pipeline + - py-sparkmanager + - py-sphinx + - py-sphinx-bootstrap-theme + - py-sphinx-rtd-theme + - py-sqlalchemy + - py-symengine + - py-sympy + - py-toml + - py-tornado + - py-urllib3 + - py-virtualenv + - py-virtualenv-clone + - py-virtualenvwrapper + - py-wheel + - py-yapf + - py-zmq + + gnu-stable-serial-lapack: target_matrix: - gnu-stable - - intel-stable - - intel-knl-stable requires: - architecture - compiler @@ -110,15 +126,14 @@ packages: - py-scikit-learn@0.19.1 ^py-numpy@1.15.1 ^py-scipy@1.1.0 - py-theano@1.0.2 ^py-numpy@1.15.1 ^py-scipy@1.1.0 - py-pytorch@0.4.0 ^py-numpy@1.15.1 - stable-parallel: + + gnu-stable-parallel-python: target_matrix: - gnu-stable - - intel-stable - - intel-knl-stable requires: - architecture - compiler - mpi - python specs: - - py-mpi4py@3.0.0 + - py-mpi4py@3.0.0 diff --git a/packages/serial-libraries.yaml b/packages/serial-libraries.yaml index 096158b9..e3fa8347 100644 --- a/packages/serial-libraries.yaml +++ b/packages/serial-libraries.yaml @@ -1,4 +1,5 @@ -# axis that defines software deployment strategy +# Serial libraries and packages + axis: - architecture - compiler @@ -6,66 +7,63 @@ axis: - lapack - python -# toolchain combinations -combinations: !include packages/compiler-toolchains.yaml.in - -serial-packages: &serial-specs - specs: - - boost~mpi~python@1.68.0 - - hdf5~mpi@1.10.2 - - highfive~mpi@1.6 - - hpe-mpi@2.16 - - intel-mpi@2018.1.163 - - intel-mkl@2018.1.163 - - libxml2@2.9.8 - - mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple - - parquet@1.4.0 - - python@2.7.15 - - python@3.6.5 - - qt@5.10.0 - - zlib@1.2.11 +combinations: !include packages/toolchains.yaml -# list of packages to build packages: - # with default gnu toolchain + gnu-stable-serial: target_matrix: - gnu-stable requires: - architecture - compiler - <<: *serial-specs - # with default intel toolchains - intel-stable-serial: - target_matrix: - - intel-stable - - intel-knl-stable - requires: - - architecture - - compiler - <<: *serial-specs - blacklist: + specs: + - arrow+parquet~python@0.11.0 + - boost~mpi@1.68.0 + - bzip2@1.0.6 + - cuda@9.2.88 + - gmsh+oce~mpi@3.0.6 + - hdf5~mpi@1.10.3 + - hadoop@2.9.0 + - highfive~mpi@1.6 + - hpe-mpi@2.16 + - intel-mpi@2018.1.163 + - intel-mkl@2018.1.163 + - jdk@1.8.0_191-b12 + - libjpeg@1.2.90 + - libjpeg@1.2.90 + - libxml2@2.9.8 + - libxslt@1.1.29 + - metis@5.1.0 + - ncurses@5.9 + - openblas@0.3.3 + - python@3.6.5 + - qhull@2015.2 - qt@5.10.0 - # with default pgi toolchain - pgi-stable-serial: + - readline@7.0 + - spark@2.3.2rc2 + - sqlite@3.23.1 + - symengine@0.3.0 + - zeromq@4.2.5 + - zlib@1.2.11 + + gnu-stable-serial-python: target_matrix: - - llvm-stable + - gnu-stable requires: - architecture - compiler - <<: *serial-specs - blacklist: - - intel-mkl@2018.1.163 - - parquet@1.4.0 - - qt@5.10.0 - # with default llvm toolchain - llvm-stable-serial: + - python + specs: + - arrow+parquet+python@0.11.0 + - neuron~mpi@2018-10 + + intel-stable-serial-python: target_matrix: - - llvm-stable + - intel-stable requires: - architecture - compiler - <<: *serial-specs - blacklist: - - parquet@1.4.0 - - qt@5.10.0 + - python + specs: + - neuron~mpi@2018-10 diff --git a/packages/build-dep-packages.yaml b/packages/system-tools.yaml similarity index 50% rename from packages/build-dep-packages.yaml rename to packages/system-tools.yaml index 6674487e..a44364e0 100644 --- a/packages/build-dep-packages.yaml +++ b/packages/system-tools.yaml @@ -1,6 +1,5 @@ -# Common (build) dependency packages +# Common dependency packages -# axis that defines software deployment strategy axis: - architecture - compiler @@ -8,41 +7,53 @@ axis: - lapack - python -# toolchain combinations -combinations: !include packages/compiler-toolchains.yaml.in +combinations: !include packages/toolchains.yaml -# list of packages to build packages: - # all serial packages with system compiler - gnu-core-serial: + + gnu-stable-serial: target_matrix: - - core + - gnu-stable requires: - architecture - compiler specs: - autoconf@2.69 - automake@1.16.1 - - astyle@3.1 + - binutils@2.31.1 - bison@3.0.5 - cmake@3.12.2 - - cuda@9.2.88 - - cuda@8.0.61 - - darshan-util - environment-modules@3.2.10 - flex@2.6.3 - - gdb@8.2 + - gdbm@1.14.1 - gettext@0.19.8.1 - git@2.18.0 - - iozone@3_465 - - jdk@10.0.2_13 + - gmp@6.1.2 + - help2man@1.47.4 + - isl@0.18 - libtool@2.4.6 - - lmod@7.8 + - libx11@1.6.5 - m4@1.4.18 - - ninja@1.8.2 + - mpc@1.1.0 + - node-js@6.12.2 + - npm@3.10.9 - pkg-config@0.29.2 - - spark@2.3.0+hadoop + - swig@3.0.12 - tcl@8.6.8 + - texinfo@6.5 - tk@8.6.8 - util-macros@1.19.1 - valgrind~mpi@3.13.0 + + gnu-stable-python-serial: + target_matrix: + - gnu-stable + requires: + - architecture + - compiler + - python + specs: + - gdb@8.2 + - ninja@1.8.2 + - py-lit@0.5.0 + - py-setuptools@40.2.0 diff --git a/packages/compiler-toolchains.yaml.in b/packages/toolchains.yaml similarity index 64% rename from packages/compiler-toolchains.yaml.in rename to packages/toolchains.yaml index 3cc0284d..6c3faa25 100644 --- a/packages/compiler-toolchains.yaml.in +++ b/packages/toolchains.yaml @@ -2,10 +2,8 @@ core: architecture: - x86_64 - - x86_64_knl compiler: - gcc@4.8.5 - - gcc@4.8.5 mpi: None lapack: None python: None @@ -14,55 +12,40 @@ core: gnu-stable: architecture: - x86_64 - - x86_64 compiler: - gcc@6.4.0 - - gcc@6.4.0 mpi: - hpe-mpi@2.16 - - mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple lapack: - - openblas@0.2.20 - - openblas@0.2.20 + - openblas@0.3.3 python: - python@3.6.5 - - python@2.7.15 # stable intel toolchain intel-stable: architecture: - x86_64 - - x86_64 compiler: - - intel@18.0.1 - - intel@18.0.1 + - intel@18.0.3 mpi: - hpe-mpi@2.16 - - mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple lapack: - intel-mkl@2018.1.163 - - intel-mkl@2018.1.163 python: - python@3.6.5 - - python@2.7.15 # stable intel knl toolchain intel-knl-stable: architecture: - x86_64_knl - - x86_64_knl compiler: - - intel@18.0.1 - - intel@18.0.1 + - intel@18.0.3 mpi: - hpe-mpi@2.16 - - hpe-mpi@2.16 lapack: - intel-mkl@2018.1.163 - - intel-mkl@2018.1.163 python: - python@3.6.5 - - python@2.7.15 # stable PGI toolchain pgi-stable: @@ -73,7 +56,7 @@ pgi-stable: mpi: - hpe-mpi@2.16 lapack: - - openblas@0.2.20 + - openblas@0.3.3 python: - python@2.7.15 @@ -86,6 +69,6 @@ llvm-stable: mpi: - hpe-mpi@2.16 lapack: - - openblas@0.2.20 + - openblas@0.3.3 python: - python@2.7.15 diff --git a/setup_env.sh b/setup_env.sh new file mode 100755 index 00000000..93340336 --- /dev/null +++ b/setup_env.sh @@ -0,0 +1,66 @@ +#!/bin/bash + +############################## SPACK REPOSITORY ############################# +export WORKSPACE=`pwd` + +echo " +===================================================================== +Preparing environment... +=====================================================================" + +export PATH=`pwd`:$PATH +export WORKSPACE=`pwd`/workspace +export SOFTS_DIR_PATH=$WORKSPACE/deployment +mkdir -p $WORKSPACE/HOME_DIR $SOFTS_DIR_PATH $SPACK_MIRROR_DIR + +# TODO : Change this +#export SPACK_MIRROR_DIR=$WORKSPACE/mirror +export SPACK_MIRROR_DIR=/gpfs/bbp.cscs.ch/apps/hpc/test/central-mirror + +# new $HOME to avoid conflict with ~/.spack +export HOME=$WORKSPACE/HOME_DIR +#rm -rf $HOME/.spack +cd $HOME + +# clone both spack related repos +if [ ! -d spack ]; then + git clone https://github.com/BlueBrain/spack.git -b fix/python-packages + git clone https://github.com/pramodk/spack-deploy.git -b base-packages + git clone ssh://bbpcode.epfl.ch/user/kumbhar/spack-licenses spack/etc/spack/licenses +fi + +export SPACK_ROOT=`pwd`/spack +export PATH=$SPACK_ROOT/bin:$PATH + +# copy config files +mkdir -p $SPACK_ROOT/etc/spack/defaults/linux +cp $WORKSPACE/HOME_DIR/spack-deploy/configs/packages.yaml $SPACK_ROOT/etc/spack/defaults/linux/ +cp $WORKSPACE/HOME_DIR/spack-deploy/configs/config.yaml $SPACK_ROOT/etc/spack/defaults/linux/ + +source $SPACK_ROOT/share/spack/setup-env.sh + +# create virtualenv +SPACKD_VIRTUALENV_PATH=`pwd`/venv +if [ ! -d $SPACKD_VIRTUALENV_PATH ]; then + virtualenv -p $(which python) ${SPACKD_VIRTUALENV_PATH} --clear + . ${SPACKD_VIRTUALENV_PATH}/bin/activate + curl https://bootstrap.pypa.io/get-pip.py | python +fi + +. ${SPACKD_VIRTUALENV_PATH}/bin/activate +pip install -U spack-deploy/ + +# create list of all packages +declare -a package_categories=(compilers.yaml system-tools.yaml serial-libraries.yaml python-packages.yaml parallel-libraries.yaml bbp-packages.yaml) + +cd $WORKSPACE/HOME_DIR/spack-deploy +for config in "${package_categories[@]}" +do + package_list=`basename $config .yaml`.txt + spackd --input packages/$config packages x86_64 --output $package_list +done + +deactivate + +# use mirror with spack +spack mirror add --scope=site central_mirror ${SPACK_MIRROR_DIR} || echo "Mirror in scope already added!" diff --git a/spackd.py b/spackd.py index b94898fd..de026b93 100755 --- a/spackd.py +++ b/spackd.py @@ -226,4 +226,9 @@ def packages(ctx, target, output, only): penv = ProductionEnvironment(ctx.parent.configuration, only=only) for item in filter(lambda x: x.architecture == target, penv.items()): - output.write(item.spec + ' %' + item.compiler + ' target=' + item.architecture + '\n') + # compiler spec should come before all dependencies + if '^' in item.spec: + spec = item.spec.replace(' ^', ' %%%s ^' % item.compiler, 1) + else: + spec = item.spec + ' %' + item.compiler + output.write(spec + ' target=' + item.architecture + '\n')