Adding base packages.yaml

README.md

@@ -1,6 +1,6 @@
 ### SpackD : Deploying Software Stack with Spack
 
-This is work-in-progress instructions for deploying softwrae stack with Spack.
+This is work-in-progress instructions for deploying software stack with Spack.
 
 ###### CREDIT : Based on [spack-packagelist](https://github.com/epfl-scitas/spack-packagelist)
 
@@ -78,17 +78,16 @@ These axis represent how the hierarchical modules (e.g. with LMOD) are exposed t
 * different python versions could be used for compiling package : python@2.7.16, python@3.6.5
 
 
-We define compiler toolchains in [packages/compiler-toolchains.yaml](packages/compiler-toolchains.yaml) as:
+We define compiler toolchains in [packages/toolchains.yaml](packages/toolchains.yaml) as:
 
 
 ```yaml
+# core toolchain typically represent system compilers
 core:
   architecture:
   - x86_64
-  - x86_64_knl
   compiler:
   - gcc@4.8.5
-  - gcc@4.8.5
   mpi: None
   lapack: None
   python: None
@@ -97,32 +96,28 @@ core:
 gnu-stable:
   architecture:
   - x86_64
-  - x86_64
   compiler:
   - gcc@6.4.0
-  - gcc@6.4.0
   mpi:
   - hpe-mpi@2.16
-  - mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple
   lapack:
   - openblas@0.2.20
-  - openblas@0.2.20
   python:
   - python@3.6.5
-  - python@2.7.15
 ```
 
 The `core` toolchain typically represent system compiler. This compiler is used only to bootstrap/install other compilers and some basic utility packages. And hence it doesn't provide any mpi, lapack or python packages.
 
-The next toolchain `gnu-stable` represent default GNU compiler. We are going to provide two mpi libraries and two python versions. Note that the length of each axis should be same (and hence there are duplicate entries).
+The next toolchain `gnu-stable` represent default GNU compiler. We are going to provide single version of HPE mpi library and python version.
 
 
 #### Packages Specifications
 
-Once compiler toolchains are defined, we can define packages to build for each toolchain as:
+Once compiler toolchains are defined, we can define packages to build for each toolchain. For example, we define compiler packages as:
 
 
 ```
+# list of packages to build
 packages:
   compilers:
     target_matrix:
@@ -132,22 +127,16 @@ packages:
       - compiler
     specs:
       - gcc@8.2.0
-      - gcc@7.3.0
       - gcc@6.4.0
-      - gcc@5.5.0
       - intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2018.3
-      - intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2017.7
       - intel@18.0.3
-      - intel@17.0.7
-      - llvm@6.0.1
-      - llvm@5.0.1
+      - llvm@7.0.0
       - pgi+network+nvidia+mpi@18.4
-      - pgi+network+nvidia+mpi@17.10
 ```
 
 Here is brief summary of what's going on:
 
-* `compilers` here is a just tag
+* `compilers` under packages is just a tag
 * `target_matrix` indicates which compiler toolchain we are going to use (see Toolchain Specifications)
 * `requires` indicates which axis will be used while building the packages
 * `specs` indicates which packages we are going to build
@@ -157,67 +146,41 @@ It would be more clear if we look at the package specs generated with `spackd` c
 To see which packages will be installed, for example, to install all compilers on the system:
 
 ```
-→ spackd --input packages/compiler-packages.yaml packages x86_64
-
+→ spackd --input packages/compilers.yaml packages x86_64
 gcc@8.2.0 %gcc@4.8.5 target=x86_64
-gcc@7.3.0 %gcc@4.8.5 target=x86_64
 gcc@6.4.0 %gcc@4.8.5 target=x86_64
-gcc@5.5.0 %gcc@4.8.5 target=x86_64
 intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2018.3 %gcc@4.8.5 target=x86_64
-intel-parallel-studio+advisor+clck+daal+gdb+inspector+ipp+itac+mkl+mpi+rpath+shared+tbb+vtune@cluster.2017.7 %gcc@4.8.5 target=x86_64
 intel@18.0.3 %gcc@4.8.5 target=x86_64
-intel@17.0.7 %gcc@4.8.5 target=x86_64
-llvm@6.0.1 %gcc@4.8.5 target=x86_64
-llvm@5.0.1 %gcc@4.8.5 target=x86_64
+llvm@7.0.0 %gcc@4.8.5 target=x86_64
 pgi+network+nvidia+mpi@18.4 %gcc@4.8.5 target=x86_64
-pgi+network+nvidia+mpi@17.10 %gcc@4.8.5 target=x86_64
 ```
 
 These specs can be the used to install packages using `spack install` command. This become more useful when multiple compilers and mpi libraries come into picture, for example, to install all parallel libraries on the system :
 
 ```
 → spackd --input packages/parallel-libraries.yaml packages x86_64
-
-hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %clang@6.0.1 target=x86_64
-omega-h@9.17.2 ^hpe-mpi@2.16 %clang@6.0.1 target=x86_64
-...
-hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64
-omega-h@9.17.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64
-...
+hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %intel@18.0.3 target=x86_64
 hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %gcc@6.4.0 target=x86_64
-omega-h@9.17.2 ^hpe-mpi@2.16 %gcc@6.4.0 target=x86_64
-...
-hdf5+mpi@1.10.2 ^mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple %gcc@6.4.0 target=x86_64
-omega-h@9.17.2 ^mvapich2@2.3 process_managers=slurm fabrics=mrail file_systems=gpfs threads=multiple %gcc@6.4.0 target=x86_64
-....
+omega-h@9.17.2 ^openblas@0.2.20 %gcc@6.4.0 target=x86_64
+petsc+int64+mpi@3.9.3 ^openblas@0.2.20 %gcc@6.4.0 target=x86_64
+trilinos@12.12.1 ^openblas@0.2.20 %gcc@6.4.0 target=x86_64
+omega-h@9.17.2 ^intel-mkl@2018.1.163 %intel@18.0.3 target=x86_64
+petsc+int64+mpi@3.9.3 ^intel-mkl@2018.1.163 %intel@18.0.3 target=x86_64
+trilinos@12.12.1 ^intel-mkl@2018.1.163 %intel@18.0.3 target=x86_64
 ```
 
-Similarly, packages for `x86_64_knl` target can be generated with :
-
-```
- → spackd --input packages/parallel-libraries.yaml packages x86_64_knl
-
-hdf5+mpi@1.10.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl
-omega-h@9.17.2 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl
-petsc+int64+mpi@3.9.3 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl
-trilinos+kokkos+teuchos@12.12.1 ^hpe-mpi@2.16 %intel@18.0.1 target=x86_64_knl
-```
-
-The specs become complicated but they are complete considering architecture, compiler and mpi dependecies.
-
-
 #### Whole Software Stack
 
 Here is list of all packages (in order) that we will be installing :
 
 ```
-spackd --input packages/compiler-packages.yaml packages x86_64 --output compiler-packages.txt
-spackd --input packages/build-dep-packages.yaml packages x86_64 --output buiild-dep-packages.txt
+spackd --input packages/compilers.yaml packages x86_64 --output compilers.txt
+spackd --input packages/system-tools.yaml packages x86_64 --output system-tools.txt
 spackd --input packages/serial-libraries.yaml packages x86_64 --output serial-libraries.txt
 spackd --input packages/python-packages.yaml packages x86_64 --output python-packages.txt
 spackd --input packages/parallel-libraries.yaml packages x86_64 --output parallel-libraries.txt
+spackd --input packages/bbp-packages.yaml packages x86_64 --output bbp-packages.txt
 ```
-Similarly for target `x86_64_knl`.
 
 #### Todo : Deployment Workflow
 
@@ -227,7 +190,7 @@ Here is how deployment workflow should look like :
 
 #### Todo : Jenkins Pipeline Workflow
 
-See exisiting scripts in `scripts/` directory and `Jenkinsfile`. Those should be refactored for new workflow.
+See existing scripts in `scripts/` directory and `Jenkinsfile`. Those should be refactored for new workflow.
 
 #### Other Commands
 
@@ -238,7 +201,6 @@ x86_64
 → spackd --input packages/parallel-libraries.yaml compilers x86_64
 clang@6.0.1%gcc@4.8.5 target=x86_64
 gcc@6.4.0%gcc@4.8.5 target=x86_64
-intel@18.0.1%gcc@4.8.5 target=x86_64
-intel@18.0.1-knl%gcc@4.8.5 target=x86_64
+intel@18.0.3%gcc@4.8.5 target=x86_64
 pgi@18.4%gcc@4.8.5 target=x86_64
 ```

check_specs.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+set +e
+source "setup_env.sh"
+
+
+check_specs() {
+    package_list=$1
+    to_be_installed=$(spack filter --not-installed $(cat $package_list))
+
+    if [[ -z "${to_be_installed}" ]]
+    then
+        echo "All specs already installed"
+    else
+        echo "spack spec -Il $to_be_installed"
+        spack spec -Il $to_be_installed
+    fi
+}
+
+
+cd $WORKSPACE/HOME_DIR/spack-deploy
+for category in "${package_categories[@]}"
+do
+    package_list=`basename $category .yaml`.txt
+    echo "Checking installed packages in category $category"
+    check_specs $package_list
+done
+
+
+echo -e "Spec concretisation check finished"

configs/config.yaml

@@ -1,12 +1,11 @@
 config:
-  # Root of the installation tree
-  install_tree: /ssoft/spack/paien/v2/opt/spack/
-
-  # Root of the module files
+  install_tree: $SOFTS_DIR_PATH/install
+  source_cache: $SOFTS_DIR_PATH/.cache
   module_roots:
-    tcl: /ssoft/spack/paien/v2/share/spack/modules
-    lmod: /ssoft/spack/paien/v2/share/spack/lmod
-    dotkit: /ssoft/spack/paien/v2/share/spack/dotkit
-
+    tcl: $SOFTS_DIR_PATH/modules/tcl
+    lmod: $SOFTS_DIR_PATH/modules/lmod
   build_stage:
-    - /tmp/paien.v2
+    - $tempdir
+    - $SOFTS_DIR_PATH/.stage
+  install_path_scheme: '${ARCHITECTURE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH:10}'
+  build_jobs: 36