Support for Elemental built against system MPI

[bluehope]

Is there any plan to support MVAPICH2 & intel MPI?

Intel MPI (which has binary compatibility with MVAPICH2) is also one of widely used MPI implementation.
It would be nice to Elemental to support MVAPICH2!

[poulson]

Elemental absolutely supports every modern MPI implementation. I assume that you mean Elemental.jl?

[bluehope]

@poulson Oh, Yes. I meant "Elemental.jl". Thank you for the correction.

[andreasnoack]

For some reason, I'd unwatched my package here so I've only seen this issue now. Soon, we'll change MPI.jl to use the C-API and also hard code various MPI implementations. When that PR is merged, I'll probably delete the MPI functions here and add MPI.jl as a dependency. Adding support for MVAPICH would then be a matter of adding support in MPI.jl and I guess it is basically a matter of copying the definitions from MPICH. Feel free to open a PR to speed up the process.

[ViralBShah]

We no longer build the sources as part of the package installation like we used to.

So the only options are - use what ships with BinaryBuilder, or provide your own custom build. Note that while MPI.jl allows system MPI to be used, Elemental.jl needs an update to allow a system build (that lets it opt out of the BB provided binaries).

[JBlaschke]

Hi, is there a way we can speed up this update? I volunteer my time. At NERSC we need to build against the system MPI, so I am happy to help out if this means we can deploy Elemental.jl on our systems sooner.

I am still a bit new to BB, so can someone give be some guidance how I can "opt out of" BB provided binaries?

[JBlaschke]

Btw @ViralBShah in Julia 1.6.0 we get

Warning: Error requiring `MPICH_jll` from `MPI`
  exception =
   MPICH_jll cannot be loaded: MPI.jl is configured to use the system MPI library

As a workaround I tried:

] dev --local Elemental
] dev --local Elemental_jll
] dev --loca MPICH_jll

but that doesn't fix it. TBH, I can only see one place where MPICH_jll is needed -- here: https://github.com/JuliaParallel/Elemental.jl/blob/83089155659739fea1aae476c6fd492b1ee20850/test/runtests.jl

Then again, I don't know much about BinaryBuilder but it looks like Requires goes through the package specs, and throws this warning. What is not clear to me is how I can tell Requires that not using the system MPI is not an option. What do you think?

[JBlaschke]

Follow-up: is there a way to drop the MPICH_jll requirement in https://github.com/JuliaBinaryWrappers/Elemental_jll.jl ?

[andreasnoack]

MPI.jl has a mechanism that allows for using a system MPI instead of the BB provided MPI. However, it's really not clear to me how that can work here unless we reintroduce the code for building Elemental as part of this package. We could try to mimic the MPI.jl code and link against a system provided libelemental but, historically, it was important to keep a tight connection between the version of the wrappers here and the version of libelemental since the API was evolving.

If you already have a build of libelemental that links against your MPI then you can try to remove Elemental_jll and just point the libEl variable to that libEl.so (or what it's called, I don't recall).

[JBlaschke]

@andreasnoack if you can share with me the BB code that was used to generate Elemental_jll (the build_tarbals.jl? As I said, I'm new to this) then I could take a stab at a locally build one. I think we can't get around an Elemental_jll because Elemental.jl references it in its source: using Elemental_jll: libEl. My strategy at NERSC would then be to provide our own Elemental_jll in our admin repo.

[andreasnoack]

It's here, https://github.com/JuliaPackaging/Yggdrasil/blob/master/E/Elemental/build_tarballs.jl, but I don't see why it would be easier to modify Elemental_jll instead of just Elemental.jl.

[JBlaschke]

Thanks @andreasnoack I'll take a stab at this and let you know. This is easier because I don't need elemental. Several of our users do. So I could explain to each of our users how to modify Elemental.jl. Or I could build our own NERSC-specific Elemental_jll which would live in our global admin depot (alongside of our MPI.jl). If I do things right, this should be automatically picked up whenever a user installs Elemental.jl, without them needing to fiddle with the source code (or build their own libEl).

[JBlaschke]

Another question @andreasnoack -- why do you use the deprecated elemental repo instead of https://github.com/LLNL/Elemental ? The LLNL version comes with CUDA support.

[ViralBShah]

I think the last time I checked, I could not get the build to work in the new repo. We are not yet ready to enable CUDA support in BinaryBuilder, because we don't have infrastructure to distribute CUDA-built binaries. @maleadt may be able to say when we can expect that.

In the meanwhile, we can certainly switch to the new upstream repo for building Elemental_jll. @would you be able to submit a PR?