Skip to content

Breaking changes with PCU #433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 204 commits into from
Dec 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
204 commits
Select commit Hold shift + click to select a range
feafdec
Add state to PCU mpi interface.
jacobmerson Mar 28, 2023
7c390ee
Compile on gcc+fix bug
jacobmerson Mar 28, 2023
a4bbbfb
remove virtual function table for pcu_mpi
jacobmerson Mar 29, 2023
baf9370
Put PCU state data into a structure
jacobmerson Mar 29, 2023
4c243cb
Use consistent initialization functions for MPI.
jacobmerson Mar 29, 2023
d891d2b
Add PCU2 interface that takes the PCU state struct
jacobmerson Mar 29, 2023
de2f57d
add PCU2.h to install files
jacobmerson Mar 29, 2023
6a46101
fix compiler warnings
jacobmerson Mar 29, 2023
ed28c05
constify mpi functions
jacobmerson Mar 30, 2023
2eada4e
first pass of PCU with making use of a high level object
jacobmerson Mar 30, 2023
eed5a0c
remove unused functions
jacobmerson Mar 30, 2023
f0b3a28
Fix gcc compilation
jacobmerson Mar 30, 2023
0e7362f
fix bug
jacobmerson Mar 31, 2023
0e0c80e
remove unnecessary whitespace
jacobmerson Mar 31, 2023
07ab364
bump version
jacobmerson Mar 31, 2023
80c829d
Add state to PCU mpi interface.
jacobmerson Mar 28, 2023
2f6238c
Compile on gcc+fix bug
jacobmerson Mar 28, 2023
a5be7ae
remove virtual function table for pcu_mpi
jacobmerson Mar 29, 2023
064ea13
Put PCU state data into a structure
jacobmerson Mar 29, 2023
e727d54
Use consistent initialization functions for MPI.
jacobmerson Mar 29, 2023
47bf6d5
Add PCU2 interface that takes the PCU state struct
jacobmerson Mar 29, 2023
ef021f6
add PCU2.h to install files
jacobmerson Mar 29, 2023
270d44e
fix compiler warnings
jacobmerson Mar 29, 2023
62bfb69
constify mpi functions
jacobmerson Mar 30, 2023
173c7ec
first pass of PCU with making use of a high level object
jacobmerson Mar 30, 2023
23c7cef
remove unused functions
jacobmerson Mar 30, 2023
438a73e
Fix gcc compilation
jacobmerson Mar 30, 2023
4fa6e37
fix bug
jacobmerson Mar 31, 2023
716133e
remove unnecessary whitespace
jacobmerson Mar 31, 2023
4d281e0
bump version
jacobmerson Mar 31, 2023
625b02b
Merge pull request #1 from flagdanger/pcu-object
jacobmerson Jan 11, 2024
412addc
added pcu_ as a variable of Mesh
flagdanger Jan 11, 2024
63bf279
Added getPCU(), and beginning to change pcu functions
flagdanger Jan 11, 2024
b364118
Can access global_pcu and some editing of function calls in /apf
flagdanger Jan 16, 2024
1010eb1
Changed functions in more files up to folder ma
flagdanger Jan 16, 2024
041b6f0
More PCU changes
flagdanger Jan 17, 2024
435172e
More fixing
flagdanger Jan 23, 2024
3f40db9
ph and pumi changes
flagdanger Jan 24, 2024
b1e41ff
more cleaning
flagdanger Jan 25, 2024
a2cb48f
test folder fixes
flagdanger Jan 25, 2024
da4383d
More test folder fixes
flagdanger Jan 25, 2024
3c7b447
fixed last of test folder and made PCU2 .h and .cc
flagdanger Jan 26, 2024
2ec87e3
Re-checked apf and pcu2 fix
flagdanger Jan 30, 2024
3101beb
re-checking up to ma
flagdanger Jan 30, 2024
ee1d7e4
ma folder redo
flagdanger Jan 30, 2024
f13132e
more fixes
flagdanger Feb 5, 2024
33a6621
Making new versions of functions to handle PCUHandle
flagdanger Feb 5, 2024
8c5633a
phCook changes, changed phInput openfile_read to take PCUHandle, and …
flagdanger Feb 7, 2024
eaeaa3d
Reverting phasta changes
flagdanger Feb 8, 2024
244a6d5
mds folder changed to use PCU2
flagdanger Feb 9, 2024
5206e77
Last mds fixes (apfMDS.cc still has some issues)
flagdanger Feb 12, 2024
8e2cdb0
Getting rid of include PCU.h in spots I missed
flagdanger Feb 12, 2024
221a982
All parma changes I can do (lots of files need some workaround)
flagdanger Feb 12, 2024
344e3c4
small pcu change
flagdanger Feb 12, 2024
5ee39e5
pcu_getglobal changes
flagdanger Feb 12, 2024
cb971df
apfCap fix
flagdanger Feb 12, 2024
2bb8171
crv fixes
flagdanger Feb 12, 2024
23bf889
dsp pcu fix
flagdanger Feb 12, 2024
60e0b6f
gmi cap and sim pcu fixes
flagdanger Feb 12, 2024
63af137
ma pcu fixes
flagdanger Feb 12, 2024
98e9d71
changing apfMDS.cc back to original, still needs work
flagdanger Feb 12, 2024
6f49fb2
pumi pcu fixes
flagdanger Feb 12, 2024
14d2e5e
spr pcu fix
flagdanger Feb 12, 2024
5f17c05
stk pcu fixes
flagdanger Feb 12, 2024
bc1ce6a
changing repartition.cc back to original, still needs work
flagdanger Feb 12, 2024
59d5a7f
zoltan pcu fixes
flagdanger Feb 14, 2024
98341a3
pumi interface fixes, replacing old PCU uses
flagdanger Mar 11, 2024
a0c53fc
files in test folder changed to use new pumi - pcu interface
flagdanger Mar 11, 2024
32ab95b
made default PCU arguments for Mesh functions and fixed old pcu issue…
flagdanger Mar 12, 2024
530a82e
repartition test file rebuilt to use new pcu
flagdanger Mar 12, 2024
fda3f51
apf folder using new pcu except apfNumbering.cc
flagdanger Mar 14, 2024
bdce405
ma folder changed to use new pcu, changed some uses of print() in crv…
flagdanger Mar 15, 2024
9075f4b
ma folder using new pcu, all tests passing
flagdanger Mar 18, 2024
efc2d77
changed phastaio_initStats to not check if PCU is intitialized. Also …
flagdanger Mar 20, 2024
d424963
Phasta folder changed to use new PCU, passes tests.
flagdanger Mar 25, 2024
9005de8
small touchups in test files, apfMDS.cc, and apf_sim
flagdanger Mar 25, 2024
6bb2806
Parma folder fixes, only thing not changed is parma_group.cc, all tes…
flagdanger Mar 25, 2024
6cbd11d
added switchPCU to apfMesh to use in parma_group.cc
flagdanger Mar 27, 2024
000418e
parma_group.cc fixed to use switchPCU, may need to change later
flagdanger Mar 28, 2024
bd265d8
parallelReduce function fix
flagdanger Mar 28, 2024
7f83066
crv corrected makeEmptyMdsMesh use
flagdanger Apr 1, 2024
8469c37
ma corrected makeEmptyMdsMesh use
flagdanger Apr 1, 2024
2b90da5
fixed BoxBuilder class and loadMdsFromANSYS to take an optional PCUObj
flagdanger Apr 1, 2024
961f09a
got rid of undef PCU_FORMAT_ATTRIBUTE, fixed compilation errors that …
flagdanger Apr 1, 2024
daed911
phasta - fixed createMdsMesh use
flagdanger Apr 1, 2024
946ba7c
got rid of an #include PCU.h
flagdanger Apr 1, 2024
8c28b15
#include PCU.h deletion
flagdanger Apr 1, 2024
7a99da1
spr folder fixes
flagdanger Apr 1, 2024
99ad2db
1d test file
flagdanger Apr 1, 2024
77cb68f
align test file
flagdanger Apr 1, 2024
1dc3a94
aniso_ma_test file
flagdanger Apr 1, 2024
becd83a
ansys test file
flagdanger Apr 1, 2024
e591d9a
assert_timing test file
flagdanger Apr 1, 2024
dc48ea7
balance test file
flagdanger Apr 1, 2024
4624408
bezierElevation test file
flagdanger Apr 1, 2024
0a7ccec
box test file
flagdanger Apr 1, 2024
54cc6be
collapse test file
flagdanger Apr 1, 2024
f6e5d7c
poisson test file
flagdanger Apr 1, 2024
5f0ef7e
mdsCGNS PCUHandle fix
flagdanger Apr 9, 2024
6452f43
maximal independent set fixes
flagdanger Apr 9, 2024
8cb06bb
Parma_GroupCode run() function default changed to take a PCUObj
flagdanger Apr 9, 2024
f36021a
Test files fixed
flagdanger Apr 9, 2024
0db8e21
loadMdsPart take optional PCU
flagdanger Apr 17, 2024
6f4c6df
loadMdsFromGmsh to take PCU
flagdanger Apr 17, 2024
9900b46
fixed runInGroups() to work with new PCU
flagdanger Apr 17, 2024
b0a6ff2
reverted the run function in GroupCode structs to not take a PCU
flagdanger Apr 17, 2024
52c9ab8
test files fixed to use PCUObj; all pass
flagdanger Apr 17, 2024
1eebc1e
Mesh interface fixes
flagdanger Apr 26, 2024
94dcc92
Test files changed to use new PCU
flagdanger Apr 26, 2024
7dbae41
Added comments for uses of switchPCU
flagdanger Apr 29, 2024
2b03519
added back reeResidualFunctionals.cc
flagdanger Apr 29, 2024
00bf25e
Delete config.sh
flagdanger Apr 29, 2024
2f23fb1
Merge pull request #2 from flagdanger/pcu-object
jacobmerson Apr 29, 2024
03eb93d
Merge branch 'develop' into pcu-object
jacobmerson Apr 29, 2024
ea3b36e
Warning fixes
flagdanger Apr 29, 2024
b2ae31a
Merge branch 'pcu-object' of https://github.com/flagdanger/core into …
flagdanger Apr 29, 2024
d95cdcd
Merge pull request #3 from flagdanger/pcu-object
jacobmerson Apr 30, 2024
769c432
compiler errors with phLinks.cc fixed
flagdanger May 2, 2024
f96b764
.github fix
flagdanger May 6, 2024
1e3d684
describe.cc getPeak() and getChunks() removed
flagdanger May 6, 2024
e33b48d
removed last uses of old pcu, removed pcu.cc and .h, got rid of defau…
flagdanger Jun 3, 2024
0dd10d1
small fixes
flagdanger Jul 1, 2024
6e9a875
PCU2 using PCU naming, changed PCUHandle to PCU_t
flagdanger Jul 5, 2024
deba052
pumi_sys functions using pumi::instance to access PCU, added setPCU a…
flagdanger Jul 9, 2024
687e160
added pumi_load_pcu to replace the use of setPCU, removed PCUObj as a…
flagdanger Jul 9, 2024
7d57abc
apfSim typo fix
flagdanger Jul 31, 2024
e90602f
pumi_rank() replacements
flagdanger Jul 31, 2024
55f887c
simmetrix create mesh typo fixed
flagdanger Aug 6, 2024
bfde63e
Merge remote-tracking branch 'SCOREC/develop' into pcu-object-breaking
flagdanger Aug 6, 2024
a904f02
phLinks ptr issue fix
flagdanger Aug 7, 2024
77812e5
phasta convert fixes
flagdanger Aug 7, 2024
a907b7e
test files small bugs
flagdanger Aug 8, 2024
8128e2a
Revert "test files small bugs"
flagdanger Aug 8, 2024
94549c7
removed config file
flagdanger Aug 8, 2024
73c5033
PCU.h small fix
flagdanger Aug 16, 2024
90f9ff3
Delete config.sh
flagdanger Aug 16, 2024
236bffb
long dereference chains fixed
flagdanger Aug 19, 2024
7025e62
PCUObj renamed to PCU, PCU remaned to PCU_C
flagdanger Aug 19, 2024
90785ed
Specified argument data type for templated functions PCU:: Add, Max, …
flagdanger Aug 19, 2024
6faa765
Capstone updated to use PCU Object
flagdanger Aug 19, 2024
b5a2190
small fixes
flagdanger Aug 21, 2024
85be809
Typo cleaning
flagdanger Aug 26, 2024
d5b7413
Test files changed to not use unique_ptr
flagdanger Aug 26, 2024
d666a1e
Removed uses of unique_ptr in other folders besides test
flagdanger Aug 27, 2024
807746c
Removed semi colon after Macro
flagdanger Aug 29, 2024
7e4cd5a
CGNS fixes
flagdanger Sep 23, 2024
6e47c85
cgns fixes
flagdanger Oct 10, 2024
1b72d7c
unique ptr removed
flagdanger Nov 1, 2024
823ba34
Update apfElement.cc
jacobmerson Jul 28, 2024
ccc67e9
spr: add thesis section to ref
cwsmith Aug 8, 2024
7f1a90e
clean under asan sanitizer with gcc 12.3.0
cwsmith Aug 24, 2024
821e359
add sim+cgns build
cwsmith Aug 25, 2024
81cf783
formatting
cwsmith Aug 25, 2024
e1e1db9
move nightly test+merge to rhel9
cwsmith Aug 25, 2024
96be949
user name change
cwsmith Aug 25, 2024
7d58e62
add approved self-hosted user
cwsmith Aug 28, 2024
65d8b61
missed a paren
cwsmith Aug 28, 2024
f8c064b
need module command
cwsmith Aug 28, 2024
ddd4312
missed the other env
cwsmith Aug 28, 2024
2e26181
append to log file
cwsmith Aug 28, 2024
77a7c6e
heredoc for multiline message
cwsmith Aug 30, 2024
cdca4be
complete user cmake example
cwsmith Sep 6, 2024
4f64750
install and build user project
cwsmith Sep 6, 2024
450a0ad
set install path
cwsmith Sep 6, 2024
edc6f65
fix typo
cwsmith Sep 6, 2024
cdfdfb9
fix ci conditional
cwsmith Sep 6, 2024
307e1e1
compiler comparison attempt
cwsmith Sep 7, 2024
e6d7d01
update checkout actions
cwsmith Sep 7, 2024
dcfd9b1
name is a field of the compiler entry
cwsmith Sep 7, 2024
1e62845
Set bob install dir to CMAKE_LIBDIR
bobpaw Aug 29, 2024
d06878e
Replace other lib with LIBDIR and bin with BINDIR
bobpaw Sep 6, 2024
000665c
Replace include with CMAKE_INSTALL_INCLUDEDIR
bobpaw Sep 6, 2024
81d60d3
Fix logic checking periodic model faces
joshia5 Aug 9, 2024
57227a9
spell check
joshia5 Aug 9, 2024
9be819d
Add apf::smoothCAPAnisoSizes
bobpaw Sep 20, 2024
792c05b
Add apf::smoothCAPAnisoSizes compile flags
bobpaw Sep 20, 2024
fc3fb1e
Add feature test for sizing metric tool
bobpaw Sep 22, 2024
1fb3c89
Add apf::smoothCAPAnisoSizes compilation test
bobpaw Sep 22, 2024
680a904
fix: Move function call outside of debug assert
bobpaw Sep 25, 2024
5b627ed
Clarify that orthogonal basis frames are columns (#459)
bobpaw Oct 8, 2024
d2203f0
Implement capstone gmi functions
bobpaw Aug 29, 2024
a151bea
Remove FIXME and replace with comment
bobpaw Sep 10, 2024
2bf983e
Add gmi_cap isInClosureOf test
bobpaw Sep 23, 2024
dbdd0a1
Add test/cap_closestPoint
bobpaw Oct 3, 2024
5bf857b
Fix smoothCAPAnisoSizes eigenvalues and transpose
bobpaw Oct 7, 2024
e7b5fd2
fix: Correct smoothCAPAnisoSizes param order
bobpaw Oct 10, 2024
f013608
increase versions for simmodsuite 2025.0.241016
cwsmith Oct 21, 2024
82b5b59
simplify and improve logic for setting C++ standard
cwsmith Oct 21, 2024
eff7b4e
bob_set_cxx_standard should only be called once
cwsmith Oct 22, 2024
ddb0740
copy inputs needed for smoke tests to repo
cwsmith Oct 25, 2024
bdc2c1c
Moved capStone2VTK to test/cap2vtk
bobpaw Aug 29, 2024
8a3bb4d
Remove exe sources which are not in CMake
bobpaw Aug 31, 2024
ad7cf77
Move some capstone_clis to test
bobpaw Sep 5, 2024
58b830a
Remove capStoneAnisoAdaptWing
bobpaw Sep 5, 2024
64f435d
Move capStoneSizeFields to test
bobpaw Sep 6, 2024
9f537e4
Remove capStoneAttachSolution.cc
bobpaw Oct 30, 2024
1160150
fix capGeomTest.cc issue
flagdanger Nov 20, 2024
7b20e57
doc folder guide changed to use new PCU
flagdanger Nov 20, 2024
5512298
PCU.cc documentation
flagdanger Nov 22, 2024
3f6b3d4
Remove exe sources which are not in CMake
bobpaw Aug 31, 2024
c99b709
generate: fix mem leaks of simmodsuite objects
cwsmith Nov 1, 2024
42dd756
Fix unused variable warning introduced by #446 (#453)
bobpaw Nov 4, 2024
0d3aaf9
test globus (#466) - refactor globus compute scripts
Angelyr Nov 11, 2024
7e54d7f
Merge branch 'develop' into pcu-object-breaking
cwsmith Dec 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ endif()
# This is the top level CMake file for the SCOREC build
cmake_minimum_required(VERSION 3.8)

project(SCOREC VERSION 2.2.10 LANGUAGES CXX C)
project(SCOREC VERSION 2.3.0 LANGUAGES CXX C)

include(cmake/bob.cmake)
include(cmake/xsdk.cmake)
Expand Down
2 changes: 1 addition & 1 deletion Doxyfile.in
Original file line number Diff line number Diff line change
Expand Up @@ -793,7 +793,7 @@ INPUT = \
@CMAKE_CURRENT_SOURCE_DIR@/gmi/gmi_mesh.h \
@CMAKE_CURRENT_SOURCE_DIR@/gmi/gmi_analytic.h \
@CMAKE_CURRENT_SOURCE_DIR@/gmi/gmi_null.h \
@CMAKE_CURRENT_SOURCE_DIR@/pcu/pcu.c \
@CMAKE_CURRENT_SOURCE_DIR@/pcu/PCU.cc \
@CMAKE_CURRENT_SOURCE_DIR@/mth/mth.dox \
@CMAKE_CURRENT_SOURCE_DIR@/mth/mth.h \
@CMAKE_CURRENT_SOURCE_DIR@/mth/mthVector.h \
Expand Down
2 changes: 1 addition & 1 deletion Doxyfile_internal.in
Original file line number Diff line number Diff line change
Expand Up @@ -826,7 +826,7 @@ INPUT = \
@CMAKE_CURRENT_SOURCE_DIR@/gmi/gmi_mesh.h \
@CMAKE_CURRENT_SOURCE_DIR@/gmi/gmi_analytic.h \
@CMAKE_CURRENT_SOURCE_DIR@/gmi/gmi_null.h \
@CMAKE_CURRENT_SOURCE_DIR@/pcu/pcu.c \
@CMAKE_CURRENT_SOURCE_DIR@/pcu/PCU.cc \
@CMAKE_CURRENT_SOURCE_DIR@/mth/mth.dox \
@CMAKE_CURRENT_SOURCE_DIR@/mth/mth.h \
@CMAKE_CURRENT_SOURCE_DIR@/mth/mthVector.h \
Expand Down
6 changes: 5 additions & 1 deletion apf/apf.h
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@
#include "apfNew.h"
#include "apfDynamicArray.h"

namespace pcu{
class PCU;
}

#include <vector>
#include <map>
#include <limits>
Expand Down Expand Up @@ -460,7 +464,7 @@ class Integrator
* process-local integrations into a global mesh integration,
* if that is the user's goal.
*/
virtual void parallelReduce();
virtual void parallelReduce(pcu::PCU*);
protected:
int order;
int ipnode;
Expand Down
50 changes: 24 additions & 26 deletions apf/apfCGNS.cc
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,13 @@
* BSD license as described in the LICENSE file in the top-level directory.
*/

#include <PCU.h>
#include "apfMesh.h"
#include "apfNumbering.h"
#include "apfNumberingClass.h"
#include "apfShape.h"
#include "apfFieldData.h"
#include <pcu_util.h>
#include <lionPrint.h>
//
#include <sstream>
#include <fstream>
#include <iostream>
Expand Down Expand Up @@ -47,7 +45,7 @@ static Count count(apf::Mesh *m, int dim)
{
const int local = apf::countOwned(m, dim);
int total = local;
PCU_Add_Ints(&total, 1); // size of total array
m->getPCU()->Add<int>(&total, 1); // size of total array
return std::make_pair(total, local);
}

Expand Down Expand Up @@ -290,7 +288,7 @@ void WriteFields(const CGNS &cgns, const std::vector<std::vector<apf::MeshEntity
}

int size = data.size();
PCU_Add_Ints(&size, 1); // size of total array
m->getPCU()->Add<int>(&size, 1); // size of total array

// oddness of the api
rmin[1] = rmin[0];
Expand Down Expand Up @@ -505,7 +503,7 @@ CellElementReturn WriteElements(const CGNS &cgns, apf::Mesh *m, apf::GlobalNumbe
m->end(cellIter);
numbersByElementType[o] = counter;
int total = counter;
PCU_Add_Ints(&total, 1); // size of total array
m->getPCU()->Add<int>(&total, 1); // size of total array
globalNumbersByElementType[o] = total;
}
cgsize_t allTotal = std::accumulate(globalNumbersByElementType.begin(), globalNumbersByElementType.end(), 0);
Expand Down Expand Up @@ -546,12 +544,12 @@ CellElementReturn WriteElements(const CGNS &cgns, apf::Mesh *m, apf::GlobalNumbe
if (cgp_section_write(cgns.index, cgns.base, cgns.zone, name.c_str(), cgnsElementOrder[o], globalStart, globalEnd, 0, &sectionNumber)) // global start, end within the file for that element type
cgp_error_exit();

std::vector<int> allNumbersForThisType(PCU_Comm_Peers(), 0);
std::vector<int> allNumbersForThisType(m->getPCU()->Peers(), 0);
MPI_Allgather(&numbersByElementType[o], 1, MPI_INT, allNumbersForThisType.data(), 1,
MPI_INT, PCU_Get_Comm());
MPI_INT, m->getPCU()->GetMPIComm());

cgsize_t num = 0;
for (int i = 0; i < PCU_Comm_Self(); i++)
for (int i = 0; i < m->getPCU()->Self(); i++)
num += allNumbersForThisType[i];

cgsize_t elStart = globalStart + num;
Expand Down Expand Up @@ -635,7 +633,7 @@ void AddBocosToMainBase(const CGNS &cgns, const CellElementReturn &cellResults,
int startOfBCBlock = startingLocation + 1;
const int number = bc.second.size();
int total = number;
PCU_Add_Ints(&total, 1); // size of total array
m->getPCU()->Add<int>(&total, 1); // size of total array
if (total > 0)
{
const auto allEnd = startOfBCBlock + total - 1; //one-based
Expand All @@ -657,12 +655,12 @@ void AddBocosToMainBase(const CGNS &cgns, const CellElementReturn &cellResults,
}
}

std::vector<int> allNumbersForThisType(PCU_Comm_Peers(), 0);
std::vector<int> allNumbersForThisType(m->getPCU()->Peers(), 0);
MPI_Allgather(&number, 1, MPI_INT, allNumbersForThisType.data(), 1,
MPI_INT, PCU_Get_Comm());
MPI_INT, m->getPCU()->GetMPIComm());

cgsize_t num = 0;
for (int i = 0; i < PCU_Comm_Self(); i++)
for (int i = 0; i < m->getPCU()->Self(); i++)
num += allNumbersForThisType[i];

cgsize_t elStart = startOfBCBlock + num;
Expand All @@ -685,34 +683,34 @@ void AddBocosToMainBase(const CGNS &cgns, const CellElementReturn &cellResults,
cacheEnd = allEnd;
}
}
std::vector<int> cacheStarts(PCU_Comm_Peers(), 0);
std::vector<int> cacheStarts(m->getPCU()->Peers(), 0);
MPI_Allgather(&cacheStart, 1, MPI_INT, cacheStarts.data(), 1,
MPI_INT, PCU_Get_Comm());
std::vector<int> cacheEnds(PCU_Comm_Peers(), 0);
MPI_INT, m->getPCU()->GetMPIComm());
std::vector<int> cacheEnds(m->getPCU()->Peers(), 0);
MPI_Allgather(&cacheEnd, 1, MPI_INT, cacheEnds.data(), 1,
MPI_INT, PCU_Get_Comm());
MPI_INT, m->getPCU()->GetMPIComm());
return std::make_pair(cacheStarts, cacheEnds);
};

const auto globalElementList = [](const std::vector<cgsize_t> &bcList, std::vector<cgsize_t> &allElements) {
std::vector<int> sizes(PCU_Comm_Peers(), 0); // important initialiser
const auto globalElementList = [&m](const std::vector<cgsize_t> &bcList, std::vector<cgsize_t> &allElements) {
std::vector<int> sizes(m->getPCU()->Peers(), 0); // important initialiser
const int l = bcList.size();
MPI_Allgather(&l, 1, MPI_INT, sizes.data(), 1,
MPI_INT, PCU_Get_Comm());
MPI_INT, m->getPCU()->GetMPIComm());

int totalLength = 0;
for (const auto &i : sizes)
totalLength += i;

std::vector<int> displacement(PCU_Comm_Peers(), -1);
std::vector<int> displacement(m->getPCU()->Peers(), -1);
displacement[0] = 0;
for (std::size_t i = 1; i < displacement.size(); i++)
displacement[i] = displacement[i - 1] + sizes[i - 1];

allElements.resize(totalLength);
MPI_Allgatherv(bcList.data(), bcList.size(), MPI_INT, allElements.data(),
sizes.data(), displacement.data(), MPI_INT,
PCU_Get_Comm());
m->getPCU()->GetMPIComm());
};

const auto doVertexBC = [&](const auto &iter) {
Expand Down Expand Up @@ -755,7 +753,7 @@ void AddBocosToMainBase(const CGNS &cgns, const CellElementReturn &cellResults,
for (const auto &p : iter->second)
{
const auto se = BCEntityAdder(EdgeLoop, p, startingLocation);
for (int i = 0; i < PCU_Comm_Peers(); i++)
for (int i = 0; i < m->getPCU()->Peers(); i++)
{
PCU_ALWAYS_ASSERT_VERBOSE(se.first[i] == se.first[0], "Must all be the same ");
PCU_ALWAYS_ASSERT_VERBOSE(se.second[i] == se.second[0], "Must all be the same ");
Expand All @@ -779,7 +777,7 @@ void AddBocosToMainBase(const CGNS &cgns, const CellElementReturn &cellResults,
{
const auto se = BCEntityAdder(FaceLoop, p, startingLocation);

for (int i = 0; i < PCU_Comm_Peers(); i++)
for (int i = 0; i < m->getPCU()->Peers(); i++)
{
PCU_ALWAYS_ASSERT_VERBOSE(se.first[i] == se.first[0], "Must all be the same ");
PCU_ALWAYS_ASSERT_VERBOSE(se.second[i] == se.second[0], "Must all be the same ");
Expand Down Expand Up @@ -1011,7 +1009,7 @@ void WriteCGNS(const char *prefix, apf::Mesh *m, const apf::CGNSBCMap &cgnsBCMap
{
static_assert(std::is_same<cgsize_t, int>::value, "cgsize_t not compiled as int");

const auto myRank = PCU_Comm_Self();
const auto myRank = m->getPCU()->Self();
const Count vertexCount = count(m, 0);
const Count edgeCount = count(m, 1);
const Count faceCount = count(m, 2);
Expand All @@ -1031,7 +1029,7 @@ void WriteCGNS(const char *prefix, apf::Mesh *m, const apf::CGNSBCMap &cgnsBCMap
// PCU_Barrier();
// }

PCU_Barrier();
m->getPCU()->Barrier();
if (myRank == 0)
{
std::cout << "*******Global Mesh Stats*****************\n";
Expand All @@ -1048,7 +1046,7 @@ void WriteCGNS(const char *prefix, apf::Mesh *m, const apf::CGNSBCMap &cgnsBCMap
sizes[2] = 0; // nodes are unsorted, as defined by api

// Copy communicator
auto communicator = PCU_Get_Comm();
auto communicator = m->getPCU()->GetMPIComm();
cgp_mpi_comm(communicator);
//
cgp_pio_mode(CGP_INDEPENDENT);
Expand Down
20 changes: 9 additions & 11 deletions apf/apfCavityOp.cc
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@
* BSD license as described in the LICENSE file in the top-level directory.
*/

#include <PCU.h>
#include "apfCavityOp.h"
#include "apf.h"
#include "apfMesh2.h"
Expand Down Expand Up @@ -134,10 +133,10 @@ bool CavityOp::requestLocality(MeshEntity** entities, int count)

bool CavityOp::sendPullRequests(std::vector<PullRequest>& received)
{
int done = PCU_Min_Int(requests.empty());
int done = mesh->getPCU()->Min<int>(requests.empty());
if (done) return false;
/* throw in the local pull requests */
int self = PCU_Comm_Self();
int self = mesh->getPCU()->Self();
received.reserve(requests.size());
APF_ITERATE(Requests,requests,it)
{
Expand All @@ -147,7 +146,7 @@ bool CavityOp::sendPullRequests(std::vector<PullRequest>& received)
received.push_back(request);
}
/* now communicate the rest */
PCU_Comm_Begin();
mesh->getPCU()->Begin();
APF_ITERATE(Requests,requests,it)
{
CopyArray remotes;
Expand All @@ -156,18 +155,17 @@ bool CavityOp::sendPullRequests(std::vector<PullRequest>& received)
{
int remotePart = rit->peer;
MeshEntity* remoteEntity = rit->entity;
PCU_COMM_PACK(remotePart,remoteEntity);
}
mesh->getPCU()->Pack(remotePart,remoteEntity); }
}
requests.clear();
PCU_Comm_Send();
while (PCU_Comm_Listen())
mesh->getPCU()->Send();
while (mesh->getPCU()->Listen())
{
PullRequest request;
request.to = PCU_Comm_Sender();
while ( ! PCU_Comm_Unpacked())
request.to = mesh->getPCU()->Sender();
while ( ! mesh->getPCU()->Unpacked())
{
PCU_COMM_UNPACK(request.e);
mesh->getPCU()->Unpack(request.e);
received.push_back(request);
}
}
Expand Down
Loading
Loading