Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
876b56e
Add shapefile format option to various incarnations of file format fl…
oehmke Feb 27, 2023
04fd53e
Add initial simple unit test for mesh create from shapefile.
oehmke Mar 2, 2023
b50ef16
Since using shapefile will mean not all Mesh I/O is PIO, move PIO dow…
oehmke Mar 13, 2023
5210dc7
Add skeleton method for creating a mesh from a shapefile. Also, fix i…
oehmke Mar 14, 2023
eb58045
Add files to contain shapefile mesh information read methods.
oehmke Mar 15, 2023
792f975
Add ESMF_SHAPEFILE environment variables for building with shapefile …
oehmke Mar 15, 2023
f834e4d
OK. Built routine infrastructure.
msl3v May 31, 2023
2b9b599
OK. Basic GDAL/OGR shapefile open-and-read routine is in place
msl3v Jun 5, 2023
0a1f831
This version will process US county files. (This was a surprise!)
msl3v Jun 13, 2023
aaa0cd1
End initial parallel code developement. Transition to data/field
msl3v Jun 15, 2023
4a8734f
Safety commit while things are in major flux. Working on parallel
msl3v Jul 11, 2023
33af210
Compiles. Defined ESMF_IOFMT_SHP. Next: GDAL_Handler class
msl3v Jul 17, 2023
83dccf0
Added GDAL_Handler class. Compiles
msl3v Jul 17, 2023
981bb97
Using the IO GDAL multitile test. File opens and info is queried
msl3v Sep 7, 2023
0410e27
Revert "Added GDAL_Handler class. Compiles"
msl3v Sep 7, 2023
10f6ee5
PIO now compiles with GDAL libs and headers. Added pioc_gdal.c module
msl3v Sep 14, 2023
bf3670b
Continuted development. To the point where we want to start reading
msl3v Sep 27, 2023
7fea943
Quick cleanup
msl3v Oct 11, 2023
248ab02
Revision to make multipolygons be a single element in ESMF.
msl3v Oct 12, 2023
76bf40d
Successful ESMF_FieldRead() of a variable in a shapefile, onto
msl3v Oct 13, 2023
bae0cba
OK. We can create a file, open it, add layer & field. Not yet
msl3v Nov 2, 2023
659d603
Stopping here to wait for a proper MeshWrite() capability from on high
msl3v Nov 24, 2023
7f04294
This should be the up-to-date Mesh Create & Read capability.
msl3v Nov 28, 2023
beacbd4
Merge branch 'msl/shapefile_io_v1.0.1' into msl/shapefile_io_v1.1.1
msl3v Dec 5, 2023
4a492cf
Fixed some incomplete merges.
msl3v Dec 5, 2023
34b551e
OK. Brought NCAR/ParallelIO into ESMF as a git subrepo.
msl3v Jan 4, 2024
0eee0f0
Upgraded PIO & determined that OpenMPI v4.1.8 & 4.1.9 was a problem.
msl3v Jan 30, 2024
a487c95
OK. Trying to figure out why a mesh generated from a shapefile gets
msl3v Apr 3, 2024
ddbd9be
Deubgging the mesh garble after call of ESMF_RegridStore()
msl3v May 14, 2024
b7b47da
Put PIO shapefile routine into pio.h to meet C99 standards
msl3v May 20, 2024
4e1c88d
Made it possible to change coord systems of the mesh thru
msl3v May 24, 2024
2dfe747
Debugging coordSys argument for ESMF_MeshCreate() using SHAPEFILE
msl3v May 24, 2024
29fd174
More fixes ...
msl3v May 24, 2024
ac82b60
Fixed the coordinate conversion inside MESH/shapefile method.
msl3v May 31, 2024
500f8c8
Freeze here before making mods to the GDAL mesh routines to distribute
msl3v Aug 7, 2024
f044d07
Fixed element indexing. This is done in an inefficient way and
msl3v Aug 14, 2024
5cd5ccb
Added data files into Mesh/tests/data. Some other small updated.
msl3v Aug 14, 2024
a42ea1e
Switch to looping over DEs in Grid to initialize and check max.
oehmke Sep 6, 2024
2551606
Fix for PIO/GDAL interface.
msl3v Sep 11, 2024
4359391
Merge remote-tracking branch 'ESMF-GDAL-v1/MESH_Garble' into develop
msl3v Sep 13, 2024
e1b8773
Fixed a segfault for when there are no local elements to process
msl3v Sep 16, 2024
f379cc5
Fixed indexing in GDAL polygon processing for X,Y coords
msl3v Sep 19, 2024
10d1bb5
Clean-up -- MSL
msl3v Sep 20, 2024
271571b
More clean-up -- MSL
msl3v Sep 20, 2024
143d7d3
First round of changes for LocStream creation from Shapefile using GDAL
msl3v Oct 28, 2024
0d772a0
Fixed Feature ID error. FIDs < 1 being passed to mesh.
msl3v Oct 28, 2024
95069b8
Fixes for indexing, reading point geometry files. Testing.
msl3v Oct 29, 2024
80891f3
OK! We can read a locstream. And we can write a field! Barely BETA
msl3v Nov 7, 2024
251ecb5
Need to set precition for new floating point fields -- MSL
msl3v Nov 14, 2024
d55ca71
Update
Jul 1, 2025
cc9621c
Got LocStream working. Needs cleanup. Made some local changes for BAS…
Jul 22, 2025
c97c467
Added UTest for LocStream/GDAL
msl3v Jul 22, 2025
dff0b32
Did some debugging and added a parallel GDAL read routine to PIO.
msl3v Jul 23, 2025
55e41ea
Revamped the LocStream/Shapefile coordinate read routine to speed
msl3v Jul 24, 2025
03bcf44
OK! I think I fixed the IO issue reading on only one PET!
msl3v Jul 24, 2025
f3a6528
Swapped sindex with rindex() in pio_gdal.c
msl3v Jul 28, 2025
21c1706
OK. This seems to work better. But it is not efficient, and it would
msl3v Jul 29, 2025
eefe80f
OK. This should be more efficient for the initial LocStream read
msl3v Jul 29, 2025
33c1f28
Merge branch 'develop' into LocStream
msl3v Aug 4, 2025
4d53b88
Clean-up and make the GDAL_OpenEx routine in PIO not READONLY by defa…
msl3v Oct 10, 2025
e0bf1ab
Issues persistently not updated, now being updated.
Oct 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ PET*.ESMF_LogFile
/examples/
/src/Infrastructure/IO/PIO/Build/
/src/Infrastructure/IO/PIO/Install/

/holding
# protex generated files #
##########################
*_ccapi.tex
Expand Down
66 changes: 66 additions & 0 deletions ParallelIO/.github/workflows/autotools.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: autotools_ubuntu_latest

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:

runs-on: ubuntu-latest

env:
CC: mpicc
FC: mpifort
CPPFLAGS: "-I/usr/include -I/usr/local/include -I/home/runner/pnetcdf/include"
LDFLAGS: "-L/home/runner/pnetcdf/lib"
PNETCDF_VERSION: 1.12.3
FCFLAGS: "-fallow-argument-mismatch"
steps:
- uses: actions/checkout@v3
- name: Installs
run: |
set -x
# sudo gem install apt-spy2
# sudo apt-spy2 check
# sudo apt-spy2 fix --commit
# after selecting a specific mirror, we need to run 'apt-get update'
sudo apt-get update
sudo apt-get install netcdf-bin
sudo apt-get install libnetcdf-dev
sudo apt-get install doxygen
sudo apt-get install graphviz
sudo apt-get install wget
sudo apt-get install gfortran
sudo apt-get install libjpeg-dev
sudo apt-get install libz-dev
sudo apt-get install openmpi-bin
sudo apt-get install libopenmpi-dev
- name: cache-pnetcdf
id: cache-pnetcdf
uses: actions/cache@v3
with:
path: ~/pnetcdf
key: pnetcdf-${{ runner.os }}-${{ env.PNETCDF_VERSION }}

- name: build-pnetcdf
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
run: |
set -x
wget https://parallel-netcdf.github.io/Release/pnetcdf-${{ env.PNETCDF_VERSION }}.tar.gz &> /dev/null
tar -xzvf pnetcdf-${{ env.PNETCDF_VERSION }}.tar.gz
pushd pnetcdf-${{ env.PNETCDF_VERSION }}
./configure --prefix=/home/runner/pnetcdf --enable-shared --disable-cxx
make
sudo make install
popd

- name: autoreconf
run: autoreconf -i
- name: configure
run: ./configure --enable-fortran --with-mpiexec='mpiexec --oversubscribe'
#run: ./configure --enable-fortran --enable-docs --with-mpiexec='mpiexec --oversubscribe'
- name: make check
run: make -j check
64 changes: 64 additions & 0 deletions ParallelIO/.github/workflows/cmake.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: cmake

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:

runs-on: ubuntu-latest

env:
CC: mpicc
FC: mpifort
CPPFLAGS: "-I/usr/include -I/usr/local/include -I/home/runner/pnetcdf/include"
LDFLAGS: "-L/home/runner/pnetcdf/lib"
LD_LIBRARY_PATH: "/home/runner/pnetcdf/lib"
PNETCDF_VERSION: 1.12.3
FCFLAGS: "-fallow-argument-mismatch"
steps:
- uses: actions/checkout@v3
- name: Installs
run: |
set -x
#sudo gem install apt-spy2
#sudo apt-spy2 check
#sudo apt-spy2 fix --commit
# after selecting a specific mirror, we need to run 'apt-get update'
sudo apt-get update
sudo apt-get install netcdf-bin libnetcdf-dev doxygen graphviz wget gfortran libjpeg-dev libz-dev openmpi-bin libopenmpi-dev cmake

- name: cache-pnetcdf
id: cache-pnetcdf
uses: actions/cache@v3
with:
path: ~/pnetcdf
key: pnetcdf-${{ runner.os }}-${{ env.PNETCDF_VERSION }}

- name: build-pnetcdf
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
run: |
set -x
wget https://parallel-netcdf.github.io/Release/pnetcdf-${{ env.PNETCDF_VERSION }}.tar.gz &> /dev/null
tar -xzvf pnetcdf-${{ env.PNETCDF_VERSION }}.tar.gz
pushd pnetcdf-${{ env.PNETCDF_VERSION }}
./configure --prefix=/home/runner/pnetcdf --enable-shared --disable-cxx
make
sudo make install
popd

- name: cmake build
run: |
set -x
mkdir build
cd build
cmake -Wno-dev -DNetCDF_C_LIBRARY=/usr/lib/x86_64-linux-gnu/libnetcdf.so \
-DNetCDF_C_INCLUDE_DIR=/usr/include -DPnetCDF_PATH='/home/runner/pnetcdf' \
-DPIO_ENABLE_FORTRAN=Off \
-DPIO_ENABLE_LOGGING=On -DPIO_ENABLE_TIMING=Off -DMPIEXEC_PREFLAGS="--oversubscribe" ..
make VERBOSE=1
make tests VERBOSE=1
ctest -VV
68 changes: 68 additions & 0 deletions ParallelIO/.github/workflows/cmake_ubuntu_latest.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
--- # cmake build without netcdf integration
name: cmake_ubuntu-latest

on:
push:
branches:
- master
pull_request:
branches:
- master

jobs:
build:

runs-on: ubuntu-latest

env:
CC: mpicc
FC: mpifort
CPPFLAGS: "-I/usr/include -I/usr/local/include -I/home/runner/pnetcdf/include"
LDFLAGS: "-L/home/runner/pnetcdf/lib"
LD_LIBRARY_PATH: "/home/runner/pnetcdf/lib"
PNETCDF_VERSION: 1.12.3
FCFLAGS: "-fallow-argument-mismatch"
steps:
- uses: actions/checkout@v3
- name: Installs
run: |
set -x
#sudo gem install apt-spy2
#sudo apt-spy2 check
#sudo apt-spy2 fix --commit
# after selecting a specific mirror, we need to run 'apt-get update'
sudo apt-get update
sudo apt-get install netcdf-bin libnetcdf-dev doxygen graphviz wget gfortran libjpeg-dev libz-dev openmpi-bin libopenmpi-dev

- name: cache-pnetcdf
id: cache-pnetcdf
uses: actions/cache@v3
with:
path: ~/pnetcdf
key: pnetcdf-${{ runner.os }}-${{ env.PNETCDF_VERSION }}

- name: build-pnetcdf
if: steps.cache-pnetcdf.outputs.cache-hit != 'true'
run: |
set -x
wget https://parallel-netcdf.github.io/Release/pnetcdf-${{ env.PNETCDF_VERSION }}.tar.gz &> /dev/null
tar -xzvf pnetcdf-${{ env.PNETCDF_VERSION }}.tar.gz
pushd pnetcdf-${{ env.PNETCDF_VERSION }}
./configure --prefix=/home/runner/pnetcdf --enable-shared --disable-cxx
make
sudo make install
popd

- name: cmake build
run: |
set -x
mkdir build
cd build
cmake -Wno-dev -DPIO_ENABLE_NETCDF_INTEGRATION=OFF \
-DNetCDF_C_LIBRARY=/usr/lib/x86_64-linux-gnu/libnetcdf.so \
-DNetCDF_C_INCLUDE_DIR=/usr/include -DPnetCDF_PATH='/home/runner/pnetcdf' \
-DPIO_ENABLE_FORTRAN=OFF \
-DPIO_ENABLE_LOGGING=On -DPIO_ENABLE_TIMING=Off -DMPIEXEC_PREFLAGS="--oversubscribe" ..
make VERBOSE=1
make tests VERBOSE=1
ctest -VV
119 changes: 119 additions & 0 deletions ParallelIO/.github/workflows/netcdf_hdf5_no_pnetcdf_ncint_mpich.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
name: netcdf_hdf5_no_pnetcdf_ncint_mpich
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:

runs-on: ubuntu-latest

env:
CPPFLAGS: "-I/home/runner/mpich/include -I/home/runner/hdf5/include -I/home/runner/netcdf-c/include -I/home/runner/netcdf-fortran/include"
LDFLAGS: "-L/home/runner/mpich/lib -L/home/runner/hdf5/lib -L/home/runner/netcdf-c/lib -L/home/runner/netcdf-fortran/lib"
# Note issue https://github.com/NCAR/ParallelIO/issues/1889 netcdf integration currently only works with netcdf 4.7.4
NETCDF_C_VERSION: 4.7.4
NETCDF_F_VERSION: 4.5.4
MPICH_VERSION: 4.0.3
HDF5_VERSION_MAJOR: 1.12
HDF5_VERSION_PATCH: 2
FFLAGS: "-fallow-argument-mismatch"
FCFLAGS: "-fallow-argument-mismatch"
steps:
- uses: actions/checkout@v3
- name: Installs
run: |
sudo apt-get install doxygen graphviz wget gfortran libjpeg-dev libz-dev
- name: cache-mpich
id: cache-mpich
uses: actions/cache@v3
with:
path: ~/mpich
key: mpich-${{ runner.os }}-${{ env.MPICH_VERSION }}

- name: build-mpich
if: steps.cache-mpich.outputs.cache-hit != 'true'
run: |
wget http://www.mpich.org/static/downloads/${{ env.MPICH_VERSION }}/mpich-${{ env.MPICH_VERSION }}.tar.gz &> /dev/null
tar -xzf mpich-${{ env.MPICH_VERSION }}.tar.gz
pushd mpich-${{ env.MPICH_VERSION }}
./configure --prefix=/home/runner/mpich
make
sudo make install
popd
- name: cache-hdf5
id: cache-hdf5
uses: actions/cache@v3
with:
path: ~/hdf5
key: hdf5-${{ runner.os }}-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}-mpich-${{ env.MPICH_VERSION }}

- name: build-hdf5
if: steps.cache-hdf5.outputs.cache-hit != 'true'
run: |
set -x
export PATH="/home/runner/mpich/bin:$PATH"
export CC=/home/runner/mpich/bin/mpicc
wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-${{ env.HDF5_VERSION_MAJOR }}/hdf5-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}/src/hdf5-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}.tar.gz &> /dev/null
tar -xzf hdf5-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}.tar.gz
pushd hdf5-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}
./configure --prefix=/home/runner/hdf5 --enable-parallel --disable-tools --disable-fortran --disable-cxx --enable-parallel-tests
make
sudo make install
popd
- name: cache-netcdf-c
id: cache-netcdf-c
uses: actions/cache@v3
with:
path: ~/netcdf-c
key: netcdf-c-${{ runner.os }}-${{ env.NETCDF_C_VERSION }}-mpich-${{ env.MPICH_VERSION }}-hdf5-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}

- name: build-netcdf-c
if: steps.cache-netcdf-c.outputs.cache-hit != 'true'
run: |
set -x
export PATH="/home/runner/mpich/bin:$PATH"
export CC=/home/runner/mpich/bin/mpicc
wget https://github.com/Unidata/netcdf-c/archive/refs/tags/v${{ env.NETCDF_C_VERSION }}.tar.gz
tar -xzf v${{ env.NETCDF_C_VERSION }}.tar.gz &> /dev/null
pushd netcdf-c-${{ env.NETCDF_C_VERSION }}
./configure --prefix=/home/runner/netcdf-c --disable-dap --disable-utilities
make -j
sudo make install
popd

- name: cache-netcdf-fortran
id: cache-netcdf-fortran
uses: actions/cache@v3
with:
path: ~/netcdf-fortran
key: netcdf-fortran-${{ runner.os }}-${{ env.NETCDF_F_VERSION }}-mpich-${{ env.MPICH_VERSION }}-hdf5-${{ env.HDF5_VERSION_MAJOR }}.${{ env.HDF5_VERSION_PATCH }}

- name: build-netcdf-fortran
if: steps.cache-netcdf-fortran.outputs.cache-hit != 'true'
run: |
set -x
export PATH="/home/runner/mpich/bin:$PATH"
export CC=/home/runner/mpich/bin/mpicc
wget https://github.com/Unidata/netcdf-fortran/archive/refs/tags/v${{ env.NETCDF_F_VERSION }}.tar.gz
tar -zxf v${{ env.NETCDF_F_VERSION }}.tar.gz &> /dev/null
pushd netcdf-fortran-${{ env.NETCDF_F_VERSION }}
./configure --prefix=/home/runner/netcdf-fortran
make -j
sudo make install
popd

- name: autotools build
run: |
set -x
gcc --version
export PATH=/home/runner/mpich/bin:/home/runner/netcdf-c/bin:$PATH
export CC=/home/runner/mpich/bin/mpicc
export FC=/home/runner/mpich/bin/mpifort
autoreconf -i
./configure --enable-fortran --enable-netcdf-integration --disable-pnetcdf
make -j check


Loading