Skip to content

Add 3D gather/scatter capability#154

Draft
pmarguinaud wants to merge 8 commits into
ecmwf-ifs:mainfrom
pmarguinaud:collapse
Draft

Add 3D gather/scatter capability#154
pmarguinaud wants to merge 8 commits into
ecmwf-ifs:mainfrom
pmarguinaud:collapse

Conversation

@pmarguinaud

@pmarguinaud pmarguinaud commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Description

Both FIELD_GATHSCAT (already in the Field API library) and FIELD_GATHSCAT_COLLAPSE2 (introduced in this PR) implement high-level gather-scatter patterns:

  1. Filter a multi-dimensional field using a logical condition (the trigger).
  2. Gather the data that satisfies the condition into a compact, contiguous array.
  3. Let the user modify that compact array (e.g., on CPU or GPU).
  4. Scatter the modified values back to their original locations in the source field.

This pattern is essential for algorithms that need to process only a subset of grid points (e.g., convective columns, cloud-covered areas, or land points); this has already proven to improve performance quite a lot. FIELD_GATHSCAT is currently used in the shallow convection algorithm of ARPEGE.

FIELD_GATHSCAT is used when the selection condition is 2-D and you want to keep all intermediate dimensions (e.g., vertical levels, variables) intact in the gathered array. It works with fields of any rank ≥ 2.

The purpose of FIELD_GATHSCAT_COLLAPSE2 is to be used when the selection condition is 3-D and you need to collapse the two leading spatial dimensions of arrays into one. This is typical for physics parametrizations that operate on a subset of atmospheric cells. We will use it in the AROME micro-physics.

Several tests have been added to exercise the newly created code.

Furthermore, we also implemented a FIELD_GATHSCAT_COLLAPSE1 whose purpose is identical to FIELD_GATHSCAT but is generated from the same fypp templates used for FIELD_GATHSCAT_COLLAPSE2; FIELD_GATHSCAT_COLLAPSE1 could replace FIELD_GATHSCAT (to be discussed).

Tested on A100 with nvhpc/26.5 and MI300X with afar/23.2.0. To be tested in AROME.

Contributor Declaration

By opening this pull request, I affirm the following:

  • All authors agree to the Contributor License Agreement.
  • The code follows the project's coding standards.
  • I have performed self-review and added comments where needed.
  • I have added or updated tests to verify that my changes are effective and functional.
  • I have run all existing tests and confirmed they pass.

@pmarguinaud pmarguinaud marked this pull request as draft June 30, 2026 09:01
@pmarguinaud pmarguinaud requested a review from awnawab June 30, 2026 09:13
@awnawab awnawab added the approved-for-ci Approved to run hpc-ci label Jul 1, 2026

@awnawab awnawab left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this development @pmarguinaud 🙏 We don't use the gather-scatter functionality, so we would support replacing the existing gather-scatter with gather-scatter-collapse1. Please proceed however you see fit.

I've left a few comments otherwise. I would prefer to restrict the fypp mapping to the rank-suff matrix, and for further templating moving the loop inside the files themselves. Since this is purely a stylistic consideration, I'll leave the final call up to you. But it would be nice to have a green CI before we merge.

Comment thread tests/gather_scatter_collapse2.F90 Outdated
#ifdef OMPGPU
!$OMP TARGET
#else
!$ACC SERIAL PRESENT (ID_FILTERED) COPYIN (YL_FGS) IF (JPASS == 2)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compilation is failing for this test in the OpenACC builds. That would be nice to fix before this can be merged. My best guess would be a missing CONTIGUOUS attribute on ID_FILTERED.


CALL FGS%SCATTER ()

DO JBLK = JBLKL, JBLKU

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're failing somewhere here in the cray builds. Since we don't use the gather scatter functionality, I would be ok with this test being excluded from the cray CI run (with something like a conditional ctest -E collapse2_lastdim.

#! granted to it by virtue of its status as an intergovernmental organisation
#! nor does it submit to any jurisdiction.

MODULE FIELD_${RANK}$${SUFF}$_GATHER_COLLAPSE${COLLAPSE}$_MODULE

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need a separate module per derived type here? If possible, I would prefer to keep the fypp expansion to the rank-suff matrix, and a loop in here for the different values of COLLAPSE. This is effectively what we are already doing in many places for WRAPPER, OWNER types in the same module. You could then simply add the relevant files to the ranksuff_srcs and non_ranksuff_srcs lists.

…ion is larger than NFLEVG. This can happen when processing arrays with extra levels such as :

REAL (KIND=JPRB) :: ZFLD (NPROMA, 0:NFLEVG)
@github-actions github-actions Bot removed the approved-for-ci Approved to run hpc-ci label Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants