(#Closes 468) Added Directive node separated from comments

[LonelyCat124]

Initial implementation of this feature.

Ready for a look from any of @arporter @sergisiso @hiker

[LonelyCat124]

I've realised that this only works with 3.10 and above due to type declarations.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.19%. Comparing base (d13e4fa) to head (ef9ae0f).
⚠️ Report is 27 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #469      +/-   ##
==========================================
+ Coverage   92.16%   92.19%   +0.02%     
==========================================
  Files          86       86              
  Lines       13752    13802      +50     
==========================================
+ Hits        12675    12725      +50     
  Misses       1077     1077              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[LonelyCat124]

Pushed some changes now to fix some docs issues and coverage updates.

[arporter]

Thanks Aidan. Everything looks sensible but I do have a couple questions about inheritance and valid locations for directives.
In fact, now that I write that, please could you add a new section to the documentation - probably just before https://fparser.readthedocs.io/en/latest/fparser2.html#preprocessing-directives.
I'm wondering whether it would be wise for us to make this behaviour configurable somehow, just in case interpreting things as directives causes problems for downstream applications. I guess we could do this in the same way as we do for the support for non-standard extensions.

[LonelyCat124]

Addressed most of the comments.
Remaining questions about where directives can appear and fixed format directive test.

[sergisiso]

@LonelyCat124 I know we talked about that but I forgot. What is the benefit of differentiating in fparser between Comments and Directives? Are you planning to parse the body of the directive in the future?

[LonelyCat124]

@LonelyCat124 I know we talked about that but I forgot. What is the benefit of differentiating in fparser between Comments and Directives? Are you planning to parse the body of the directive in the future?

@sergisiso yes - its to separate directives from comments to make handling them in PSyclone better in the future (which I plan to continue down that avenue now). It also cleans up even the current implementation I think.

[LonelyCat124]

@arporter I think this is cleaned up and ready for another look now.

[arporter]

I'm happy with the functionality now but I think the docs need a bit more work - readfortran.py has its own section that needs updating.
Apart from that, we need to hear back from @hiker about the Fab issue.

[hiker]

Bad news; we appear to have broken Fab. The LFRic extraction test fails:

Building 'gungho_model_extract-fast-debug-mpif90-gfortran' took 0:00:27.212276
read_metric: recorded 526 metrics
matplotlib not installed, no metrics summary charts produced
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/apps/packages/python/3.13.7/lib/python3.13/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ~~~~^^^^^^^^^^^^^^^
  File "/apps/packages/python/3.13.7/lib/python3.13/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/archive/psyclone-tests/action-runner-software/actions-runner/_work/PSyclone-mirror/PSyclone-mirror/.runner_venv/lib/python3.13/site-packages/fab/parse/fortran_common.py", line 116, in run
    if node_tree.content[0] is None:
       ~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Although the LFRic integration test itself passes. Probably we need @hiker's expertise here... ...this might well not be the fault of this PR though - it could be due to some earlier change to fparser.

I think the PSyclone CI branch of Fab needs a patch that I had applied a while ago. Sorry, with UKMO moving slow, I have too many branches in progress. I will verify that.

[hiker]

Confirmed, I needed to update the PSyclone CI branch of our lfric Fab build repo. That's done now, please try again.

[arporter]

Thanks @hiker. I've just tried again and we now get a failure to find netcdf.mod:

Error compiling /archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output/psydata/extract_netcdf_base.f90:
Command failed with return code 1:
['/archive/psyclone-spack/psyclone-spack-Jun25/spack-repo/opt/spack/linux-rhel10-zen5/gcc-14.2.1/openmpi-5.0.6-7643sxs65dmw462noa4pwhem2tnpewyv/bin/mpif90', '-ffree-line-length-none', '-Wall', '-g', '-Werror=conversion', '-Werror=character-truncation', '-Werror=unused-value', '-Werror=tabs', '-std=f2008', '-fdefault-real-8', '-fdefault-double-8', '-fcheck=all', '-ffpe-trap=invalid,zero,overflow', '-Og', '-I', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output', '-J', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output', '-c', '-fopenmp', 'extract_netcdf_base.f90', '-o', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output/_prebuild/extract_netcdf_base.2ca77f7a6.o']
extract_netcdf_base.f90:180:9:

  180 |     USE netcdf, ONLY: nf90_noerr, nf90_strerror
      |         1
Fatal Error: Cannot open module file ‘netcdf.mod’ for reading at (1): No such file or directory

Does that ring a bell for either yourself or @sergisiso ?

[LonelyCat124]

@arporter I think I've addressed the remaining comments now, and fixed up the confusing test to test what it actually wants to test. Once we have the issues with building the integration tests resolved this should be ok for another look.

[sergisiso]

Thanks @hiker. I've just tried again and we now get a failure to find netcdf.mod:

Error compiling /archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output/psydata/extract_netcdf_base.f90:
Command failed with return code 1:
['/archive/psyclone-spack/psyclone-spack-Jun25/spack-repo/opt/spack/linux-rhel10-zen5/gcc-14.2.1/openmpi-5.0.6-7643sxs65dmw462noa4pwhem2tnpewyv/bin/mpif90', '-ffree-line-length-none', '-Wall', '-g', '-Werror=conversion', '-Werror=character-truncation', '-Werror=unused-value', '-Werror=tabs', '-std=f2008', '-fdefault-real-8', '-fdefault-double-8', '-fcheck=all', '-ffpe-trap=invalid,zero,overflow', '-Og', '-I', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output', '-J', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output', '-c', '-fopenmp', 'extract_netcdf_base.f90', '-o', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output/_prebuild/extract_netcdf_base.2ca77f7a6.o']
extract_netcdf_base.f90:180:9:

  180 |     USE netcdf, ONLY: nf90_noerr, nf90_strerror
      |         1
Fatal Error: Cannot open module file ‘netcdf.mod’ for reading at (1): No such file or directory

Does that ring a bell for either yourself or @sergisiso ?

I repeated this in a brach that previously successed and now it fails. So I don't think it is from this branch. https://github.com/stfc/PSyclone-mirror/actions/runs/17321986403/job/49677018062

[hiker]

Thanks @hiker. I've just tried again and we now get a failure to find netcdf.mod:

Error compiling /archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output/psydata/extract_netcdf_base.f90:
Command failed with return code 1:
['/archive/psyclone-spack/psyclone-spack-Jun25/spack-repo/opt/spack/linux-rhel10-zen5/gcc-14.2.1/openmpi-5.0.6-7643sxs65dmw462noa4pwhem2tnpewyv/bin/mpif90', '-ffree-line-length-none', '-Wall', '-g', '-Werror=conversion', '-Werror=character-truncation', '-Werror=unused-value', '-Werror=tabs', '-std=f2008', '-fdefault-real-8', '-fdefault-double-8', '-fcheck=all', '-ffpe-trap=invalid,zero,overflow', '-Og', '-I', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output', '-J', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output', '-c', '-fopenmp', 'extract_netcdf_base.f90', '-o', '/archive/psyclone-tests/latest-run/extraction_test/lfric_apps/applications/gungho_model/fab_workspace/gungho_model_extract-fast-debug-mpif90-gfortran/build_output/_prebuild/extract_netcdf_base.2ca77f7a6.o']
extract_netcdf_base.f90:180:9:

  180 |     USE netcdf, ONLY: nf90_noerr, nf90_strerror
      |         1
Fatal Error: Cannot open module file ‘netcdf.mod’ for reading at (1): No such file or directory

Does that ring a bell for either yourself or @sergisiso ?

@arporter , @sergisiso It appears that the include path is missing. Typically, that should be taken from $FFLAGS (unless manually added). There was some change with regards to handling flags, which is now on your Fab. What would you usually use to get the include paths?
For the libraries we are adding nf-config --flibs at the end.

[sergisiso]

@hiker The include path to netcdf is in FFLAGS. Could it be that fab does not take the existing environment values, I noticed that in this line:

        ${LFRIC_CORE}/build.sh ./fab_gungho_model_extract.py   \
          --profile ${COMPILER_PROFILE} --nproc 4 -fc $LDMPI -ld $LDMPI  -cc $CC

you add them explictily but not FFLAGS.

[hiker]

@hiker The include path to netcdf is in FFLAGS. Could it be that fab does not take the existing environment values, I noticed that in this line:

        ${LFRIC_CORE}/build.sh ./fab_gungho_model_extract.py   \
          --profile ${COMPILER_PROFILE} --nproc 4 -fc $LDMPI -ld $LDMPI  -cc $CC

you add them explictily but not FFLAGS.

Yes, indeed - after I updated the Fab version here, we now have an inconsistent set of lfric scripts and Fab in the PSyclone branch :( In the new Fab version, support for environment variable has been removed, but it has been moved into the (new) FabBase class instead. BUT, the lfric script doesn't yet use the new FabBase class, it uses an older version (when it was still called BafBase), and the version used did not yet support FFLAGS. Too many branches and repos (fab, lfric-baf, and then our LFRic CI build).

I've just tried to update the PSyclone CI branch for PSyclone's lfric builds again, it should now have the right version. But I don't have time to test, so feel free to ignore this (including failures in the extraction step for now), and I will verify and if necessary properly fix this on Monday.

Sorry for that mess, we are just in the middle of migrating scripts from Baf to Fab (and while this is technically only a name change, the code reviews do cause a few differences and conflicts), so this work is still sitting on a branch, not really ready to be merged in yet (i.e. has not been fully propagated through all three repos to check that all LFRic scripts still work). I shouldn't have updated fab itself (then again, not sure how I would have fixed the fparser change, which does require a small change in Fab ... maybe we should have created a custom Fab branch for you as well :) )

[arporter]

Thanks Aidan (and Joerg). Very nearly there now - we'll ignore the problem with the LFRic extraction test as it's nothing to do with this PR.
There's just a tweak needed to the documentation and I've requested an extension to a docstring in the tests to clarify where the reader functionality is actually tested. (I think this again points to it being in the wrong place but that's not for this PR).

[hiker]

FWIW, I've fixed the CI. Sorry, our fab and lfric branches are changing quit heavily atm, and we are struggling keeping everything after Fab *(lfric-Fab scripts, and our lfric CI) up-to-date.

[LonelyCat124]

@arporter I think I got everything - I wasn't sure if you wanted anything else adding to the FortranReaderBase introduction - if so let me know and I'll try to add more.

[arporter]

Thanks Aidan. Looks good now - the autoclass gives all the information I think was missing.
I'll proceed to merge.