From bd3449ce1961b3e36e9e38b5989baa229527af42 Mon Sep 17 00:00:00 2001 From: Ralph Castain Date: Thu, 9 Jul 2026 12:36:35 -0600 Subject: [PATCH 01/67] Import portable contributor rules from the PMIx AGENTS.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The PRRTE AGENTS.md and its PMIx counterpart share the same authorship and much of the same guidance, but the PMIx file had accumulated several general, project-agnostic rules that PRRTE's copy was missing. Carrying them over keeps the two orientation guides consistent so an agent moving between the code bases meets the same expectations, and it fills real gaps in the PRRTE guidance around style, hygiene, and process. Added: the conditional-spacing style rule; a directive to update .gitignore for any build product a change introduces; the "never bend a test to accommodate a bug" testing rule; a shared-repository/worktree section warning against repo-wide git commands; the preference for C++-style comments; commit-message guidance (body line wrapping, no AI tooling attribution, incidental fixes as standalone commits); and a maintainer-mode caution for build-system edits. While porting the build-system guidance, correct an over-broad claim in the existing text: because PRRTE builds in maintainer mode, editing a Makefile.am does not require the full autogen.pl plus configure cycle — a plain make regenerates the affected Makefiles. Also repair a garbled duplicate line in the copyright-header rule. Signed-off-by: Ralph Castain --- AGENTS.md | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 87 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f89cc0d573..5b5a57fe74 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -202,7 +202,6 @@ Do not use `PMIX_` or `pmix_` prefixes for new PRRTE symbols. ### New files need the standard copyright/license header. -Copy the multi-institution BSD header block — including the `Copyright (c) 2026 Nanook Consulting All rights reserved. Copy the multi-institution BSD header block — including the `$COPYRIGHT$` and `$HEADER$` tokens — from a neighboring file. If you substantially change an existing file, add your copyright line to its block. @@ -231,6 +230,17 @@ Use `{ }` around every conditional or loop body, even single-line ones. 4 spaces, never tab characters. +### Spacing on conditional statements + +Use a space to separate the condition from the surrounding keywords — +write `if (condition) {`, not `if(condition){`. When a condition spans +multiple lines, put the combining operator at the end of the preceding +line: + +```c +if (condition1 || + condition2) { +``` ### Stay compiler-warning-free @@ -245,6 +255,18 @@ be warning-free before submitting. Do not modify files produced by autotools (`configure`, `Makefile.in`, etc.), pre-rendered documentation, or third-party vendored code. Edit the source code instead. +### Update `.gitignore` for build products you introduce + +If your change adds a new source file, component, or generated artifact +that the build produces something new from — a new executable or test +binary, a newly generated source/header, an object or library in a +directory that did not have one before — add the resulting build product +to the appropriate `.gitignore` so it does not show up as an untracked +file. Never commit the build product itself; ignore it. Check +`git status` after a clean build to confirm no generated file you created +is left untracked, and match the nearest existing `.gitignore` pattern +style (many component directories carry their own `.gitignore`). + ### GOLDEN RULE: regenerate `show_help` content after touching any help file The `show_help` messages are embedded into the binary from a generated @@ -315,7 +337,8 @@ buffers unless interfacing with PMIx routines that require it. ### C standard PRRTE targets C11. Do not add `-Wno-*` flags to suppress warnings — -fix the underlying issue. +fix the underlying issue. C++-style `//` comments are allowed and +preferred. ### Use the `__prte_attribute_*__` macros for compiler attributes.** [`src/include/prte_config_bottom.h`](src/include/prte_config_bottom.h), @@ -338,8 +361,11 @@ the git repository; it is produced by running: ./autogen.pl ``` -This must be re-run whenever `configure.ac`, any `Makefile.am`, or any -`*.m4` file under `config/` is modified. After `autogen.pl`: +This must be re-run whenever `configure.ac` or any `*.m4` file under +`config/` is modified. Editing a `Makefile.am` does **not** require the +full `autogen.pl` + `./configure` cycle — PRRTE builds in maintainer +mode, so a plain `make` regenerates the affected `Makefile[.in]` files and +completes the build. After `autogen.pl`: ```sh ./configure [options] @@ -363,6 +389,31 @@ Common configure options: Version requirements: PMIx ≥ 6.1.0, hwloc ≥ 2.1.0, libevent ≥ 2.0.21. +### Modifying the configure / build system + +Editing the build system means regenerating it — `make` alone can't, and +trying will wedge the tree. If you change `configure.ac` or any +`config/*.m4` file (including the embedded oac/Autotools macros), the +change does not take effect until the build system is regenerated. Do +not rely on a plain `make`: PRRTE builds in maintainer mode, so `make` +auto-triggers a partial in-tree Autotools regeneration that frequently +fails (e.g., unexpanded `OAC_*` macros, `config.status` errors) and can +leave the tree half-regenerated and unbuildable. Instead, regenerate and +reconfigure explicitly: + +```sh +./autogen.pl +./configure +make -j +``` + +Recover the original configure invocation options from the existing tree +with `./config.status --config` (or read the header of `config.log`). +This process is slow but mandatory after any build-system source change — +there is no safe shortcut. As noted above, editing a `Makefile.am` alone +is the exception: a plain `make` regenerates the relevant `Makefile[.in]` +files and completes the build without the full cycle. + --- ## State Machines @@ -386,6 +437,23 @@ Key job states in order: `PRTE_JOB_STATE_INIT` → --- +## Working in a shared repository + +Don't assume you're the only agent (or person) using this clone. In +particular, if you're working in a **git worktree**, other worktrees may +be active against the same underlying repository at the same time. Avoid +repo-wide git commands that reach outside your own working area and can +disrupt others — for example, `git worktree prune`, or `git stash` +(which writes to the repository-wide stash ref shared by all worktrees). +Keep your git operations scoped to your own branch and worktree. + +As a narrow exception, creating a **new branch** when you need to park +work in progress (for example, instead of `git stash`) is fine. Just be +careful not to collide with branches that other agents or people may be +using in the same clone — pick a clearly-scoped, unlikely-to-clash name. + +--- + ## Contributing ### Commit messages @@ -393,7 +461,14 @@ Key job states in order: `PRTE_JOB_STATE_INIT` → Write prose commit messages, not bullet lists. The subject line should complete the sentence "If applied, this commit will …". The body must explain **why** the change is needed, not just what it does. Keep the -subject line under 72 characters. +subject line under 72 characters, and wrap body lines at around 75 +characters. Don't add AI tooling attribution to commit messages. + +Keep incidental fixes as their own commits. Small "drive-by" bug fixes +you notice while working on something else are welcome, but it is usually +best to land them as standalone commits, separate from your main change, +so each can be evaluated and reviewed on its own. One logical change per +commit keeps history reviewable and easy to bisect. All commits require a `Signed-off-by:` line (DCO): @@ -431,6 +506,13 @@ pterm # shut down DVM For resource manager integration (SLURM, PBS, LSF), test within an actual allocation on the relevant system. +**Never bend a test to accommodate a bug.** Do not weaken, skip, or +rewrite an existing test — and do not craft a new one — merely to make +buggy behavior pass. Tests encode intended behavior: when one fails, the +default assumption is that the code is wrong, not the test. If you find a +genuine bug in the code base, identify it, report it, and where +appropriate fix it — don't paper over it in the test suite. + ### Reporting bugs File issues at https://github.com/openpmix/prrte/issues. Include the From c9e31c00ddd7fc556da736dbbdf527a4666c8624 Mon Sep 17 00:00:00 2001 From: Ralph Castain Date: Mon, 13 Jul 2026 09:11:10 -0600 Subject: [PATCH 02/67] Sort MCA components by priority in m4 instead of a Perl script The m4-configure component list for each framework is ordered from highest to lowest priority so that STOP_AT_FIRST, STOP_AT_FIRST_PRIORITY, and PRIORITY frameworks configure their components in the intended order. Until now that ordering was produced by shelling out to config/prte_mca_priority_sort.pl through esyscmd during configure, which forks an external Perl interpreter at autogen time and adds Perl as an implicit build dependency for this one small task. Do the sort entirely in m4 instead. MCA_ORDER_COMPONENT_LIST now scans the framework's components to find the highest and lowest priorities, then walks the priorities from high to low, emitting the components that carry each priority. Walking the priorities rather than sorting the components preserves the original relative order among components that share a priority, matching the previous behavior. The Perl script and its EXTRA_DIST entry are removed. Signed-off-by: Ralph Castain --- config/Makefile.am | 1 - config/prte_mca.m4 | 50 ++++++++++++++++++++++++++++++-- config/prte_mca_priority_sort.pl | 32 -------------------- 3 files changed, 48 insertions(+), 35 deletions(-) delete mode 100755 config/prte_mca_priority_sort.pl diff --git a/config/Makefile.am b/config/Makefile.am index b1d2532053..0e03a5eece 100644 --- a/config/Makefile.am +++ b/config/Makefile.am @@ -42,7 +42,6 @@ EXTRA_DIST = \ prte_get_version.sh \ ltmain_nag_pthread.diff \ ltmain_pgi_tp.diff \ - prte_mca_priority_sort.pl \ find_common_syms \ getdate.sh \ from-savannah/upstream-config.guess \ diff --git a/config/prte_mca.m4 b/config/prte_mca.m4 index 4c68b1a0c5..1fbb44d50a 100644 --- a/config/prte_mca.m4 +++ b/config/prte_mca.m4 @@ -234,15 +234,61 @@ AC_DEFUN([PRTE_MCA],[ ]) +# _MCA_TRACK_PRIORITY_RANGE(priority) +# ----------------------------------- +# Fold the given priority into the running +# mca_component_max_priority / mca_component_min_priority values. Both +# must be defined (to the empty string, if this is the first priority +# seen) before invoking this macro. +m4_define([_MCA_TRACK_PRIORITY_RANGE], + [m4_ifval(mca_component_max_priority, + [m4_if(m4_eval([$1] > mca_component_max_priority), [1], + [m4_define([mca_component_max_priority], [$1])])dnl + m4_if(m4_eval([$1] < mca_component_min_priority), [1], + [m4_define([mca_component_min_priority], [$1])])], + [m4_define([mca_component_max_priority], [$1])dnl + m4_define([mca_component_min_priority], [$1])])]) + +# _MCA_EMIT_COMPONENT_IF_PRIORITY(framework_name, component_name, priority) +# ------------------------------------------------------------------------ +# Emit component_name (preceded by a separator, if it is not the first +# component emitted) if and only if its priority is the given priority. +# mca_component_separator must be defined to the empty string before +# the first invocation of this macro. +m4_define([_MCA_EMIT_COMPONENT_IF_PRIORITY], + [m4_if(m4_eval(PRTE_EVAL_ARG([MCA_prte_]$1[_]$2[_PRIORITY]) == [$3]), [1], + [mca_component_separator[]$2[]m4_define([mca_component_separator], [, ])])]) + # MCA_ORDER_COMPONENT_LIST(framework_name) +# ---------------------------------------- +# Define component_list to be the framework's m4-configure component +# list, ordered from highest to lowest +# MCA_prte___PRIORITY. Every component in the +# list must have a priority. AC_DEFUN([MCA_ORDER_COMPONENT_LIST], [ m4_foreach(mca_component, [mca_prte_$1_m4_config_component_list], [m4_ifval(mca_component, [m4_ifdef([MCA_prte_]$1[_]mca_component[_PRIORITY], [], [m4_fatal([MCA_prte_$1_]mca_component[_PRIORITY not found, but required.])])])]) +dnl Find the highest and lowest priorities in the framework. + m4_define([mca_component_max_priority], [])dnl + m4_define([mca_component_min_priority], [])dnl + m4_foreach([mca_component], [mca_prte_$1_m4_config_component_list], + [m4_ifval(mca_component, + [_MCA_TRACK_PRIORITY_RANGE(PRTE_EVAL_ARG([MCA_prte_]$1[_]mca_component[_PRIORITY]))])])dnl +dnl Walk the priorities from highest to lowest, emitting the components +dnl that have each priority. Walking the priorities (vs. sorting the +dnl components) means that components of equal priority are emitted in +dnl the same relative order in which they appear in the original list. + m4_define([mca_component_separator], [])dnl m4_define([component_list], - [esyscmd([config/prte_mca_priority_sort.pl] m4_foreach([mca_component], [mca_prte_$1_m4_config_component_list], - [m4_ifval(mca_component, [mca_component ]PRTE_EVAL_ARG([MCA_prte_]$1[_]mca_component[_PRIORITY ]))]))]) + m4_dquote(m4_ifval(mca_component_max_priority, + [m4_for([mca_component_priority], + mca_component_max_priority, mca_component_min_priority, -1, + [m4_foreach([mca_component], [mca_prte_$1_m4_config_component_list], + [m4_ifval(mca_component, + [_MCA_EMIT_COMPONENT_IF_PRIORITY($1, mca_component, + mca_component_priority)])])])]))) ]) AC_DEFUN([MCA_CHECK_IGNORED_PRIORITY], [ diff --git a/config/prte_mca_priority_sort.pl b/config/prte_mca_priority_sort.pl deleted file mode 100755 index f3b4890180..0000000000 --- a/config/prte_mca_priority_sort.pl +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env perl -# -# Copyright (c) 2010 Sandia National Laboratories. All rights reserved. -# -# Copyright (c) 2019 Intel, Inc. All rights reserved. -# $COPYRIGHT$ -# -# Additional copyrights may follow -# -# $HEADER$ -# - -my $components; -my @result; - -while (@ARGV) { - my $component; - $component->{"name"} = shift(@ARGV); - $component->{"value"} = shift(@ARGV); - push(@{$components}, $component); -} - -foreach my $component (sort { $b->{value} <=> $a->{value} } @{$components}) { - push(@result, $component->{name}); -} -sub commify_series { - (@_ == 0) ? '' : - (@_ == 1) ? $_[0] : - join(", ", @_[0 .. ($#_-1)], "$_[-1]"); -} - -print commify_series(@result); From 0e5894879ef0b0b65d707a78398b8f9d288e7fd8 Mon Sep 17 00:00:00 2001 From: Ralph Castain Date: Wed, 15 Jul 2026 11:36:32 -0600 Subject: [PATCH 03/67] Reorganize AGENTS.md to track the PMIx contributor guide PRRTE and PMIx are developed in lockstep and most contributors work in both trees, yet the two AGENTS.md guides had drifted into different section orders and different levels of coverage. A rule learned in one project was hard to relocate in the other, and PRRTE's guide had fallen behind the code. Reorder the page to follow the companion PMIx AGENTS.md so each topic sits in the same place in both. The thread model, caddy pattern, and blocking/non-blocking examples move out of the Coding Rules grab-bag into their own "Thread Safety & the Progress Thread" section, and the build and testing material is consolidated into a single "Build & Test Procedures" section rather than being split between a Build section and a stray Testing subsection under Contributing. Bring the content current. The Testing text still claimed PRRTE had no standalone unit tests; it now documents the test/unit suite wired into make check and the offline mapper harness driven by make check-offline over the shared synthetic topologies. Add the two procedures the old guide lacked -- how to test-build a change and a "did I break it" verification checklist -- and import the portable rules the PMIx guide carried that apply equally here: PRTE_EXPORT symbol visibility, unique numeric values for status and state codes, a Performance Considerations section, and a closing General Guidance section. Finally, repoint the stale cross-reference in the rmaps unit-test Makefile.am at the section that now describes the offline harness. Signed-off-by: Ralph Castain --- AGENTS.md | 290 ++++++++++++++++++++++++++++-------- test/unit/rmaps/Makefile.am | 6 +- 2 files changed, 234 insertions(+), 62 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5b5a57fe74..f0114b4edf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,6 +16,13 @@ portable code — not plausible-looking code that solves one problem in one environment or use-case at the expense of others. Hold yourself to the same bar as a thoughtful human contributor. +> **Note on layout:** PRRTE and PMIx are developed together, and many +> contributors work in both trees. This guide deliberately follows the +> same section order as the companion +> [PMIx `AGENTS.md`](https://github.com/openpmix/openpmix/blob/master/AGENTS.md) +> so that a rule you learned in one project sits in the same place in the +> other. + --- ## What is PRRTE? @@ -200,6 +207,11 @@ Nothing comes before it — not system headers, not other PRRTE headers. Do not use `PMIX_` or `pmix_` prefixes for new PRRTE symbols. +Use `PRTE_EXPORT` to annotate symbols that must be visible outside the +compilation unit that defines them (for example, functions a tool or +another framework calls). Leave purely internal symbols unannotated — +do not mark them `PRTE_EXPORT`. + ### New files need the standard copyright/license header. Copy the multi-institution BSD header block — including the `$COPYRIGHT$` and @@ -291,6 +303,24 @@ and pick up your help-text changes. Skipping this step leaves the binary serving the old (or missing) messages even though the `.txt` source looks correct. +### Unique numeric values for status and state codes + +Error constants, job states, and process states are hand-assigned +numeric values, and every value must be **unique** within its family — a +duplicate silently makes two distinct codes compare equal and is a +miserable bug to track down. + +- Error/return codes live in [`src/include/constants.h`](src/include/constants.h), + numbered as offsets from `PRTE_ERR_BASE` and `PRTE_ERR_SPLIT`. +- Job and process states live in + [`src/mca/plm/plm_types.h`](src/mca/plm/plm_types.h), numbered as + offsets from `PRTE_JOB_STATE_ERROR` and `PRTE_PROC_STATE_ERROR` (note + the sequences already skip some offsets — do not reuse a skipped one + assuming it is free). + +When adding a code, append it at the end of its list with the next +unused offset, and grep the relevant header to confirm no existing entry +already claims that value. ### Error handling @@ -299,34 +329,6 @@ Functions that can fail return `int`. Check every return value. Use `PRTE_ACTIVATE_JOB_STATE(jdata, PRTE_JOB_STATE_*)` to trigger state transitions rather than handling errors inline in launch paths. -### Thread model - -PRRTE is event-driven and single-threaded on the progress thread. Use -the PRRTE event loop (`prte_event_base`) for deferred work. Do not block on -the progress thread. - -### Thread-shifting with caddies - -A **caddy** is a short-lived heap object whose sole job is to carry a request's parameters across states within the progress thread. Every caddy struct must contain at minimum: - -| Field | Type | Purpose | -|-------|------|---------| -| ev | `pmix_event_t` | Required by Libevent to queue the caddy; **must be named `ev`** | -| lock | `pmix_lock_t` | Thread synchronization (blocking operations wait on this; handlers wake it) | -| cbdata | `void *` | Opaque pointer passed through to the callback | -| callback pointer(s) | function pointer(s) | Cache the caller-supplied callback function(s) | - -The pattern: - -1. Allocate a caddy with `PMIX_NEW(caddy_type_t)`. -2. Assign the caddy's fields to point at the caller's parameters — **do not copy the data**. -3. Call `PRTE_PMIX_THREADSHIFT(cd, evbase, handler_fn)` to post the caddy to the progress thread's event queue. -4. The progress thread fires `handler_fn(cd)`, which performs the actual work. - -Never read or write shared library state outside of the progress thread; do it only inside the handler that runs on the progress thread. -Do not allocate a caddy on the stack — it must outlive the function that creates it. - - ### Memory management Use `PMIX_NEW` / `PMIX_RELEASE` (PMIx's reference-counted object system) @@ -340,19 +342,20 @@ PRRTE targets C11. Do not add `-Wno-*` flags to suppress warnings — fix the underlying issue. C++-style `//` comments are allowed and preferred. -### Use the `__prte_attribute_*__` macros for compiler attributes.** - [`src/include/prte_config_bottom.h`](src/include/prte_config_bottom.h), - pulled in transitively by `prte_config.h`, defines portable wrappers — - `__prte_attribute_unused__`, `__prte_attribute_noreturn__`, - `__prte_attribute_format__`, `__prte_attribute_deprecated__`, and many - more — that expand to the appropriate `__attribute__((...))` on - compilers that support it and to nothing elsewhere. Reach for these - (for example, to mark an unused function parameter) rather than writing - a bare `__attribute__` or leaving a warning unaddressed. +### Use the `__prte_attribute_*__` macros for compiler attributes. + +[`src/include/prte_config_bottom.h`](src/include/prte_config_bottom.h), +pulled in transitively by `prte_config.h`, defines portable wrappers — +`__prte_attribute_unused__`, `__prte_attribute_noreturn__`, +`__prte_attribute_format__`, `__prte_attribute_deprecated__`, and many +more — that expand to the appropriate `__attribute__((...))` on +compilers that support it and to nothing elsewhere. Reach for these +(for example, to mark an unused function parameter) rather than writing +a bare `__attribute__` or leaving a warning unaddressed. --- -## Build System +## Build & Test Procedures PRRTE uses GNU Autotools. The generated `configure` script is **not** in the git repository; it is produced by running: @@ -362,7 +365,7 @@ the git repository; it is produced by running: ``` This must be re-run whenever `configure.ac` or any `*.m4` file under -`config/` is modified. Editing a `Makefile.am` does **not** require the +`config/` is modified or added. Editing a `Makefile.am` does **not** require the full `autogen.pl` + `./configure` cycle — PRRTE builds in maintainer mode, so a plain `make` regenerates the affected `Makefile[.in]` files and completes the build. After `autogen.pl`: @@ -389,6 +392,20 @@ Common configure options: Version requirements: PMIx ≥ 6.1.0, hwloc ≥ 2.1.0, libevent ≥ 2.0.21. +### Test-building your changes + +Build from the repository root with `make -j$(nproc)`. Running `make` +from the root is what respects the generated headers and the per-target +compiler flags; building from deep inside a subdirectory can miss a +regenerated header and give you a misleading result. + +If you configured with `--enable-mca-dso` (components built as separate +DSOs rather than statically linked into the tools), you can rebuild a +single component after editing it by running `make install` in that +component's build directory — you do not have to relink every tool. In +the default static build, a component change requires a normal +root-level `make` so the tools are relinked. + ### Modifying the configure / build system Editing the build system means regenerating it — `make` alone can't, and @@ -414,6 +431,165 @@ there is no safe shortcut. As noted above, editing a `Makefile.am` alone is the exception: a plain `make` regenerates the relevant `Makefile[.in]` files and completes the build without the full cycle. +### Testing + +PRRTE now ships real automated tests in addition to integration-level +launch testing. Use the right layer for what you touched. + +**Unit tests (`make check`).** Self-contained unit tests live under +[`test/unit/`](test/unit/) and are wired into `make check` (for example, +`test/unit/rmaps/test_rmaps` exercises the mapper's policy parsing, +option resolution, and each mapping component — `round_robin`, `ppr`, +`seq`, `rank_file` — with no live DVM). Run them from the build tree: + +```sh +make check +``` + +Add new unit tests here, under the framework they cover, and wire them +into the appropriate `Makefile.am` `TESTS =` list so `make check` picks +them up. + +**Offline mapper harness (`make check-offline`).** Mapping, ranking, and +binding behavior can be exercised without launching anything. +[`test/offline/run_offline_maps.py`](test/offline/) drives +`prterun --rtos donotlaunch --display map` over a matrix of `--map-by`, +`--rank-by`, and `--bind-to` directives crossed with the synthetic hwloc +topologies in [`test/topologies/`](test/topologies/), then checks each +printed map against invariants derived from the topology. It is **not** +part of `make check` (it needs a freshly built `prterun` and runs well +over a thousand cases); run it on demand from a build tree: + +```sh +make -C test/offline check-offline +``` + +Whenever you change the mapper, run this harness — it is the cheapest way +to catch a mapping regression. + +**Integration testing.** For launch, I/O forwarding, and lifecycle +changes, run an actual DVM: + +```sh +prte --daemonize # start DVM +prun -n 4 hostname # basic launch smoke test +pterm # shut down DVM +``` + +For resource-manager integration (SLURM, PBS, LSF), test within an actual +allocation on the relevant system. + +**Never bend a test to accommodate a bug.** Do not weaken, skip, or +rewrite an existing test — and do not craft a new one — merely to make +buggy behavior pass. Tests encode intended behavior: when one fails, the +default assumption is that the code is wrong, not the test. If you find a +genuine bug in the code base, identify it, report it, and where +appropriate fix it — don't paper over it in the test suite. + +### Did I break it? — verification checklist + +Before you consider a change finished, work down this list; do the steps +that apply to what you touched: + +1. **Clean, warning-free build.** Configure with `--enable-debug` (which + turns warnings into errors) and confirm the tree builds clean — pay + special attention to conditionally-compiled paths (code behind a + capability flag or an RM `--with-*` option) that your local build may + not even be exercising. +2. **`make check`.** The unit tests must pass. +3. **Offline mapper harness** (`make -C test/offline check-offline`) for + any change to mapping, ranking, or binding. +4. **Live smoke test.** Start a DVM, launch a small job, and shut it down + (`prte --daemonize` → `prun -n 4 hostname` → `pterm`) for any change to + launch, I/O forwarding, or the state machine. +5. **Docs build.** For user-visible changes, update the RST under + [`docs/`](docs/) and build the docs (`make` in `docs/` produces the + Sphinx HTML) to confirm they render warning-free. +6. **Broaden when feasible.** Where you can, repeat across environments + and resource managers — PRRTE's whole reason for existing is + portability across systems you may not have in front of you. + +--- + +## Thread Safety & the Progress Thread + +### Thread model + +PRRTE is event-driven and single-threaded on the progress thread. Use +the PRRTE event loop (`prte_event_base`) for deferred work. Do not block on +the progress thread. + +### Thread-shifting with caddies + +A **caddy** is a short-lived heap object whose sole job is to carry a request's parameters across states within the progress thread. Every caddy struct must contain at minimum: + +| Field | Type | Purpose | +|-------|------|---------| +| ev | `pmix_event_t` | Required by Libevent to queue the caddy; **must be named `ev`** | +| lock | `prte_pmix_lock_t` | Thread synchronization (blocking operations wait on this; handlers wake it) | +| cbdata | `void *` | Opaque pointer passed through to the callback | +| callback pointer(s) | function pointer(s) | Cache the caller-supplied callback function(s) | + +The pattern: + +1. Allocate a caddy with `PMIX_NEW(caddy_type_t)`. +2. Assign the caddy's fields to point at the caller's parameters — **do not copy the data**. +3. Call `PRTE_PMIX_THREADSHIFT(cd, evbase, handler_fn)` to post the caddy to the progress thread's event queue. +4. The progress thread fires `handler_fn(cd)`, which performs the actual work. + +Never read or write shared library state outside of the progress thread; do it only inside the handler that runs on the progress thread. +Do not allocate a caddy on the stack — it must outlive the function that creates it. + +#### Blocking operations + +When the caller must wait for the work to finish, embed a +`prte_pmix_lock_t` and block on it after the thread-shift. The handler +wakes the caller by calling `PRTE_PMIX_WAKEUP_THREAD` on the same lock +before it returns: + +```c +prte_pmix_lock_t lock; +PRTE_PMIX_CONSTRUCT_LOCK(&lock); +cd->lock = &lock; /* caddy carries the lock */ +PRTE_PMIX_THREADSHIFT(cd, prte_event_base, _do_the_work); +PRTE_PMIX_WAIT_THREAD(&lock); /* block until woken */ +rc = lock.status; +PRTE_PMIX_DESTRUCT_LOCK(&lock); +``` + +The handler running on the progress thread does its work, records the +result in `lock->status`, and finishes with `PRTE_PMIX_WAKEUP_THREAD(cd->lock)`. + +#### Non-blocking operations + +When the caller supplies a callback and must not block, thread-shift and +return immediately: + +```c +PRTE_PMIX_THREADSHIFT(cd, prte_event_base, _do_the_work); +return PRTE_SUCCESS; +``` + +Here the handler performs the work, invokes the caller's callback, and +releases the caddy with `PMIX_RELEASE(cd)` — there is **no** +`PRTE_PMIX_WAKEUP_THREAD` because no one is waiting on a lock. + +--- + +## Performance Considerations + +PRRTE launches and manages jobs at extreme scale, so the daemon startup, +mapping, and collective paths are performance-sensitive. + +- Do not add allocations, locks, or branches to hot paths (launch, + mapping, RML message handling, collectives) without a measured + justification. +- Guard verbose debug output and expensive assertions behind + `#if PRTE_ENABLE_DEBUG` (or a framework verbosity check) so they cost + nothing in a production build. +- Prefer an MCA parameter over a hard-coded constant for any value that + might need tuning across different systems or scales. + --- ## State Machines @@ -491,29 +667,25 @@ instead, and open the pull request from the fork against the upstream `master` (or the appropriate release branch). If you are unsure which remote is the fork, run `git remote -v` and ask rather than guessing. -### Testing +### Reporting bugs -PRRTE does not have a standalone unit test suite. Integration-level -testing is done by running actual parallel jobs through the DVM. When -modifying launch, mapping, or I/O forwarding paths, test with: +File issues at https://github.com/openpmix/prrte/issues. Include the +output of `prte_info --all` and a minimal reproduction case. -```sh -prte --daemonize # start DVM -prun -n 4 hostname # basic launch smoke test -pterm # shut down DVM -``` +--- -For resource manager integration (SLURM, PBS, LSF), test within an actual -allocation on the relevant system. +## General Guidance -**Never bend a test to accommodate a bug.** Do not weaken, skip, or -rewrite an existing test — and do not craft a new one — merely to make -buggy behavior pass. Tests encode intended behavior: when one fails, the -default assumption is that the code is wrong, not the test. If you find a -genuine bug in the code base, identify it, report it, and where -appropriate fix it — don't paper over it in the test suite. +When in doubt: -### Reporting bugs +- Match the style and conventions of the surrounding code. +- Read the relevant pages under [`docs/developers/`](docs/developers/) + before inventing a new pattern — the mechanism you need often already + exists. +- For significant work, raise a GitHub issue to discuss the approach + before implementing it. -File issues at https://github.com/openpmix/prrte/issues. Include the -output of `prte_info --all` and a minimal reproduction case. +PRRTE runs on the world's largest supercomputers across a wide range of +operating systems and hardware. The project values careful, portable +code — not plausible-looking solutions that work in one environment or +use-case at the expense of others. diff --git a/test/unit/rmaps/Makefile.am b/test/unit/rmaps/Makefile.am index 3ce15b034f..706df1287c 100644 --- a/test/unit/rmaps/Makefile.am +++ b/test/unit/rmaps/Makefile.am @@ -21,8 +21,8 @@ test_rmaps_SOURCES = \ test_rmaps_LDADD = $(top_builddir)/src/libprrte.la -# The synthetic hwloc topology used to exercise map/rank/bind patterns -# without a live DVM now lives in the shared test/topologies directory -# (see the "Testing the mapper without launching" section in AGENTS.md). +# The synthetic hwloc topologies used to exercise map/rank/bind patterns +# without a live DVM now live in the shared test/topologies directory +# (see the "Testing" section, "Offline mapper harness", in AGENTS.md). TESTS = test_rmaps From 3a503cb6fe5a01321239e1bc96981e78a878328e Mon Sep 17 00:00:00 2001 From: Ralph Castain Date: Wed, 15 Jul 2026 13:21:44 -0600 Subject: [PATCH 04/67] Add per-framework AGENTS.md orientation guides under src/mca Give every MCA framework under src/mca a developer-level AGENTS.md that orients a contributor to what the framework does, where it runs in the DVM and job lifecycle, the module/component vtable contract, and a detailed walk of the functions its base/ provides. Each component directory gains its own AGENTS.md describing that component's files, selection gates, and the specifics of how it implements the framework interface. Every AGENTS.md carries a CLAUDE.md symlink so either name resolves to the same guide. These follow the model already established by the rmaps guides and let an AI agent or a human new to a subsystem come up to speed from the code itself rather than by reverse-engineering it each time. The companion rmaps framework guide is refreshed to match the current source (the map_job size note and a ranking-wording correction). The empty common framework gets a short guide explaining why its Makefile.am placeholder exists. Documentation only; no source or build files are changed. Signed-off-by: Ralph Castain --- src/mca/common/AGENTS.md | 62 +++ src/mca/common/CLAUDE.md | 1 + src/mca/errmgr/AGENTS.md | 298 ++++++++++++++ src/mca/errmgr/CLAUDE.md | 1 + src/mca/errmgr/dvm/AGENTS.md | 215 ++++++++++ src/mca/errmgr/dvm/CLAUDE.md | 1 + src/mca/errmgr/prted/AGENTS.md | 203 ++++++++++ src/mca/errmgr/prted/CLAUDE.md | 1 + src/mca/ess/AGENTS.md | 382 ++++++++++++++++++ src/mca/ess/CLAUDE.md | 1 + src/mca/ess/env/AGENTS.md | 100 +++++ src/mca/ess/env/CLAUDE.md | 1 + src/mca/ess/hnp/AGENTS.md | 148 +++++++ src/mca/ess/hnp/CLAUDE.md | 1 + src/mca/ess/lsf/AGENTS.md | 92 +++++ src/mca/ess/lsf/CLAUDE.md | 1 + src/mca/ess/pals/AGENTS.md | 91 +++++ src/mca/ess/pals/CLAUDE.md | 1 + src/mca/ess/slurm/AGENTS.md | 98 +++++ src/mca/ess/slurm/CLAUDE.md | 1 + src/mca/filem/AGENTS.md | 267 +++++++++++++ src/mca/filem/CLAUDE.md | 1 + src/mca/filem/raw/AGENTS.md | 262 +++++++++++++ src/mca/filem/raw/CLAUDE.md | 1 + src/mca/grpcomm/AGENTS.md | 255 ++++++++++++ src/mca/grpcomm/CLAUDE.md | 1 + src/mca/grpcomm/direct/AGENTS.md | 315 +++++++++++++++ src/mca/grpcomm/direct/CLAUDE.md | 1 + src/mca/iof/AGENTS.md | 414 ++++++++++++++++++++ src/mca/iof/CLAUDE.md | 1 + src/mca/iof/hnp/AGENTS.md | 170 ++++++++ src/mca/iof/hnp/CLAUDE.md | 1 + src/mca/iof/prted/AGENTS.md | 179 +++++++++ src/mca/iof/prted/CLAUDE.md | 1 + src/mca/odls/AGENTS.md | 431 +++++++++++++++++++++ src/mca/odls/CLAUDE.md | 1 + src/mca/odls/pdefault/AGENTS.md | 187 +++++++++ src/mca/odls/pdefault/CLAUDE.md | 1 + src/mca/plm/AGENTS.md | 412 ++++++++++++++++++++ src/mca/plm/CLAUDE.md | 1 + src/mca/plm/lsf/AGENTS.md | 103 +++++ src/mca/plm/lsf/CLAUDE.md | 1 + src/mca/plm/pals/AGENTS.md | 118 ++++++ src/mca/plm/pals/CLAUDE.md | 1 + src/mca/plm/slurm/AGENTS.md | 146 +++++++ src/mca/plm/slurm/CLAUDE.md | 1 + src/mca/plm/ssh/AGENTS.md | 237 +++++++++++ src/mca/plm/ssh/CLAUDE.md | 1 + src/mca/prtebacktrace/AGENTS.md | 241 ++++++++++++ src/mca/prtebacktrace/CLAUDE.md | 1 + src/mca/prtebacktrace/execinfo/AGENTS.md | 110 ++++++ src/mca/prtebacktrace/execinfo/CLAUDE.md | 1 + src/mca/prtebacktrace/none/AGENTS.md | 94 +++++ src/mca/prtebacktrace/none/CLAUDE.md | 1 + src/mca/prtebacktrace/printstack/AGENTS.md | 103 +++++ src/mca/prtebacktrace/printstack/CLAUDE.md | 1 + src/mca/prtedl/AGENTS.md | 266 +++++++++++++ src/mca/prtedl/CLAUDE.md | 1 + src/mca/prtedl/dlopen/AGENTS.md | 163 ++++++++ src/mca/prtedl/dlopen/CLAUDE.md | 1 + src/mca/prtedl/libltdl/AGENTS.md | 191 +++++++++ src/mca/prtedl/libltdl/CLAUDE.md | 1 + src/mca/prteinstalldirs/AGENTS.md | 343 ++++++++++++++++ src/mca/prteinstalldirs/CLAUDE.md | 1 + src/mca/prteinstalldirs/config/AGENTS.md | 130 +++++++ src/mca/prteinstalldirs/config/CLAUDE.md | 1 + src/mca/prteinstalldirs/env/AGENTS.md | 139 +++++++ src/mca/prteinstalldirs/env/CLAUDE.md | 1 + src/mca/prtereachable/AGENTS.md | 272 +++++++++++++ src/mca/prtereachable/CLAUDE.md | 1 + src/mca/prtereachable/netlink/AGENTS.md | 190 +++++++++ src/mca/prtereachable/netlink/CLAUDE.md | 1 + src/mca/prtereachable/weighted/AGENTS.md | 132 +++++++ src/mca/prtereachable/weighted/CLAUDE.md | 1 + src/mca/ras/AGENTS.md | 367 ++++++++++++++++++ src/mca/ras/CLAUDE.md | 1 + src/mca/ras/bootstrap/AGENTS.md | 53 +++ src/mca/ras/bootstrap/CLAUDE.md | 1 + src/mca/ras/flux/AGENTS.md | 61 +++ src/mca/ras/flux/CLAUDE.md | 1 + src/mca/ras/gridengine/AGENTS.md | 50 +++ src/mca/ras/gridengine/CLAUDE.md | 1 + src/mca/ras/hosts/AGENTS.md | 92 +++++ src/mca/ras/hosts/CLAUDE.md | 1 + src/mca/ras/lsf/AGENTS.md | 58 +++ src/mca/ras/lsf/CLAUDE.md | 1 + src/mca/ras/pbs/AGENTS.md | 55 +++ src/mca/ras/pbs/CLAUDE.md | 1 + src/mca/ras/pmix/AGENTS.md | 60 +++ src/mca/ras/pmix/CLAUDE.md | 1 + src/mca/ras/simulator/AGENTS.md | 60 +++ src/mca/ras/simulator/CLAUDE.md | 1 + src/mca/ras/slurm/AGENTS.md | 116 ++++++ src/mca/ras/slurm/CLAUDE.md | 1 + src/mca/ras/testrm/AGENTS.md | 46 +++ src/mca/ras/testrm/CLAUDE.md | 1 + src/mca/rmaps/AGENTS.md | 7 +- src/mca/schizo/AGENTS.md | 294 ++++++++++++++ src/mca/schizo/CLAUDE.md | 1 + src/mca/schizo/ompi/AGENTS.md | 173 +++++++++ src/mca/schizo/ompi/CLAUDE.md | 1 + src/mca/schizo/prte/AGENTS.md | 166 ++++++++ src/mca/schizo/prte/CLAUDE.md | 1 + src/mca/state/AGENTS.md | 341 ++++++++++++++++ src/mca/state/CLAUDE.md | 1 + src/mca/state/dvm/AGENTS.md | 184 +++++++++ src/mca/state/dvm/CLAUDE.md | 1 + src/mca/state/prted/AGENTS.md | 148 +++++++ src/mca/state/prted/CLAUDE.md | 1 + 109 files changed, 9941 insertions(+), 3 deletions(-) create mode 100644 src/mca/common/AGENTS.md create mode 120000 src/mca/common/CLAUDE.md create mode 100644 src/mca/errmgr/AGENTS.md create mode 120000 src/mca/errmgr/CLAUDE.md create mode 100644 src/mca/errmgr/dvm/AGENTS.md create mode 120000 src/mca/errmgr/dvm/CLAUDE.md create mode 100644 src/mca/errmgr/prted/AGENTS.md create mode 120000 src/mca/errmgr/prted/CLAUDE.md create mode 100644 src/mca/ess/AGENTS.md create mode 120000 src/mca/ess/CLAUDE.md create mode 100644 src/mca/ess/env/AGENTS.md create mode 120000 src/mca/ess/env/CLAUDE.md create mode 100644 src/mca/ess/hnp/AGENTS.md create mode 120000 src/mca/ess/hnp/CLAUDE.md create mode 100644 src/mca/ess/lsf/AGENTS.md create mode 120000 src/mca/ess/lsf/CLAUDE.md create mode 100644 src/mca/ess/pals/AGENTS.md create mode 120000 src/mca/ess/pals/CLAUDE.md create mode 100644 src/mca/ess/slurm/AGENTS.md create mode 120000 src/mca/ess/slurm/CLAUDE.md create mode 100644 src/mca/filem/AGENTS.md create mode 120000 src/mca/filem/CLAUDE.md create mode 100644 src/mca/filem/raw/AGENTS.md create mode 120000 src/mca/filem/raw/CLAUDE.md create mode 100644 src/mca/grpcomm/AGENTS.md create mode 120000 src/mca/grpcomm/CLAUDE.md create mode 100644 src/mca/grpcomm/direct/AGENTS.md create mode 120000 src/mca/grpcomm/direct/CLAUDE.md create mode 100644 src/mca/iof/AGENTS.md create mode 120000 src/mca/iof/CLAUDE.md create mode 100644 src/mca/iof/hnp/AGENTS.md create mode 120000 src/mca/iof/hnp/CLAUDE.md create mode 100644 src/mca/iof/prted/AGENTS.md create mode 120000 src/mca/iof/prted/CLAUDE.md create mode 100644 src/mca/odls/AGENTS.md create mode 120000 src/mca/odls/CLAUDE.md create mode 100644 src/mca/odls/pdefault/AGENTS.md create mode 120000 src/mca/odls/pdefault/CLAUDE.md create mode 100644 src/mca/plm/AGENTS.md create mode 120000 src/mca/plm/CLAUDE.md create mode 100644 src/mca/plm/lsf/AGENTS.md create mode 120000 src/mca/plm/lsf/CLAUDE.md create mode 100644 src/mca/plm/pals/AGENTS.md create mode 120000 src/mca/plm/pals/CLAUDE.md create mode 100644 src/mca/plm/slurm/AGENTS.md create mode 120000 src/mca/plm/slurm/CLAUDE.md create mode 100644 src/mca/plm/ssh/AGENTS.md create mode 120000 src/mca/plm/ssh/CLAUDE.md create mode 100644 src/mca/prtebacktrace/AGENTS.md create mode 120000 src/mca/prtebacktrace/CLAUDE.md create mode 100644 src/mca/prtebacktrace/execinfo/AGENTS.md create mode 120000 src/mca/prtebacktrace/execinfo/CLAUDE.md create mode 100644 src/mca/prtebacktrace/none/AGENTS.md create mode 120000 src/mca/prtebacktrace/none/CLAUDE.md create mode 100644 src/mca/prtebacktrace/printstack/AGENTS.md create mode 120000 src/mca/prtebacktrace/printstack/CLAUDE.md create mode 100644 src/mca/prtedl/AGENTS.md create mode 120000 src/mca/prtedl/CLAUDE.md create mode 100644 src/mca/prtedl/dlopen/AGENTS.md create mode 120000 src/mca/prtedl/dlopen/CLAUDE.md create mode 100644 src/mca/prtedl/libltdl/AGENTS.md create mode 120000 src/mca/prtedl/libltdl/CLAUDE.md create mode 100644 src/mca/prteinstalldirs/AGENTS.md create mode 120000 src/mca/prteinstalldirs/CLAUDE.md create mode 100644 src/mca/prteinstalldirs/config/AGENTS.md create mode 120000 src/mca/prteinstalldirs/config/CLAUDE.md create mode 100644 src/mca/prteinstalldirs/env/AGENTS.md create mode 120000 src/mca/prteinstalldirs/env/CLAUDE.md create mode 100644 src/mca/prtereachable/AGENTS.md create mode 120000 src/mca/prtereachable/CLAUDE.md create mode 100644 src/mca/prtereachable/netlink/AGENTS.md create mode 120000 src/mca/prtereachable/netlink/CLAUDE.md create mode 100644 src/mca/prtereachable/weighted/AGENTS.md create mode 120000 src/mca/prtereachable/weighted/CLAUDE.md create mode 100644 src/mca/ras/AGENTS.md create mode 120000 src/mca/ras/CLAUDE.md create mode 100644 src/mca/ras/bootstrap/AGENTS.md create mode 120000 src/mca/ras/bootstrap/CLAUDE.md create mode 100644 src/mca/ras/flux/AGENTS.md create mode 120000 src/mca/ras/flux/CLAUDE.md create mode 100644 src/mca/ras/gridengine/AGENTS.md create mode 120000 src/mca/ras/gridengine/CLAUDE.md create mode 100644 src/mca/ras/hosts/AGENTS.md create mode 120000 src/mca/ras/hosts/CLAUDE.md create mode 100644 src/mca/ras/lsf/AGENTS.md create mode 120000 src/mca/ras/lsf/CLAUDE.md create mode 100644 src/mca/ras/pbs/AGENTS.md create mode 120000 src/mca/ras/pbs/CLAUDE.md create mode 100644 src/mca/ras/pmix/AGENTS.md create mode 120000 src/mca/ras/pmix/CLAUDE.md create mode 100644 src/mca/ras/simulator/AGENTS.md create mode 120000 src/mca/ras/simulator/CLAUDE.md create mode 100644 src/mca/ras/slurm/AGENTS.md create mode 120000 src/mca/ras/slurm/CLAUDE.md create mode 100644 src/mca/ras/testrm/AGENTS.md create mode 120000 src/mca/ras/testrm/CLAUDE.md create mode 100644 src/mca/schizo/AGENTS.md create mode 120000 src/mca/schizo/CLAUDE.md create mode 100644 src/mca/schizo/ompi/AGENTS.md create mode 120000 src/mca/schizo/ompi/CLAUDE.md create mode 100644 src/mca/schizo/prte/AGENTS.md create mode 120000 src/mca/schizo/prte/CLAUDE.md create mode 100644 src/mca/state/AGENTS.md create mode 120000 src/mca/state/CLAUDE.md create mode 100644 src/mca/state/dvm/AGENTS.md create mode 120000 src/mca/state/dvm/CLAUDE.md create mode 100644 src/mca/state/prted/AGENTS.md create mode 120000 src/mca/state/prted/CLAUDE.md diff --git a/src/mca/common/AGENTS.md b/src/mca/common/AGENTS.md new file mode 100644 index 0000000000..ad0dba6d4a --- /dev/null +++ b/src/mca/common/AGENTS.md @@ -0,0 +1,62 @@ +# AGENTS.md — The `common` Framework (shared component code) + +Orientation for AI agents and human contributors working in +`src/mca/common/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework is + +`common` is not a real MCA framework. It has **no `base/` directory, no +framework interface header, no component selection, and no components**. +It is a container reserved for *shared "common" libraries* — bodies of +code that more than one component (often across different frameworks) +need to link against, factored out so they are compiled exactly once. + +This pattern is inherited from the wider PMIx/Open MPI MCA lineage, where +`mca/common//` directories hold code such as shared-memory helpers +that several components reuse. **PRRTE currently ships no such shared +library**, so the directory holds only its build-glue placeholder. + +--- + +## Directory layout + +``` +common/ + Makefile.am # intentionally (almost) empty — see below + Makefile.in # generated by autogen.pl — do not hand-edit + Makefile # generated by configure — do not hand-edit +``` + +The only meaningful file is `Makefile.am`, and it is empty of build +rules on purpose. Its own comment explains why: + +> Note that this file must exist, even though it is empty (there is no +> "base" directory for the common framework). `autogen.pl` and +> `prte_mca.m4` assume that every framework has a top-level +> `Makefile.am`. We *could* adjust the framework glue code to exclude +> "common" from this requirement, but it's just a lot easier to have an +> empty `Makefile.am` here. + +In other words the file exists solely to satisfy the Autotools/MCA glue +that walks every `src/mca/*` directory expecting a top-level +`Makefile.am`. + +--- + +## If you need to add shared code here + +Should PRRTE ever need a genuinely shared "common" library, add it as a +subdirectory `common//` with its own `Makefile.am` building a +convenience (or DSO) library, following the established PMIx `mca/common` +convention and the copyright/license header rules in the top-level +[`AGENTS.md`](../../../AGENTS.md). Regenerating the build system after +adding a new directory here requires the full +`./autogen.pl` → `./configure` → `make` cycle, not a plain `make` — see +the build-system rules in the top-level guide. + +Until then, there is nothing to modify in this directory. diff --git a/src/mca/common/CLAUDE.md b/src/mca/common/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/common/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/errmgr/AGENTS.md b/src/mca/errmgr/AGENTS.md new file mode 100644 index 0000000000..1557edfa76 --- /dev/null +++ b/src/mca/errmgr/AGENTS.md @@ -0,0 +1,298 @@ +# AGENTS.md — The `errmgr` Framework (Error and Recovery Manager) + +Orientation for AI agents and human contributors working in +`src/mca/errmgr/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`errmgr` (Error and Recovery Manager) is the **central clearing house +for process- and daemon-state changes** in a running DVM. When any part +of PRRTE decides a process or daemon has entered an error state — it +aborted, exited non-zero, called `PMIx_Abort`, failed to launch, or its +communication link dropped — that fact is funneled into this framework, +which then decides **what to do about it**: report it, notify peers, +recover the resources, terminate the offending job, or tear down the +whole DVM. + +Unlike most frameworks, `errmgr` does **not** expose a rich vtable that +callers invoke directly. Its module struct carries only `init`, +`finalize`, and a `logfn` (see below). The real work happens through the +**state machine**: at `init()` time the selected component registers +callbacks on specific job and process error states, and the `state` +framework fires those callbacks — as thread-shifted events on the +progress thread — whenever code anywhere calls +`PRTE_ACTIVATE_JOB_STATE(jdata, PRTE_JOB_STATE_*)` or +`PRTE_ACTIVATE_PROC_STATE(&name, PRTE_PROC_STATE_*)` with an error +state. So the errmgr's behavior *is* its set of state-machine handlers. + +Where it runs matters, because the two components implement completely +different policies for the same states: + +| Process role | Selected component | Job | +|--------------|--------------------|-----| +| HNP / DVM master (`PRTE_PROC_IS_MASTER`) | `dvm` | Decide DVM-wide policy: notify the job's submitter, terminate a failed job while keeping the DVM alive, or tear the DVM down. | +| prted daemon (`PRTE_PROC_IS_DAEMON`) | `prted` | Report local process/daemon state changes **up to the HNP**, kill local children, and exit cleanly when ordered. | +| tool (`prun`, `pterm`, …) | **none** | Tools never call `prte_errmgr_base_select()`, so they keep the default log-only module. | + +Its place in the lifecycle is orthogonal to the launch path: `rmaps`, +`plm`, etc. drive a job *forward* through +`INIT → ALLOCATE → MAP → LAUNCH_DAEMONS → RUNNING → TERMINATED`; the +errmgr is what catches the job (or a daemon) when any of those steps — +or a running proc — goes wrong, and routes it to `TERMINATED` (or, for +recoverable jobs, back to a survivable state). + +--- + +## Directory layout + +``` +errmgr/ + errmgr.h # module + component structs; the (thin) vtable; version macro + base/ + base.h # framework struct + prte_errmgr_base_select() prototype + errmgr_private.h # prte_errmgr_default_fns + prte_errmgr_base_log() prototype + errmgr_base_frame.c # framework open/close/DECLARE; the global prte_errmgr module + errmgr_base_select.c # pmix_mca_base_select — classic single-winner pick-one + errmgr_base_fns.c # prte_errmgr_base_log() — the default logfn implementation + help-errmgr-base.txt # user-facing error/help text (failed-daemon, node-died, …) + static-components.h # generated: lists dvm + prted as the static components + dvm/ # HNP component (pri 1000, gated PRTE_PROC_IS_MASTER) + prted/ # daemon component (pri 1000, gated PRTE_PROC_IS_DAEMON) +``` + +Read `errmgr.h` first (it is short — the whole contract is ~40 lines), +then jump straight to the component you care about; the two `_module.c` +files (`dvm/errmgr_dvm.c`, `prted/errmgr_prted.c`) are where all the +policy lives. + +--- + +## The module contract + +The module struct (`prte_errmgr_base_module_2_3_0_t`, in `errmgr.h`) is +deliberately tiny: + +```c +typedef int (*prte_errmgr_base_module_init_fn_t)(void); +typedef int (*prte_errmgr_base_module_finalize_fn_t)(void); +typedef void (*prte_errmgr_base_module_log_fn_t)(int error_code, char *filename, int line); + +struct prte_errmgr_base_module_2_3_0_t { + prte_errmgr_base_module_init_fn_t init; /* register state callbacks here */ + prte_errmgr_base_module_finalize_fn_t finalize; + prte_errmgr_base_module_log_fn_t logfn; +}; +``` + +| Member | Meaning / protocol | +|--------|--------------------| +| `init` | Return `PRTE_SUCCESS`/`PRTE_ERROR`. **This is where a component wires itself into the state machine** via `prte_state.add_job_state()` / `prte_state.add_proc_state()`. May be `NULL` (the default module leaves it `NULL`). | +| `finalize` | Return `PRTE_SUCCESS`/`PRTE_ERROR`. May be `NULL`; both real components make it a no-op. | +| `logfn` | `void`, no error return. Formats and prints a `PRTE_ERROR_LOG`-style message. **Must always be non-NULL** — see the gotcha below. | + +The public global that everyone links against is +`PRTE_EXPORT extern prte_errmgr_base_module_t prte_errmgr;` — after +selection this holds a *copy* of the winning component's module struct. + +### The `logfn` gotcha + +`logfn` looks like the hook behind `PRTE_ERROR_LOG`, but it is not. +`PRTE_ERROR_LOG(rc)` is a standalone macro in +[`src/util/error.h`](../../../src/util/error.h) that calls `pmix_output` +directly; it never touches `prte_errmgr.logfn`. In the current tree +**nothing outside this framework calls `prte_errmgr.logfn`** — it is +effectively vestigial, retained because the module struct is initialized +with it even before the framework is opened. The header comment in +`errmgr_base_frame.c` is emphatic about that initialization: + +```c +/* NOTE: ABSOLUTELY MUST initialize this struct to include the log + * function as it gets called even if the errmgr hasn't been opened + * yet due to error */ +prte_errmgr_base_module_t prte_errmgr = { .logfn = prte_errmgr_base_log }; +``` + +Do not "clean up" that initialization to `{0}`; a NULL `logfn` used +during a very early failure would crash instead of reporting. + +--- + +## The component struct + +`prte_errmgr_base_component_3_0_0_t` (in `errmgr.h`) is a standard MCA +component wrapper plus three ints — `verbose`, `output_handle`, +`priority`. Both real components only ever register `priority` as an MCA +param; `verbose`/`output_handle` are unused in the current components +(framework verbosity comes from `errmgr_base_verbose`, below). The +version macro components must use is: + +```c +#define PRTE_ERRMGR_BASE_VERSION_3_0_0 PRTE_MCA_BASE_VERSION_3_0_0("errmgr", 3, 0, 0) +``` + +--- + +## What `base/` provides + +The base is intentionally thin — it does selection, open/close, and one +log function; it holds **no** error policy of its own (that all lives in +the components). + +### `errmgr_base_frame.c` — framework plumbing and the default module + +- `PMIX_MCA_BASE_FRAMEWORK_DECLARE(prte, errmgr, …, prte_errmgr_base_open, + prte_errmgr_base_close, prte_errmgr_base_static_components, …)` declares + `prte_errmgr_base_framework`. Because the register hook is `NULL`, the + only framework-level MCA param is the auto-provided `errmgr_base_verbose`. +- `prte_errmgr_base_open()` loads `prte_errmgr = prte_errmgr_default_fns` + (log-only) and opens all components. +- `prte_errmgr_base_close()` calls the selected module's `finalize` (if + any), then **restores the default log-only fns** so `prte_errmgr.logfn` + stays valid through shutdown, then closes the components. +- Defines the two module globals: + - `prte_errmgr_default_fns` — `{ init=NULL, finalize=NULL, + logfn=prte_errmgr_base_log }`. The fallback used by tools and during + early errors. + - `prte_errmgr` — the live module, pre-seeded with the default `logfn`. + +### `errmgr_base_select.c` — `prte_errmgr_base_select()` + +Classic **single-winner** MCA selection (unlike `rmaps`, which keeps all +modules): it calls `pmix_mca_base_select("errmgr", …)`, which queries +every component's `query` and keeps the one returning the highest +priority. The winner's module struct is **copied** into the global +`prte_errmgr`, and its `init()` is invoked (a non-`PRTE_SUCCESS` return +fails the whole selection). If no component is selectable it returns +`PRTE_ERROR`. Only the HNP (`ess/hnp`) and daemons (`ess/base`'s +`ess_base_std_prted`) call this; tools skip it and keep the default +module. + +### `errmgr_base_fns.c` — `prte_errmgr_base_log()` + +The default `logfn`. Turns an error code into a string via +`PRTE_ERROR_NAME` (`prte_strerror`) and prints +`" PRTE_ERROR_LOG: in file at line "` through +`pmix_output(0, …)`. If `prte_strerror` returns `NULL` (a "silent" +error) it prints nothing. This is the only executable code the base +contributes to error handling. + +### `help-errmgr-base.txt` + +The user-facing messages the components emit via `pmix_show_help`: +`failed-daemon-launch`, `failed-daemon`, `node-died`, `simple-message`. +Per the top-level GOLDEN RULE, if you touch this file you must +`rm src/util/prte_show_help_content.* && make` to force the generated +show-help content to be rebuilt. + +--- + +## Component selection + +`query` (in each component's `_component.c`) is a pure **role gate**: + +| Component | Gate | Priority when it applies | +|-----------|------|--------------------------| +| `dvm` | `PRTE_PROC_IS_MASTER` | `1000` (MCA param `errmgr_dvm_priority`) | +| `prted` | `PRTE_PROC_IS_DAEMON` | `1000` (MCA param `errmgr_prted_priority`) | + +Both default to priority **1000**, but that never collides: each returns +`*module = NULL; *priority = -1; return PRTE_ERROR` when the process is +not its role, so in any given process **at most one** component is +selectable. This is why the "highest priority wins" machinery is almost +decorative here — role, not priority, does the selecting. (The priority +params still let you swap in an out-of-tree replacement for one role by +registering a higher number.) + +There is deliberately **no** generic/default component: a tool that +never calls `prte_errmgr_base_select()` simply runs with +`prte_errmgr_default_fns`, which does nothing but log. + +--- + +## Threading and the state machine + +Every real errmgr handler (`job_errors`, `proc_errors`, and the daemon's +`prted_abort`/`wakeup`) runs **on the progress thread**, invoked by the +`state` framework as a libevent callback. The callback signature is the +state-machine one, `(int fd, short args, void *cbdata)`, where `cbdata` +is a `prte_state_caddy_t *` carrying the `jdata`, the target `name`, and +the `job_state`/`proc_state`. The very first thing each handler does is +`PMIX_ACQUIRE_OBJECT(caddy)` and the last thing is `PMIX_RELEASE(caddy)` +on every exit path (watch the `goto cleanup;` discipline — a missed +release leaks the caddy). Because everything is single-threaded on the +progress thread, the handlers freely read and mutate global runtime +state (`prte_local_children`, `prte_process_info.num_daemons`, +`prte_rml_base.n_children`, the abort/term-ordered flags) without locks. + +Handlers **must not block** and must not do their own thread-shifting for +the common path — they are already on the right thread. They cause +further work by activating *other* states +(`PRTE_ACTIVATE_JOB_STATE`/`PRTE_ACTIVATE_PROC_STATE`), which enqueue +new events, rather than calling termination logic inline. + +--- + +## Conventions and gotchas specific to this framework + +- **The interesting code is not in the vtable.** Reading `errmgr.h` tells + you almost nothing about behavior. Grep each component's `init()` for + `add_job_state` / `add_proc_state` to find the real entry points. +- **Two components, divergent policy, shared state names.** `dvm` and + `prted` both register handlers for `PRTE_JOB_STATE_ERROR`, + `PRTE_PROC_STATE_COMM_FAILED`, and `PRTE_PROC_STATE_ERROR`, but do + opposite things (decide vs. report-upward). When you change how one + role treats a state, check whether the other role needs the mirror + change. +- **`finalizing` short-circuits.** Both `job_errors` handlers bail + immediately if `prte_finalizing` is set — a shutting-down DVM must not + re-drive error policy. Preserve that guard. +- **The daemon-job special case.** In both components a `jdata` whose + nspace equals `PRTE_PROC_MY_NAME->nspace` is *the daemon job itself*, + not an application job, and is handled on a separate branch (its + failure means the DVM is in trouble). `caddy->jdata == NULL` is treated + as "the daemon job" and back-filled from + `prte_get_job_data_object(PRTE_PROC_MY_NAME->nspace)`. +- **Elastic-DVM interactions live in `dvm/proc_errors`.** Grow-target + rollback (`prte_plm_base_grow_target_failed`) and the shrink echo-guard + (`prte_elastic_mode && !ALIVE && state >= TERMINATED`) are subtle and + gated on `prte_elastic_mode`; see the `dvm` component guide and the + repo memory on elastic-DVM ordering bugs before editing them. +- Standard PRRTE rules still apply: `prte_config.h` first, braces on + every block, `NULL ==`/constant-on-left comparisons, no new compiler + warnings, `PRTE_ERROR_LOG`/`PMIX_ERROR_LOG` on unexpected `rc`, + `PRTE_ACTIVATE_*_STATE` to drive transitions rather than inline work. + +--- + +## Debugging + +```sh +prte --prtemca errmgr_base_verbose 5 ... # trace errmgr decisions +prte --prtemca state_base_verbose 5 ... # see the state transitions that fire errmgr +prte --prtemca plm_base_verbose 5 ... # daemon launch/termination context +``` + +The errmgr handlers emit at verbosity 1–5 on +`prte_errmgr_base_framework.framework_output`: level 1 logs each +job/proc state received, level 5 traces the per-state decision (which +proc, which policy branch, whether a notification was sent, whether the +DVM is being torn down). Because the errmgr sits downstream of the state +machine, pairing `errmgr_base_verbose` with `state_base_verbose` is the +fastest way to see *what* transitioned and *how* the errmgr reacted. + +--- + +## Where to go next + +Each component directory has its own `AGENTS.md`: + +- [`dvm/AGENTS.md`](dvm/AGENTS.md) — the HNP-side policy engine: notify + submitters, terminate jobs, tear down the DVM, elastic grow/shrink + fault handling. +- [`prted/AGENTS.md`](prted/AGENTS.md) — the daemon-side reporter: push + local proc/daemon state to the HNP, kill local children, exit on order. diff --git a/src/mca/errmgr/CLAUDE.md b/src/mca/errmgr/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/errmgr/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/errmgr/dvm/AGENTS.md b/src/mca/errmgr/dvm/AGENTS.md new file mode 100644 index 0000000000..90b65585ba --- /dev/null +++ b/src/mca/errmgr/dvm/AGENTS.md @@ -0,0 +1,215 @@ +# AGENTS.md — `errmgr/dvm` (the HNP error policy engine) + +Component guide for `src/mca/errmgr/dvm/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +state-machine callback model, and the progress-thread threading notes +referenced throughout. + +--- + +## Role and priority + +`dvm` is the errmgr component that runs **only on the HNP / DVM master**. +Priority **1000** (MCA param `errmgr_dvm_priority`), but selection is by +role, not number: `dvm_component_query()` returns the module only when +`PRTE_PROC_IS_MASTER`, otherwise `*module = NULL; *priority = -1; +return PRTE_ERROR`. It is the process that decides DVM-wide policy when +something fails — whether to notify a job's submitter, terminate a single +job while keeping the DVM up, or tear the whole DVM down. + +Files: + +| File | Contents | +|------|----------| +| `errmgr_dvm_component.c` | Registration, `priority` MCA param (default 1000), `query` gated on `PRTE_PROC_IS_MASTER`, open/close no-ops. | +| `errmgr_dvm.c` | The module: `init`/`finalize`, the two state handlers `job_errors` and `proc_errors`, and helpers `_terminate_job` and `check_send_notification`. | +| `errmgr_dvm.h` | Exports `prte_mca_errmgr_dvm_component` and `prte_errmgr_dvm_module`. | + +--- + +## How it wires into the state machine (`init`) + +`init()` registers three callbacks and returns `PRTE_SUCCESS`: + +```c +prte_state.add_job_state(PRTE_JOB_STATE_ERROR, job_errors); +prte_state.add_proc_state(PRTE_PROC_STATE_COMM_FAILED, proc_errors); +prte_state.add_proc_state(PRTE_PROC_STATE_ERROR, proc_errors); +``` + +`PRTE_JOB_STATE_ERROR` and `PRTE_PROC_STATE_ERROR` are the **generic +error catch-alls**: activating any job/proc *error* state routes here +(the actual `caddy->job_state` / `caddy->proc_state` carries the specific +value). `COMM_FAILED` is registered separately because it is meant to run +at message priority so last messages from the failing peer can still be +drained. `finalize()` is a no-op. + +--- + +## `job_errors` — job-level failures + +Fires when a job is activated into an error state. Flow: + +1. `PMIX_ACQUIRE_OBJECT(caddy)`; bail immediately if `prte_finalizing`. +2. If `caddy->jdata == NULL`, this refers to the **daemon job** — back-fill + it from `prte_get_job_data_object(PRTE_PROC_MY_NAME->nspace)` and + `PMIX_RETAIN` it. +3. Copy `caddy->job_state` into `jdata->state`. +4. **Two policies, chosen by whose job it is:** + + **The daemon job itself** (`nspace == PRTE_PROC_MY_NAME->nspace`) — + the DVM is in trouble: + - `FAILED_TO_START` / `NEVER_LAUNCHED` / `FAILED_TO_LAUNCH` / + `CANNOT_LAUNCH`: disable routing (`prte_routing_is_enabled = false`) + and activate `PRTE_JOB_STATE_DAEMONS_TERMINATED` to exit. + - `ABORTED` while `num_procs != num_reported`: a daemon likely died + without finding its way back — show `help-errmgr-base.txt: + failed-daemon` and disable routing. + - Otherwise mark `num_terminated = num_procs` and activate + `PRTE_JOB_STATE_TERMINATED` — there is nothing to do but exit, + because the failure is in the DVM plumbing. + + **A submitted application job** (any other nspace) — keep the DVM + alive, only the job dies: + - Convert the job state to a PMIx error + (`prte_pmix_convert_job_state_to_error`) and send a **spawn + response** (`prte_plm_base_spawn_response`) so a quick-failing job + still generates a reply to its requestor (`jdata->originator`). + - `_terminate_job(jdata->nspace)` to kill any of its procs still + running. + - If the job never actually launched (`FAILED_TO_START`, + `NEVER_LAUNCHED`, `FAILED_TO_LAUNCH`, `ALLOC_FAILED`, `MAP_FAILED`, + `CANNOT_LAUNCH`), activate `PRTE_JOB_STATE_TERMINATED` explicitly — + no proc states will fire to drive termination otherwise. +5. `PMIX_RELEASE(caddy)` on every exit path. + +The `MAP_FAILED`/`ALLOC_FAILED`/`NEVER_LAUNCHED` cases are exactly the +failure states `rmaps`/`ras`/`plm` funnel here — this is the far end of +those frameworks' `..._FAILED` transitions. + +--- + +## `proc_errors` — process- and daemon-level failures + +The larger handler. After the standard acquire / `finalizing` / `jdata` +lookup, it splits into **daemon** vs **application** proc handling. + +### Daemon proc errors (`nspace == PRTE_PROC_MY_NAME->nspace`) + +For the communication-loss family — `COMM_FAILED`, `HEARTBEAT_FAILED`, +`UNABLE_TO_SEND_MSG`, `FAILED_TO_CONNECT`, `FAILED_TO_START`: + +1. **Ignore my own connection** (`proc->rank == PRTE_PROC_MY_NAME->rank`). +2. **Elastic shrink echo-guard.** If `prte_elastic_mode` and the daemon + is already not-`ALIVE` with `state >= PRTE_PROC_STATE_TERMINATED`, this + is a harmless late comm-failure for a daemon already torn out of the + DVM (the collective shrink-completion handler in `ras_base_allocate.c` + proactively marked it) — ignore it, or we would double-decrement + `num_daemons` and re-drive the abort logic. The `state >= TERMINATED` + test is deliberate: it lets a genuine `FAILED_TO_START` daemon (never + alive, but state still below TERMINATED) fall through and be handled. +3. Mark the daemon gone: `PRTE_FLAG_UNSET(pptr, PRTE_PROC_FLAG_ALIVE)`, + record `pptr->state = state`, and `--prte_process_info.num_daemons`. +4. **Elastic grow rollback.** If `prte_plm_base_grow_target_failed(rank)` + claims this rank (it was an in-flight grow target), the grow campaign + absorbs the loss — `goto cleanup` and skip the general daemon-loss + handling, which would otherwise abort the whole DVM over a failure the + rollback already handled. +5. **Ordered termination in progress** (`prte_prteds_term_ordered || + prte_abnormal_term_ordered`): record the daemon as gone via + `prte_rml_route_lost`, and if no routed children remain + (`prte_rml_base.n_children == 0`) and all local children are dead, + activate `DAEMONS_TERMINATED` to exit; else just note the remaining + routes. `goto cleanup`. +6. **Unexpected daemon loss** (the real fault path): show + `node-died` (unless `FAILED_TO_START`), call `prte_rml_route_lost`. + On success **the HNP walks every job** and marks each proc that lived + on the lost daemon's node `PRTE_PROC_STATE_TERM_WO_SYNC` (only rank 0 + / the HNP does this sweep), then `goto cleanup`. Otherwise mark the + daemon job `PRTE_JOB_STATE_COMM_FAILED`, stash the offending proc in + `PRTE_JOB_ABORTED_PROC`, set `PRTE_JOB_FLAG_ABORTED`, and set + `exit_code` (defaulting to `PRTE_ERR_COMM_FAILURE`). +7. Because comms have failed we cannot trust the routing tree — force + `prte_abnormal_term_ordered = true` and activate `DAEMONS_TERMINATED`. + +Any non-comm daemon state hits the `pmix_output(0, "UNSUPPORTED DAEMON +ERROR STATE …")` branch — a real one indicates a bug upstream. + +### Application proc errors + +First, idempotency: `pptr->state = state` only if +`pptr->state < PRTE_PROC_STATE_TERMINATED` (a proc can be reported more +than once). If `prte_prteds_term_ordered`, check whether any local child +is still alive and, if not and no routed children remain, exit. + +Then it always marks the waitpid fired +(`PRTE_ACTIVATE_PROC_STATE(WAITPID_FIRED)`) and, for a **remote** proc, +also marks `IOF_COMPLETE` (we'll hear nothing more about it). It computes +`flag = RECOVERABLE || CONTINUOUS` from the job attributes — the pivot +between "notify and keep going" and "abort the job": + +| Proc state | `flag` set (recoverable/continuous) | `flag` clear | +|------------|-------------------------------------|--------------| +| `KILLED_BY_CMD` | notify `PMIX_ERR_PROC_KILLED_BY_CMD` + recover resources | if all procs terminated → `TERMINATED` | +| `ABORTED_BY_SIG` | notify `PMIX_ERR_PROC_ABORTED_BY_SIG` + recover | set `JOB_STATE_ABORTED_BY_SIG`, record aborted proc, `_terminate_job` | +| `TERM_WO_SYNC` | notify `PMIX_ERR_PROC_TERM_WO_SYNC` + recover | set `ABORTED_WO_SYNC`; if `exit_code == 0` force `PRTE_ERROR_DEFAULT_EXIT_CODE` so the user sees an error; `_terminate_job` | +| `FAILED_TO_START` / `FAILED_TO_LAUNCH` | *(unconditional)* set `FAILED_TO_START`/`_LAUNCH`, `_terminate_job`, activate `FAILED_TO_START`; if it was a daemon, show `failed-daemon-launch` | same | +| `CALLED_ABORT` | notify `PMIX_ERR_PROC_REQUESTED_ABORT` + recover | set `CALLED_ABORT`, `_terminate_job` | +| `TERM_NON_ZERO` | if `PRTE_JOB_ERROR_NONZERO_EXIT` also set: notify `PMIX_ERR_EXIT_NONZERO_TERM` + recover | set `NON_ZERO_TERM`, `_terminate_job`; always bump `PRTE_JOB_NUM_NONZERO_EXIT` | +| default | if `num_terminated == num_procs` → `TERMINATED` | + +The abort branches all guard on `!PRTE_JOB_FLAG_ABORTED` so only the +**first** offending proc drives the abort, `PMIX_RETAIN` the recorded +proc so it survives, and stash it in `PRTE_JOB_ABORTED_PROC` for the +eventual error report. `prte_state_base_recover_resources` is what makes +a recoverable/continuous job survivable — it frees the dead proc's slot +so a replacement can be mapped. + +--- + +## Helpers + +### `_terminate_job(nspace)` +Builds a one-element proc array holding `{nspace, PMIX_RANK_WILDCARD}` +and calls `prte_plm.terminate_procs()` — the standard "kill this whole +job" call. + +### `check_send_notification(jdata, proc, event)` +Emits a PMIx event to the *surviving* procs of a recoverable/continuous +job. Bails if `PRTE_JOB_NOTIFY_ERRORS` is not set, if +`prte_dvm_abort_ordered`, or if the job is already `ABORTED`. Otherwise +it hand-packs a data buffer — source rank (`PRTE_NAME_INVALID->rank` so +the PMIx server injects it locally), the `pmix_status_t` event, the +source proc, a `PMIX_RANGE_CUSTOM` range, then an info array +(`PMIX_EVENT_AFFECTED_PROC`, `PMIX_EVENT_CUSTOM_RANGE` = the whole job, +and `PMIX_EXIT_CODE` when `exit_code != -1`) — and `prte_grpcomm.xcast`s +it on `PRTE_RML_TAG_NOTIFICATION`. This is the mechanism a fault-tolerant +application uses to learn a peer died without the whole job being killed. + +--- + +## Things to watch when editing + +- **Daemon-job vs application-job is the top-level fork** in both + handlers (`PMIX_CHECK_NSPACE(jdata->nspace, + PRTE_PROC_MY_NAME->nspace)`). Keep the two policies distinct: a daemon + failure may kill the DVM; an app failure must not. +- **The elastic guards are load-bearing and gated on `prte_elastic_mode`.** + The shrink echo-guard (`!ALIVE && state >= TERMINATED`) and the grow + rollback (`prte_plm_base_grow_target_failed`) each `goto cleanup` to + skip the general daemon-loss abort. Removing or reordering them + re-introduces the double-decrement / spurious-abort bugs recorded in + the repo memory — do not touch without re-running the dockerswarm + grow/shrink harness. +- **`flag` (recoverable/continuous) decides notify-vs-abort** for every + application proc state. Preserve the `!PRTE_JOB_FLAG_ABORTED` "first + failure wins" guard and the `PMIX_RETAIN` on the recorded proc, or you + will either double-abort or free a proc that the later error report + still needs. +- **Every path must `PMIX_RELEASE(caddy)`.** The handlers are littered + with `goto cleanup`; a new early return that skips it leaks the caddy. +- **`FAILED_TO_START` self-check bug smell.** In `proc_errors` the line + `if (PRTE_PROC_STATE_FAILED_TO_START)` tests a constant (always true) + rather than `state == …`; it happens to be harmless because both arms + are reached only for the two FAILED states, but do not copy that + pattern — if you extend this branch, compare `state` explicitly. diff --git a/src/mca/errmgr/dvm/CLAUDE.md b/src/mca/errmgr/dvm/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/errmgr/dvm/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/errmgr/prted/AGENTS.md b/src/mca/errmgr/prted/AGENTS.md new file mode 100644 index 0000000000..7281add188 --- /dev/null +++ b/src/mca/errmgr/prted/AGENTS.md @@ -0,0 +1,203 @@ +# AGENTS.md — `errmgr/prted` (the daemon-side reporter) + +Component guide for `src/mca/errmgr/prted/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +state-machine callback model, and the progress-thread threading notes +referenced throughout. + +--- + +## Role and priority + +`prted` is the errmgr component that runs on **every prted daemon** (not +the HNP). Priority **1000** (MCA param `errmgr_prted_priority`), selected +by role: `errmgr_prted_component_query()` returns the module only when +`PRTE_PROC_IS_DAEMON`, otherwise `*module = NULL; *priority = -1; +return PRTE_ERROR`. Its job is the mirror image of the `dvm` component: +where `dvm` *decides* policy, `prted` *reports upward* — it detects local +process and daemon state changes, kills local children when required, +notifies the HNP over the RML, and exits cleanly when ordered. + +Files: + +| File | Contents | +|------|----------| +| `errmgr_prted_component.c` | Registration, `priority` MCA param (default 1000), `query` gated on `PRTE_PROC_IS_DAEMON`, open/close no-ops. | +| `errmgr_prted.c` | The module: `init`/`finalize`, handlers `job_errors`/`proc_errors`, the `prted_abort` distress path, and packing/killing helpers. | +| `errmgr_prted.h` | Exports `prte_mca_errmgr_prted_component` and `prte_errmgr_prted_module`. | + +--- + +## How it wires into the state machine (`init`) + +Same three registrations as `dvm`, but the handlers do daemon-local work: + +```c +prte_state.add_job_state(PRTE_JOB_STATE_ERROR, job_errors); +prte_state.add_proc_state(PRTE_PROC_STATE_COMM_FAILED, proc_errors); +prte_state.add_proc_state(PRTE_PROC_STATE_ERROR, proc_errors); +``` + +`finalize()` is a no-op. + +--- + +## The HNP report protocol + +Most of this component's work is packing a state update and sending it to +the HNP. The wire format built by the packing helpers is, in order: + +1. `PRTE_PLM_UPDATE_PROC_STATE` command (`PMIX_UINT8`). +2. the job nspace (`PMIX_PROC_NSPACE`). +3. for each relevant proc: **rank, pid, state, exit_code** + (`pack_state_for_proc`). +4. a terminator rank `PMIX_RANK_INVALID` marking the end / job complete. + +The buffer is delivered with `PRTE_RML_RELIABLE_SEND(rc, +PRTE_PROC_MY_HNP->rank, alert, PRTE_RML_TAG_PLM)`. `pack_state_update` +packs *all* local children of a job (steps 2–4); the single-proc paths +pack the nspace then one `pack_state_for_proc` then the terminator. + +A per-job **dedup flag**, `PRTE_JOB_FAIL_NOTIFIED`, ensures the daemon +reports a given job's failure to the HNP only once — except for +`PRTE_JOB_RECOVERABLE` jobs, which must receive *every* notification, so +the flag is deliberately not set for them. + +--- + +## `job_errors` — daemon's view of job-level failures + +After the standard acquire / `finalizing` / daemon-job back-fill and +`jdata->state = jobstate`, it switches on `jobstate`: + +- `FAILED_TO_START` → `failed_start(jdata)` (see helpers), then fall + through to send a state update to the HNP. +- `COMM_FAILED` → `killprocs(NULL, PMIX_RANK_WILDCARD)` (kill all local + procs) then `prted_abort(...)` to order our own termination; `goto + cleanup` (no HNP report — comms are gone). +- `HEARTBEAT_FAILED` → let the HNP handle it; `goto cleanup`. +- default → fall through. + +The fall-through path packs `PRTE_PLM_UPDATE_PROC_STATE` + +`pack_state_update(alert, jdata)` and reliable-sends it to the HNP. + +--- + +## `proc_errors` — daemon's view of process/daemon failures + +The core reporter. After acquire and the `finalizing` guard it filters in +order: + +1. **`HEARTBEAT_FAILED`** → ignore, the HNP owns it. +2. **Lifeline / unreachable family** (`LIFELINE_LOST`, + `UNABLE_TO_SEND_MSG`, `NO_PATH_TO_TARGET`, `PEER_UNKNOWN`, + `FAILED_TO_CONNECT`) → we've lost our lifeline to the HNP: set exit + status, `killprocs` all children, and `prte_quit()`. Our routed + children will see us leave and die on their own. +3. Look up `jdata`; if `NULL`, the job is already complete — ignore. +4. **`COMM_FAILED`:** + - to self → ignore. + - to a **non-daemon** (an application proc) → we can't trust we'll + catch its waitpid, so re-inject it as a waitpid event: build a + `prte_wait_tracker_t`, `PMIX_RETAIN` the child, and activate + `prte_odls_base_default_wait_local_proc` on the event base. This + reuses the normal local-termination path instead of duplicating it. + - to a **daemon** → if `prte_prteds_term_ordered`, check whether any + local child is still alive and whether routed children remain + (`prte_rml_base.n_children`); when all are gone activate + `DAEMONS_TERMINATED` to exit; otherwise just continue. +5. Look up the `child` proc_t; a `NULL` is a `PRTE_ERR_NOT_FOUND` and + forces `PRTE_JOB_STATE_FORCED_EXIT`. + +Then, per specific application-proc state: + +- **`CALLED_ABORT`** → record `child->state`; unless + `PRTE_JOB_FAIL_NOTIFIED`, pack just this proc and reliable-send to the + HNP, then set the dedup flag (skipped for recoverable jobs). +- **not local** → ignore (only the owning daemon reports a proc). +- **`TERM_NON_ZERO`** → same report-once-to-HNP dance; afterward, if the + proc is fully done (`IOF_COMPLETE && WAITPID && !RECORDED`), activate + `PRTE_PROC_STATE_TERMINATED`. +- **`FAILED_TO_START` / `FAILED_TO_LAUNCH`** → set state, bump + `jdata->num_terminated`, and **defer to the state machine**: only when + `num_local_procs == num_terminated` (all local procs have attempted + start) activate the corresponding job state so the HNP gets one + consolidated report. +- **`state > TERMINATED`** (abnormal) → if `prte_prteds_term_ordered`, + update the child's ALIVE/RECORDED bookkeeping and terminate the daemon + when all children/routes are gone (no HNP alert — we're already + leaving). Otherwise (`keep_going:`) report the abnormal termination to + the HNP once, set `PRTE_PROC_FLAG_TERM_REPORTED` and the + `PRTE_JOB_FAIL_NOTIFIED` dedup flag, and activate `TERMINATED` if the + proc is fully done. +- **plain `TERMINATED`** (the final `else`) → if no live children of the + job remain (`any_live_children`), pack a full job state update, **remove + this job's children from `prte_local_children` and `PMIX_RELEASE` the + jdata locally** (the job is complete on this node), then reliable-send + the update to the HNP. + +Note the `WAITPID`/`IOF_COMPLETE`/`RECORDED` flag triad gates the final +`PRTE_PROC_STATE_TERMINATED` activation — a proc is only "done" once both +its waitpid has fired and its I/O forwarding has drained, and it must not +be recorded twice. + +--- + +## `prted_abort` — the FORCED_TERMINATE distress path + +Called when an internal failure (e.g. a pack/unpack error) means the +daemon must die but a bare `exit()` would give the user no message. It: + +1. Runs at most once (`prte_abnormal_term_ordered` guard), then sets that + flag. +2. Emits the message via `help-errmgr-base.txt: simple-message`. +3. Packs a `PRTE_PLM_UPDATE_PROC_STATE` alert describing *itself* + (nspace, its own vpid, its pid, `PRTE_PROC_STATE_CALLED_ABORT`, the + error code, terminator rank) and reliable-sends it to the HNP on + `PRTE_RML_TAG_PLM`, giving mpirun the chance to order termination. +4. Sets a **5-second self-destruct timer** (`wakeup` → `prte_quit`) so + that if the messaging system itself is broken and the HNP never + replies, the daemon still exits. If the reliable send fails outright + it calls `prte_quit()` immediately. + +--- + +## Local helpers + +| Helper | Role | +|--------|------| +| `any_live_children(job)` | True if any child in `prte_local_children` for `job` (or any job, if nspace invalid) still has `PRTE_PROC_FLAG_ALIVE`. | +| `pack_state_for_proc(alert, child)` | Pack one proc's `{rank, pid, state, exit_code}`. | +| `pack_state_update(alert, jobdat)` | Pack nspace + every local child of `jobdat` + `PMIX_RANK_INVALID` terminator. | +| `failed_start(jobdat)` | Set the job `FAILED_TO_START`; for each local child in that state, force `IOF_COMPLETE`+`WAITPID` flags (so we don't hang on pipes that never opened) and activate `PRTE_PROC_STATE_TERMINATED`. | +| `killprocs(job, vpid)` | Kill local procs via `prte_odls.kill_local_procs` — all of them when `job` is invalid and `vpid == PMIX_RANK_WILDCARD`, else the single `{job, vpid}`. | + +--- + +## Things to watch when editing + +- **This component reports; it does not decide.** Keep DVM-wide policy + (terminate the DVM, notify submitters) out of here — that belongs to + `dvm`. The daemon's contract is: tell the HNP, manage local children, + and exit only when ordered or when its lifeline is lost. +- **`PRTE_JOB_FAIL_NOTIFIED` dedup, with the recoverable exception.** + Every "alert the HNP" branch checks the flag and sets it afterward — + *except* for `PRTE_JOB_RECOVERABLE` jobs, which must keep receiving + notifications so the set is deliberately skipped. Preserve both halves. +- **The `WAITPID`/`IOF_COMPLETE`/`RECORDED` flag triad** is what + prevents premature or double `TERMINATED` activation and double-counted + `num_terminated`. Don't collapse or reorder these checks. +- **`COMM_FAILED` to a non-daemon is re-routed as a waitpid**, not + handled inline, to avoid duplicating the odls termination path — keep + the `prte_wait_tracker_t` + `prte_odls_base_default_wait_local_proc` + hand-off intact (including the `PMIX_RETAIN` guarding the race). +- **`prted_abort` must stay idempotent and must always arm the timer.** + The 5-second `wakeup` timer is the only guarantee of exit when + messaging is itself broken; a refactor that skips it on some path can + wedge a daemon forever. +- **Watch the send-failure error paths.** Several branches `return` + directly on a pack error *without* releasing the caddy (they predate + the `goto cleanup` convention). If you touch those blocks, prefer + routing through `cleanup` so the caddy is released — but verify against + the current code, since a few paths intentionally `return` after + already having released or transferred ownership. diff --git a/src/mca/errmgr/prted/CLAUDE.md b/src/mca/errmgr/prted/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/errmgr/prted/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/ess/AGENTS.md b/src/mca/ess/AGENTS.md new file mode 100644 index 0000000000..0329bb24aa --- /dev/null +++ b/src/mca/ess/AGENTS.md @@ -0,0 +1,382 @@ +# AGENTS.md — The `ess` Framework (Environment-Specific Services) + +Orientation for AI agents and human contributors working in +`src/mca/ess/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`ess` (Environment-Specific Services) is the framework that **brings a +PRRTE process up and tears it down**. It answers one question: given the +environment I was launched into and the role I am supposed to play, +how do I initialize the entire runtime — my identity, my session +directory, the PMIx server, the communication stack, and every other +framework — and how do I shut it all back down cleanly? + +It runs in exactly two PRRTE process roles: + +| Role | Process | Selected component | +|------|---------|--------------------| +| **HNP / DVM master** | `prte` | `hnp` | +| **prted daemon** | `prted` | `env` (ssh/default), or an RM-specific module: `slurm`, `pals`, `lsf` | + +`ess` is the **earliest** framework to do real work in a process's life. +It is opened and its module selected inside `prte_init()` +(`src/runtime/prte_init.c`), immediately after `schizo` (which `ess` +uses to help pick a personality): + +``` +prte_init() + → open schizo, select schizo + → open ess, prte_ess_base_select() ← pick the one winning module + → prte_ess.init(argc, argv) ← THE bring-up: opens every other framework + ... +prte_finalize() + → prte_ess.finalize() ← tear-down in reverse + → close ess framework +``` + +Nothing else in the process is usable until `prte_ess.init()` returns. +The selected module's `init` is where `state`, `errmgr`, `plm`, +`grpcomm`, `odls`, `rmaps`, `ras` (HNP only), `iof`, `filem`, +`prtereachable`, and the `rml` are opened and selected, the PMIx server +is started, and the process's own job/proc/node data objects are +created. In other words, **`ess` is the orchestrator of startup**; the +other frameworks are its dependents. + +Note: PRRTE's user-facing *tools* (`prun`, `pterm`, `prte_info`) are +PMIx tools — they attach to a running DVM through the PMIx tool +interface and do **not** select an `ess` module. They may open the `ess` +framework for symbol availability, but `prte_ess_base_select()` / +`prte_ess.init()` are only exercised by `prte` (HNP) and `prted` +(daemon). Historically `ess` also carried tool/singleton components; +today the surviving components serve only the HNP and daemon roles. + +--- + +## Directory layout + +``` +ess/ + ess.h # module/component vtable: the init + finalize fn ptrs + base/ + base.h # framework-global struct, base API prototypes, signal class + ess_base_frame.c # framework open/close/register; MCA params; signal-forwarding infra + ess_base_select.c # prte_ess_base_select() — PICK-ONE highest-priority winner + ess_base_std_prolog.c # prte_ess_base_std_prolog() — dt_init + wait_init (all modules) + ess_base_std_prted.c # prte_ess_base_prted_setup/_finalize() — shared daemon bring-up/tear-down + ess_base_bootstrap.c # launcher-less bootstrap: parse config, publish identity, synth peer URIs + help-ess-base.txt # user-facing signal-forwarding error text + static-components.h # generated: the components statically linked into this build + hnp/ # HNP / DVM master (pri 100, gated on PRTE_PROC_IS_MASTER) + env/ # generic daemon, ssh-launched (pri 1, the daemon default) + slurm/ # daemon under SLURM (pri 50, gated on SLURM_JOBID + hnp_uri) + pals/ # daemon under HPE/Cray PALS aprun (pri 50, gated on PALS_APID + hnp_uri) + lsf/ # daemon under IBM LSF (pri 40, gated on LSB_JOBID + hnp_uri) +``` + +Read `ess.h` first (it is tiny — two function pointers), then +`base/ess_base_std_prted.c`, which is where most of the framework's +real work actually lives: every daemon module is a thin wrapper around +it. + +--- + +## The module contract + +An `ess` module is the thinnest vtable in the tree. Every component +exposes exactly two functions through `ess.h`: + +```c +typedef int (*prte_ess_base_module_init_fn_t)(int argc, char **argv); +typedef int (*prte_ess_base_module_finalize_fn_t)(void); + +struct prte_ess_base_module_3_0_0_t { + prte_ess_base_module_init_fn_t init; + prte_ess_base_module_finalize_fn_t finalize; +}; +``` + +| Function | Meaning | Return protocol | +|----------|---------|-----------------| +| `init(argc, argv)` | Bring the entire runtime up for this process in this environment: establish identity, discover topology, open/select all downstream frameworks, start the PMIx server. | `PRTE_SUCCESS`, or a `PRTE_ERR_*`. Modules typically return `PRTE_ERR_SILENT` after emitting their own `show_help` so `prte_init` does not double-report. | +| `finalize()` | Reverse of `init`: close the frameworks the module opened, kill local procs, shut down the PMIx server. | `PRTE_SUCCESS` (errors are logged with `PRTE_ERROR_LOG` but generally not propagated). | + +The component struct is a bare `pmix_mca_base_component_t` (typedef'd to +`prte_ess_base_component_t`) — `ess` has **no** component-level data +beyond the standard MCA header. The version macro is +`PRTE_ESS_BASE_VERSION_3_0_0` (declared in `ess.h`). + +The selected module's two function pointers are copied into the single +global: + +```c +PRTE_EXPORT extern prte_ess_base_module_t prte_ess; /* ess.h */ +``` + +`prte_init` calls `prte_ess.init(...)`; `prte_finalize` calls +`prte_ess.finalize()`. There is no per-component global structure kept +after selection (`ess_base_select.c` explicitly discards the winning +*component* and keeps only `*best_module`). + +--- + +## Component selection **is** "pick one" + +Unlike `rmaps` (which keeps every module priority-sorted), `ess` selects +a **single** winning module. `prte_ess_base_select()` +(`ess_base_select.c`) delegates to the generic `pmix_mca_base_select()`: +every component's `query` returns a priority, the highest wins, and its +module is copied into the `prte_ess` global: + +```c +prte_ess = *best_module; +``` + +Each component's `query` gates itself on the process role and the +environment, so at most one component ever claims a positive priority +for a given process: + +| Component | Priority | Gate (all must hold) | +|-----------|----------|----------------------| +| `hnp` | **100** | `PRTE_PROC_IS_MASTER` | +| `slurm` | **50** | `PRTE_PROC_IS_DAEMON` **and** `getenv("SLURM_JOBID")` **and** `prte_process_info.my_hnp_uri != NULL` | +| `pals` | **50** | `PRTE_PROC_IS_DAEMON` **and** `getenv("PALS_APID")` **and** `my_hnp_uri != NULL` | +| `lsf` | **40** | `PRTE_PROC_IS_DAEMON` **and** `getenv("LSB_JOBID")` **and** `my_hnp_uri != NULL` | +| `env` | **1** | `PRTE_PROC_IS_DAEMON` (always available to any daemon) | + +The logic is: the HNP is unambiguous (`hnp`, priority 100, only ever +selected in the `prte` process). For a daemon, the RM-specific modules +sit above the generic `env` default so that if a daemon *is* running +under SLURM/PALS/LSF with a path home to the HNP, the RM module wins and +`env` is the fallback for everything else (notably ssh-launched +daemons). A `query` that does not apply returns `priority = -1`, +`module = NULL`, and `PRTE_ERROR`, so it cannot be chosen. + +Because `lsf` and `pals` link against vendor libraries, they are only +*built* where those libraries are present (see each component's +`configure.m4` → `PRTE_CHECK_LSF` / `PRTE_CHECK_PALS`). On a platform +without them the framework's `static-components.h` will not list them at +all — do not assume every component compiled into your local build. + +--- + +## What `base/` provides + +The base is not a "default behavior" fallback the way some frameworks' +bases are — it is the **shared implementation** that the modules call +into. The daemon modules (`env`, `slurm`, `pals`, `lsf`) are almost +identical: each is little more than *set my name from the environment* +plus a call to the base's `prted_setup`. + +### `ess_base_frame.c` — framework plumbing + signal forwarding + +- Standard `PMIX_MCA_BASE_FRAMEWORK_DECLARE(prte, ess, ...)` with + register/open/close hooks. +- **MCA parameters** (registered in `prte_ess_base_register`), each with + a deprecated `prte_ess_*` synonym: + - `ess_base_nspace` → `prte_ess_base_nspace` — the namespace string a + daemon adopts as its identity. + - `ess_base_vpid` → `prte_ess_base_vpid` — the daemon's vpid (rank), + parsed with `strtoul`. + - `ess_base_num_procs` → `prte_ess_base_num_procs` — the number of + daemons in the DVM (becomes `prte_process_info.num_daemons`). + - `ess_base_forward_signals` → the comma-delimited list of signals to + forward to application processes (`"all"`, `"none"`, or names/ints). +- **Signal-forwarding infrastructure**, shared by every daemon: + - `known_signals[]` — a table mapping signal name → number → + `can_forward` flag. `SIGTERM`/`SIGHUP`/`SIGINT` are always handled + (not forwardable via this param); `SIGKILL`/`SIGPIPE` can never be + forwarded. + - `prte_ess_base_setup_signals(char *signals)` — parses the requested + list (handling `"none"`, `"all"`, names, and integers), rejects + unknown or non-forwardable signals via `help-ess-base.txt`, and + appends `prte_ess_base_signal_t` items onto the global + `prte_ess_base_signals` list. Guarded by a `signals_added` latch so + it only runs once. + - `prte_ess_base_signal_t` — a `pmix_list_item_t` subclass + (`signame`/`signal`/`can_forward`), instantiated here with + constructor/destructor. The actual libevent signal handlers that + consume this list are installed by `prte_ess_base_prted_setup()`. + +### `ess_base_select.c` — the winner + +`prte_ess_base_select()`: pick-one selection described above. ~20 lines. + +### `ess_base_std_prolog.c` — the universal preamble + +`prte_ess_base_std_prolog()`: the first thing **every** module's `init` +calls. It does the two things that must happen before anything else in +any role: + +1. `prte_dt_init()` — register PRRTE's data-type (pack/unpack) support. +2. `prte_wait_init()` — set up the `waitpid`/`SIGCHLD` machinery. + +On failure it shows `prte_init:startup:internal-failure` and returns the +error. + +### `ess_base_std_prted.c` — the shared daemon bring-up + +This is the heart of the framework for daemons. +`prte_ess_base_prted_setup()` is what `env`/`slurm`/`pals`/`lsf` all call +after setting their name. In order, it: + +1. Installs signal handlers: `SIGPIPE` (ignored), `SIGTERM`/`SIGINT` + (→ `shutdown_signal` → `PRTE_JOB_STATE_FORCED_EXIT`), and one + `signal_forward_callback` handler per entry on + `prte_ess_base_signals` (each forwards the signal to local procs by + sending a `PRTE_DAEMON_SIGNAL_LOCAL_PROCS` command to itself over the + RML). +2. Discovers the local hwloc topology if not already set. +3. Defines the HNP name (`PRTE_PROC_MY_HNP` = my nspace, rank 0). +4. Opens and selects `state`, opens `errmgr`. +5. Opens/selects `plm` **only if** `PRTE_MCA_plm` is set in the + environment (ssh-style remote launch); an ordinary prted has no need + of a PLM. +6. Creates the daemon job data object (`prte_job_t`), gives it the + `"prte"` schizo personality by default, adds one app context and one + proc object for itself, and marks the daemon job RUNNING/reported. +7. Creates the session directory tree, redirects `pmix_output` into it, + and (under `prte_debug_daemons_file_flag`) sends stdout/stderr to a + per-daemon log file. +8. Starts the PMIx server (`pmix_server_init` → later + `pmix_server_start`), gathers interface aliases. +9. Opens/selects the communication stack: `prtereachable`, `rml`. +10. Selects `errmgr`; opens/selects `grpcomm`, `odls`, `rmaps`. +11. Adds the local topology to `prte_node_topologies`. +12. If a PLM was opened, calls `prte_plm.init()` (must come after comms). +13. Opens/selects `iof` and `filem`. + +`prte_ess_base_prted_finalize()` reverses this: removes the signal +handlers, finalizes `errmgr`, closes `filem`/`grpcomm`/`iof`/`plm`, +kills local procs (`prte_odls.kill_local_procs`), closes +`odls`/`errmgr`, closes the `rml`, closes `prtereachable`/`state`, and +finalizes the PMIx server. + +The HNP does **not** use `prted_setup`; `ess/hnp` open a very similar but +distinct set of frameworks inline (it additionally opens `ras`, sets the +HNP name via the PLM, and sets up its own node object). See +[`hnp/AGENTS.md`](hnp/AGENTS.md). + +### `ess_base_bootstrap.c` — launcher-less DVM bootstrap + +Support for starting a DVM without a launcher (each node's `prted` +reads a shared bootstrap configuration file and self-assigns identity). +This is **not** part of a module `init`; it is called directly by +`prted.c` and `prte_init.c` at specific, timing-sensitive points: + +- `prte_ess_base_bootstrap_params()` — **phase 1**: parse the config + file (`prte_bootstrap_parse`) and publish the DVM-wide MCA parameters + (static ports, IP version/family, radix routing, retry backoff, + tmpdir, fqdn handling). Must run *before* `prte_register_params()` so + those variables read the environment on first registration; called + from `prte_init.c`. +- `prte_ess_base_bootstrap(bool *is_controller)` — **phase 2**: once the + local hostname is known, determine this node's role and rank + (`prte_bootstrap_my_identity`). The controller adopts nspace + `"-prte-dvm"` and rank 0; an ordinary daemon publishes its + identity through the `ess_base_*` params and synthesizes the + controller's contact URI so it can phone home before any nidmap + exists. Called from `prted.c`. +- `prte_ess_base_bootstrap_peer_uri(rank, &uri)` — synthesize the RML + contact URI of any peer daemon purely from the config, so a daemon can + reach a parent (or a re-parented grandparent after a lifeline heals) + before contact info has been distributed. Called from `prted.c` and + the OOB (`src/rml/oob/oob_base_stubs.c`). +- Helpers `parse_cidr`, `same_inaddr`, `pick_host_address`, `synth_uri` + build a correctly-shaped `;tcp://ip:port:mask` URI, using the + `DVMNetworks` CIDRs to disambiguate a multi-homed host (and failing + loudly rather than baking in a wrong interface). + +The parsed `bootstrap_cfg` is deliberately retained for the life of the +process (see the comment at the end of `prte_ess_base_bootstrap`) +because peer-URI synthesis can happen much later. See the repo memory on +the bootstrap DVM work. + +### Stale declarations — do not trust `base.h` blindly + +`base.h` still declares `prte_ess_env_get`, `prte_ess_env_put`, +`prte_ess_base_proc_binding`, and the `prte_ess_base_std_buffering` +variable, but **none of these are defined anywhere in the tree**. They +are vestigial. Do not wire new code to them expecting them to work, and +consider removing the declarations if you are cleaning up. + +--- + +## Global state and data structures + +| Symbol | Where | Meaning | +|--------|-------|---------| +| `prte_ess` | `ess_base_frame.c` | The selected module's `{init, finalize}`; the framework's only runtime entry point. | +| `prte_ess_base_framework` | `ess_base_frame.c` | The MCA framework object (its `framework_output` is the verbosity channel). | +| `prte_ess_base_nspace` / `_vpid` / `_num_procs` | `ess_base_frame.c` | Daemon identity, from MCA params/env; consumed by each daemon module's `*_set_name`. | +| `prte_ess_base_signals` | `ess_base_frame.c` | List of `prte_ess_base_signal_t` to forward; populated by `setup_signals`, consumed by `prted_setup`. | + +--- + +## Conventions and gotchas + +- **Daemon modules are near-clones.** `env`/`slurm`/`pals`/`lsf` each do: + `std_prolog` → `_set_name()` → `prte_ess_base_prted_setup()`, and + finalize with `prte_ess_base_prted_finalize()`. The *only* real + per-component logic is `*_set_name`: how the daemon derives its vpid + and nodename from that RM's environment. If you are adding a new + RM-launched daemon environment, that is the ~40-line function you + write; everything else is base. +- **`set_name` derives the true vpid.** The base params give a starting + vpid; the RM module adds a per-node offset (`SLURM_NODEID`, + `PALS_NODEID`, `LSF_PM_TASKID - 1`) so each daemon lands on a unique + rank. `env` uses the param verbatim (ssh launch assigns the vpid + directly). Getting this offset wrong collides daemon ranks — a nasty, + silent failure. +- **Selection is pick-one; keep `query` gates mutually exclusive.** A new + component must return a positive priority *only* for the precise + role+environment it serves, and `-1`/`PRTE_ERROR` otherwise, or you + will contend with an existing module. Match the RM gate idiom: + `PRTE_PROC_IS_DAEMON && getenv("") && my_hnp_uri != NULL`. +- **`init` errors should be `PRTE_ERR_SILENT` after a `show_help`.** The + modules emit their own `prte_init:startup:internal-failure` help and + return `PRTE_ERR_SILENT` (respecting `prte_report_silent_errors`) so + `prte_init` does not print a second, redundant message. On the `init` + error path, release any partially-built `jdata`. +- **Order is load-bearing in `prted_setup`.** Comms must be up before + `plm.init()`; the PMIx server must be init'd before gathering aliases; + IOF comes after routes. Do not reorder the framework opens casually. +- **Version macro is `PRTE_ESS_BASE_VERSION_3_0_0`.** Bump deliberately; + the module struct is `prte_ess_base_module_3_0_0_t`. +- Standard PRRTE rules still apply: `prte_config.h` first, braces on + every block, `NULL ==`/constant-on-left comparisons, no new compiler + warnings, `PRTE_ERROR_LOG` on unexpected errors. + +--- + +## Debugging + +```sh +prte --prtemca ess_base_verbose 5 ... # trace HNP bring-up +prted --prtemca ess_base_verbose 5 ... # (via the DVM) trace daemon bring-up +``` + +At verbosity ≥1 the daemon modules print the name they set for +themselves; `prted_setup` dumps session-dir setup at ≥2 and full +topology at >15; the HNP module dumps its node aliases at >0 and +topology at >15. Because `ess` runs so early, an `init` failure usually +surfaces as the `prte_init:startup:internal-failure` help message naming +the failing step (e.g. `prte_ess_base_prted_setup`, +`prte_state_base_select`) — that string is your first clue. + +--- + +## Where to go next + +Each component directory has its own `AGENTS.md`: + +- [`hnp/AGENTS.md`](hnp/AGENTS.md) — the DVM master; read this second. +- [`env/AGENTS.md`](env/AGENTS.md) — the generic ssh-launched daemon default. +- [`slurm/AGENTS.md`](slurm/AGENTS.md) — daemon under SLURM. +- [`pals/AGENTS.md`](pals/AGENTS.md) — daemon under HPE/Cray PALS. +- [`lsf/AGENTS.md`](lsf/AGENTS.md) — daemon under IBM LSF. diff --git a/src/mca/ess/CLAUDE.md b/src/mca/ess/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/ess/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/ess/env/AGENTS.md b/src/mca/ess/env/AGENTS.md new file mode 100644 index 0000000000..1ade2a6af5 --- /dev/null +++ b/src/mca/ess/env/AGENTS.md @@ -0,0 +1,100 @@ +# AGENTS.md — `ess/env` (the generic daemon default) + +Component guide for `src/mca/ess/env/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +pick-one selection model, and `prte_ess_base_prted_setup()`, which this +module is a thin wrapper around. + +--- + +## Role and priority + +`env` brings up a **generic `prted` daemon** — the fallback for any +daemon that is not running under a recognized resource manager. It is +the lowest-priority daemon component, priority **1**, so any +RM-specific module (`slurm`/`pals`/`lsf`) that recognizes its +environment outranks it. In practice `env` is what serves an +**ssh-launched** daemon (the `plm/ssh` component), where the daemon's +identity is handed to it directly via MCA parameters rather than derived +from an RM. + +Files: + +| File | Contents | +|------|----------| +| `ess_env_component.c` | Registration; `prte_mca_ess_env_component_query` returning priority 1 iff `PRTE_PROC_IS_DAEMON`. | +| `ess_env_module.c` | `rte_init` / `rte_finalize` + `env_set_name()`. | +| `ess_env.h` | Component struct + open/close/query prototypes. | + +--- + +## Selection (`prte_mca_ess_env_component_query`) + +```c +if (PRTE_PROC_IS_DAEMON) { + *priority = 1; + *module = &prte_ess_env_module; + return PRTE_SUCCESS; +} +*priority = -1; *module = NULL; return PRTE_ERROR; +``` + +It is available to **any** daemon (the source comment: "only used by +daemons that are launched by ssh so allow any enviro-specific modules to +override us"). Because its priority is 1 and every RM module gates on a +positive check with priority ≥40, `env` wins only when no RM module +claims the process. + +--- + +## `rte_init` — the minimal daemon path + +`rte_init` is the canonical example of a daemon module — three steps: + +1. **`prte_ess_base_std_prolog()`** — `prte_dt_init` + `prte_wait_init`. +2. **`env_set_name()`** — establish this daemon's identity from the base + MCA params (see below). +3. **`prte_ess_base_prted_setup()`** — the entire shared daemon bring-up + (signals, topology, state/errmgr/grpcomm/odls/rmaps/iof/filem/comms, + PMIx server, session dir, job/proc objects). See the framework guide. + +`rte_finalize` is just `prte_ess_base_prted_finalize()`. + +All the environment-specific logic in this component is the ~25-line +`env_set_name`. + +--- + +## `env_set_name` — identity from parameters + +Unlike the RM modules, `env` takes the daemon's vpid **verbatim** — there +is no per-node offset to add, because the ssh launcher assigns each +daemon a distinct vpid directly: + +1. Require `prte_ess_base_nspace` (from `ess_base_nspace`); load it into + `PRTE_PROC_MY_NAME->nspace`. Missing → `PRTE_ERR_NOT_FOUND`. +2. Require `prte_ess_base_vpid` (from `ess_base_vpid`); `strtoul` it into + `PRTE_PROC_MY_NAME->rank`. Missing → `PRTE_ERR_NOT_FOUND`. +3. Set `prte_process_info.num_daemons = prte_ess_base_num_procs`. + +These three parameters (`ess_base_nspace`, `ess_base_vpid`, +`ess_base_num_procs`) are the standard channel by which the HNP tells a +launched daemon who it is; they are set on the daemon's command +line/environment by the PLM. The launcher-less bootstrap path +(`ess_base_bootstrap.c`) publishes the same three parameters, so a +bootstrapped ordinary daemon also comes up through `env`. + +--- + +## Things to watch when editing + +- **No vpid offset here — that is deliberate.** If you find yourself + wanting to add `+ nodeid`, you are writing an RM module, not editing + `env`. Keep `env` the verbatim-identity default. +- **Return codes from `set_name` are swallowed.** `rte_init` calls + `env_set_name()` without checking its return (like the other daemon + modules). A missing nspace/vpid still logs via `PRTE_ERROR_LOG`, but + the subsequent `prted_setup` is what will actually fail loudly. If you + tighten this, tighten it consistently across all daemon modules. +- **This is the model to copy.** A new generic-daemon variant should + follow this exact shape: `std_prolog` → `set_name` → `prted_setup`. diff --git a/src/mca/ess/env/CLAUDE.md b/src/mca/ess/env/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/ess/env/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/ess/hnp/AGENTS.md b/src/mca/ess/hnp/AGENTS.md new file mode 100644 index 0000000000..05b0e50366 --- /dev/null +++ b/src/mca/ess/hnp/AGENTS.md @@ -0,0 +1,148 @@ +# AGENTS.md — `ess/hnp` (the DVM master) + +Component guide for `src/mca/ess/hnp/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +pick-one selection model, and the shared base functions referenced +throughout. + +--- + +## Role and priority + +`hnp` brings up the **HNP (Head Node Process) / DVM master** — the +`prte` process that controls the Distributed Virtual Machine. It is the +highest-priority `ess` component, priority **100**, and is selected in +exactly one process: the one where `PRTE_PROC_IS_MASTER` is true. No +other role ever selects it, and it never contends with the daemon +modules (they gate on `PRTE_PROC_IS_DAEMON`). + +Files: + +| File | Contents | +|------|----------| +| `ess_hnp_component.c` | Registration; `hnp_component_query` returning priority 100 iff `PRTE_PROC_IS_MASTER`. | +| `ess_hnp_module.c` | `rte_init` / `rte_finalize` — the full HNP bring-up and tear-down. | +| `ess_hnp.h` | Declares the component struct only (no module-private API). | + +Unlike the daemon modules, `hnp` does **not** call the shared +`prte_ess_base_prted_setup()`. It opens a similar but distinct set of +frameworks inline in `rte_init`, because the master has extra +responsibilities (it names itself via the PLM, discovers the allocation +via `ras`, and builds its own node object). + +--- + +## Selection (`hnp_component_query`) + +```c +if (PRTE_PROC_IS_MASTER) { + *priority = 100; + *module = &prte_ess_hnp_module; + return PRTE_SUCCESS; +} +*priority = -1; *module = NULL; return PRTE_ERROR; +``` + +The `PRTE_PROC_MASTER` bit in `prte_process_info.proc_type` is set by the +`prte` tool during its own startup, before `prte_init` opens `ess`. So by +the time `query` runs, the process already knows it is the master. + +--- + +## `rte_init` — HNP bring-up + +`rte_init(argc, argv)` in `ess_hnp_module.c` is the master's startup +sequence. Its ordering matters; the highlights, in order: + +1. **`prte_ess_base_std_prolog()`** — `prte_dt_init` + `prte_wait_init` + (shared with every module). +2. **Topology discovery** — `prte_hwloc_base_get_topology()` if not + already set. +3. **`state`** framework open + select. +4. **`errmgr`** framework open (selected later, after comms). +5. **`plm`** open + select, then **`prte_plm.set_hnp_name()`** — the HNP + name (nspace + rank 0) is defined by the PLM component for this + environment, which is why `plm` must be opened this early. A + `PRTE_ERR_FATAL` from `plm` select is downgraded to `PRTE_ERR_SILENT` + (the PLM already showed help). +6. **Daemon job object** — create the `prte_job_t` for the daemon job, + register it (`prte_set_job_data_object`), assign the `"prte"` schizo + personality by default, attach it to `prte_default_session`, and mark + it `PRTE_JOB_STATE_DAEMONS_REPORTED`. Add one app context (argv[0] + + argv), and build the HNP's own `prte_node_t` (this node) and + `prte_proc_t` (this daemon), wiring the proc into the node's `daemon` + field and flagging the node UP / DAEMON_LAUNCHED / LOC_VERIFIED. The + daemon job is marked RUNNING with one proc reported. +7. **Session directory** — `prte_session_dir(PRTE_PROC_MY_NAME)`. +8. **PMIx server** — `pmix_server_init()`, then gather interface aliases + (`pmix_ifgetaliases`) and copy them onto the node object + (`node->aliases`). Emit the XML start tag if `prte_xml_output`. +9. **Comms** — open/select `prtereachable`, `prte_rml_open()`, then + `pmix_server_start()`. +10. **`grpcomm`** open + select, then **`errmgr` select**. +11. **`prte_plm.init()`** — module-specific PLM init, after comms (may + start a non-blocking recv). +12. **`ras`** open + select — resource allocation discovery. This is a + key HNP-only step the daemon path never performs. +13. **`rmaps`** open + select. Add the local topology to + `prte_node_topologies`, set `node->topology` and + `node->available` (the filtered CPU set). +14. **`odls`** open + select. +15. Redirect `pmix_output` into the proc-specific session dir. +16. **`iof`** and **`filem`** open + select. + +On any failure it shows `prte_init:startup:internal-failure` (unless +`PRTE_ERR_SILENT`/`prte_report_silent_errors`), releases the partially +built `jdata` (which cleans up the session directory tree), and returns +`PRTE_ERR_SILENT`. + +The frameworks the HNP opens but the daemon path does **not** are +`ras` and (unconditionally) `plm` — the master must discover the +allocation and name/launch daemons. Conversely, the daemon path opens +`plm` only when `PRTE_MCA_plm` is set. + +--- + +## `rte_finalize` — HNP tear-down + +Reverse order: finalize `errmgr`; close +`filem`/`grpcomm`/`iof`/`plm`; kill local procs +(`prte_odls.kill_local_procs`) unless `prte_abnormal_term_ordered`; close +`odls`; close the `rml`; close `prtereachable`/`errmgr`/`state`; emit the +XML end tag; finalize the PMIx server; flush stdout/stderr. + +--- + +## Key structs the HNP builds + +The HNP is the only role that constructs the *authoritative* DVM data +structures for itself at init: + +- `prte_job_t` for the daemon job (nspace = my nspace), stored via + `prte_set_job_data_object`, tied to `prte_default_session`. +- `prte_node_t` for this node, placed in `prte_node_pool` at + `PRTE_PROC_MY_NAME->rank`, with `daemon`, `topology`, `available`, + `aliases`, and the UP/LAUNCHED/VERIFIED flags all set. +- `prte_proc_t` for this daemon, in the job's `procs` array and + cross-referenced from `node->daemon`. + +These are the seeds the rest of the launch machinery grows: `ras` +populates more nodes, `rmaps` maps jobs onto them, `plm` launches +daemons onto them. + +--- + +## Things to watch when editing + +- **Do not reorder the early PLM steps.** The HNP name comes from + `prte_plm.set_hnp_name()`, so `plm` must be opened/selected before the + job/proc/node objects (which key off `PRTE_PROC_MY_NAME`) are built. +- **`ras` and `rmaps` are HNP-only.** Keep allocation/mapping bring-up + here, not in the shared `prted_setup` — daemons must never open them. +- **Session-dir cleanup rides on `jdata`.** The error path releases + `jdata` to tear down the session directory tree; if you add new + early-out paths before `jdata` exists, guard the `NULL != jdata` check + (already present) and do not leak the tree. +- **This module intentionally diverges from `prted_setup`.** Resist the + urge to "unify" it with the daemon path — the master's extra steps + (self node object, ras, plm naming) are the whole point. diff --git a/src/mca/ess/hnp/CLAUDE.md b/src/mca/ess/hnp/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/ess/hnp/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/ess/lsf/AGENTS.md b/src/mca/ess/lsf/AGENTS.md new file mode 100644 index 0000000000..21860c8fe6 --- /dev/null +++ b/src/mca/ess/lsf/AGENTS.md @@ -0,0 +1,92 @@ +# AGENTS.md — `ess/lsf` (daemon under IBM LSF) + +Component guide for `src/mca/ess/lsf/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +pick-one selection model, and `prte_ess_base_prted_setup()`, which this +module wraps. + +--- + +## Role and priority + +`lsf` brings up a **`prted` daemon launched under IBM Spectrum LSF**. +Priority **40** — above the generic `env` default (1) but below +`slurm`/`pals` (50). The environment gates are mutually exclusive in +practice, so the ordering rarely matters; it exists so that if two RM +markers were somehow both present, the choice is deterministic. + +Files: + +| File | Contents | +|------|----------| +| `ess_lsf_component.c` | Registration; `prte_mca_ess_lsf_component_query` (priority 40 under LSF). | +| `ess_lsf_module.c` | `rte_init` / `rte_finalize` + `lsf_set_name()`. | +| `ess_lsf.h` | Component struct + open/close/query prototypes. | +| `configure.m4` | Gates the build on `PRTE_CHECK_LSF` — only built where the LSF libraries are present. | + +Because `configure.m4` gates on `PRTE_CHECK_LSF` (and sets +`ess_lsf_CPPFLAGS`/`LDFLAGS`/`LIBS`), this component is **only compiled +where LSF is available**. On a platform without LSF it will not appear in +the framework's `static-components.h`. + +--- + +## Selection (`prte_mca_ess_lsf_component_query`) + +```c +if (PRTE_PROC_IS_DAEMON && NULL != getenv("LSB_JOBID") + && NULL != prte_process_info.my_hnp_uri) { + *priority = 40; + *module = &prte_ess_lsf_module; + return PRTE_SUCCESS; +} +``` + +All three: we are a daemon, we are inside an LSF batch job (`LSB_JOBID`), +and we have a home URI to the HNP. + +--- + +## `rte_init` — the LSF daemon path + +The standard three-step daemon shape: + +1. `prte_ess_base_std_prolog()`. +2. `lsf_set_name()` — LSF-specific identity. +3. `prte_ess_base_prted_setup()` — the shared bring-up. + +`rte_finalize` is `prte_ess_base_prted_finalize()`. + +--- + +## `lsf_set_name` — identity with a 1-based LSF task offset + +1. Require `prte_ess_base_nspace`; load into `PRTE_PROC_MY_NAME->nspace`. +2. Require `prte_ess_base_vpid`; `strtoul` it to a base `vpid`. +3. **`PRTE_PROC_MY_NAME->rank = vpid + atoi(getenv("LSF_PM_TASKID")) - 1`** + — note the **`- 1`**: `LSF_PM_TASKID` is **1-based** (LSF's process + manager numbers tasks from 1), so it is decremented to a 0-based + offset before being added to the base vpid. This is the single most + important detail in the file; the other RM modules + (`slurm`/`pals`) use 0-based node ids and do not subtract. +4. Set `prte_process_info.num_daemons = prte_ess_base_num_procs`. + +Like `pals` (and unlike `slurm`), `lsf` does **not** rewrite +`prte_process_info.nodename`. + +--- + +## Things to watch when editing + +- **The `- 1` is not a typo.** `LSF_PM_TASKID` counts from 1; dropping + the decrement shifts every daemon's rank by one and collides ranks. + This is the classic bug to avoid here. +- **Build gating.** Any new LSF symbol must be covered by + `PRTE_CHECK_LSF` in `configure.m4`, or the build breaks on non-LSF + systems. The wrapper flags (`ess_lsf_CPPFLAGS`/`LDFLAGS`/`LIBS`) are + substituted from there. +- **Minor quirk:** `rte_finalize` has a stray trailing empty statement + (`;` after `return`); harmless dead code, but if you touch the file it + is worth cleaning up. +- Daemon-only. LSF allocation/launch integration lives in the `ras`/`plm` + frameworks; this component is only the daemon's own RTE bring-up. diff --git a/src/mca/ess/lsf/CLAUDE.md b/src/mca/ess/lsf/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/ess/lsf/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/ess/pals/AGENTS.md b/src/mca/ess/pals/AGENTS.md new file mode 100644 index 0000000000..0899079445 --- /dev/null +++ b/src/mca/ess/pals/AGENTS.md @@ -0,0 +1,91 @@ +# AGENTS.md — `ess/pals` (daemon under HPE/Cray PALS) + +Component guide for `src/mca/ess/pals/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +pick-one selection model, and `prte_ess_base_prted_setup()`, which this +module wraps. + +--- + +## Role and priority + +`pals` brings up a **`prted` daemon launched by the PALS variant of +`aprun`** (HPE/Cray's Parallel Application Launch Service). Priority +**50** — above the generic `env` default (1) and above `lsf` (40), tied +with `slurm`; the mutually exclusive environment gates keep them from +actually contending. + +Files: + +| File | Contents | +|------|----------| +| `ess_pals_component.c` | Registration; `prte_mca_ess_pals_component_query` (priority 50 under PALS). | +| `ess_pals_module.c` | `rte_init` / `rte_finalize` + `pals_set_name()`. | +| `ess_pals.h` | Component struct + open/close/query prototypes. | +| `configure.m4` | Gates the build on `PRTE_CHECK_PALS` — only built where PALS is present. | + +Because `configure.m4` gates on `PRTE_CHECK_PALS`, this component is +**only compiled where the PALS environment/headers are available**. On a +platform without PALS it will not appear in the framework's +`static-components.h` at all. + +--- + +## Selection (`prte_mca_ess_pals_component_query`) + +```c +if (PRTE_PROC_IS_DAEMON && NULL != getenv("PALS_APID") + && NULL != prte_process_info.my_hnp_uri) { + *priority = 50; + *module = &prte_ess_pals_module; + return PRTE_SUCCESS; +} +``` + +All three: we are a daemon, we are inside a PALS application +(`PALS_APID`, the PALS application id), and we have a home URI to the +HNP. Note the gate uses `PALS_APID` while identity uses `PALS_NODEID` +(below) — different variables for different purposes. + +--- + +## `rte_init` — the PALS daemon path + +The standard three-step daemon shape: + +1. `prte_ess_base_std_prolog()`. +2. `pals_set_name()` — PALS-specific identity. +3. `prte_ess_base_prted_setup()` — the shared bring-up. + +`rte_finalize` is `prte_ess_base_prted_finalize()`. + +--- + +## `pals_set_name` — identity with a PALS node offset + +1. Require `prte_ess_base_nspace`; load into `PRTE_PROC_MY_NAME->nspace`. +2. Require `prte_ess_base_vpid`; `strtoul` it to a base `vpid`. +3. **`PRTE_PROC_MY_NAME->rank = vpid + atoi(getenv("PALS_NODEID"))`**, + but only if `PALS_NODEID` is present — if it is **not** set, + `pals_set_name` returns `PRTE_ERR_NOT_FOUND` rather than defaulting + the offset to 0. (Contrast `slurm`, which calls `atoi` on + `SLURM_NODEID` unconditionally.) +4. Set `prte_process_info.num_daemons = prte_ess_base_num_procs`. + +Unlike `slurm`, `pals` does **not** rewrite `prte_process_info.nodename` +— it trusts the hostname already established during `prte_init`. + +--- + +## Things to watch when editing + +- **`PALS_NODEID` gates identity, `PALS_APID` gates selection.** They are + distinct variables; do not conflate them. A daemon can be selected + (has `PALS_APID`) yet fail `set_name` if `PALS_NODEID` is absent. +- **The node offset is load-bearing**, exactly as in `slurm`: the base + vpid is shared, and `PALS_NODEID` disambiguates each daemon's rank. +- **Build gating.** Any new PALS dependency must be reflected in + `configure.m4`'s `PRTE_CHECK_PALS`; do not introduce a hard PALS + reference that breaks the build on non-PALS systems. +- This component is daemon-only. PALS allocation/launch integration lives + in the `ras`/`plm` frameworks; here we only bring the daemon's RTE up. diff --git a/src/mca/ess/pals/CLAUDE.md b/src/mca/ess/pals/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/ess/pals/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/ess/slurm/AGENTS.md b/src/mca/ess/slurm/AGENTS.md new file mode 100644 index 0000000000..a6161dd4d8 --- /dev/null +++ b/src/mca/ess/slurm/AGENTS.md @@ -0,0 +1,98 @@ +# AGENTS.md — `ess/slurm` (daemon under SLURM) + +Component guide for `src/mca/ess/slurm/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +pick-one selection model, and `prte_ess_base_prted_setup()`, which this +module wraps. + +--- + +## Role and priority + +`slurm` brings up a **`prted` daemon that was launched by `srun`** as +part of an mpirun-in-SLURM job. Priority **50** — above the generic +`env` default (1) and above `lsf` (40), tied with `pals`. It is selected +only when all three conditions hold, so it never contends with `pals` +(which requires `PALS_APID`) or `lsf` (`LSB_JOBID`) in practice. + +Files: + +| File | Contents | +|------|----------| +| `ess_slurm_component.c` | Registration; `prte_mca_ess_slurm_component_query` (priority 50 under SLURM). | +| `ess_slurm_module.c` | `rte_init` / `rte_finalize` + `slurm_set_name()`. | +| `ess_slurm.h` | Component struct + open/close/query prototypes. | +| `configure.m4` | Builds unconditionally (SLURM support is env-var based; no vendor library). | + +Note: the `ess/slurm` component is **always built** — detecting SLURM is +just reading environment variables, so there is no `PRTE_CHECK_SLURM` +gate in its `configure.m4` the way `lsf`/`pals` gate on their libraries. + +--- + +## Selection (`prte_mca_ess_slurm_component_query`) + +```c +if (PRTE_PROC_IS_DAEMON && NULL != getenv("SLURM_JOBID") + && NULL != prte_process_info.my_hnp_uri) { + *priority = 50; + *module = &prte_ess_slurm_module; + return PRTE_SUCCESS; +} +``` + +All three must hold: we are a daemon, we are inside a SLURM allocation +(`SLURM_JOBID`), and we were given a path home to the HNP +(`my_hnp_uri`). The last condition is what distinguishes "launched by +mpirun under SLURM" from merely "a SLURM allocation exists" — without a +home URI there is nothing for this daemon to attach to. + +--- + +## `rte_init` — the SLURM daemon path + +Identical three-step shape to every daemon module: + +1. `prte_ess_base_std_prolog()`. +2. `slurm_set_name()` — SLURM-specific identity. +3. `prte_ess_base_prted_setup()` — the shared bring-up. + +`rte_finalize` is just `prte_ess_base_prted_finalize()`. + +--- + +## `slurm_set_name` — identity with a SLURM node offset + +The only SLURM-specific logic. It differs from `env` by adding a +**per-node vpid offset** so that each `srun`-placed daemon lands on a +unique rank, and by correcting the nodename from SLURM's own value: + +1. Require `prte_ess_base_nspace`; load into `PRTE_PROC_MY_NAME->nspace`. +2. Require `prte_ess_base_vpid`; `strtoul` it to a base `vpid`. +3. **`PRTE_PROC_MY_NAME->rank = vpid + atoi(getenv("SLURM_NODEID"))`** — + the base vpid plus this node's SLURM node id. This is the crucial + difference from `env`: a single base vpid is broadcast to all + daemons, and each adds its `SLURM_NODEID` to get a distinct rank. +4. Replace `prte_process_info.nodename` with `getenv("SLURMD_NODENAME")` + so the daemon's hostname matches exactly what SLURM reports (missing + → `PRTE_ERR_NOT_FOUND`). This keeps node matching consistent with the + allocation the HNP saw. +5. Set `prte_process_info.num_daemons = prte_ess_base_num_procs`. + +--- + +## Things to watch when editing + +- **The `SLURM_NODEID` offset is load-bearing.** Getting it wrong (or + dropping it) collides daemon ranks — a silent, miserable failure. The + base vpid is the same for every daemon; the node id is what + disambiguates. +- **`SLURMD_NODENAME` correction matters for node matching.** The HNP's + allocation (from `ras/slurm`) uses SLURM's node names; if the daemon + reports a different hostname (e.g. an FQDN vs short name), node + reconciliation can fail. Do not remove the rename. +- **Don't add a library dependency.** SLURM support here is purely + environmental; keep it that way so the component stays always-built. +- The `slurm` module is `PRTE_PROC_IS_DAEMON`-only. SLURM *allocation* + discovery for the HNP lives in `ras/slurm`, and daemon *launch* in + `plm/slurm` — this component is only the daemon's own RTE bring-up. diff --git a/src/mca/ess/slurm/CLAUDE.md b/src/mca/ess/slurm/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/ess/slurm/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/filem/AGENTS.md b/src/mca/filem/AGENTS.md new file mode 100644 index 0000000000..856b47bffc --- /dev/null +++ b/src/mca/filem/AGENTS.md @@ -0,0 +1,267 @@ +# AGENTS.md — The `filem` Framework (File Management) + +Orientation for AI agents and human contributors working in +`src/mca/filem/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`filem` (File Management) **pre-positions files across the DVM before a +job launches.** When a user asks for their executable and/or data files +to be staged out to every node — `prun --preload-binary`, +`--preload-files a,b,c` (which set the `PRTE_APP_PRELOAD_BIN` and +`PRTE_APP_PRELOAD_FILES` app-context attributes) — `filem` is what +actually moves the bytes from the HNP to the daemons and then links them +into each local process's session directory so the app finds them by a +relative path. + +Unlike most PRRTE frameworks, `filem` runs on **both** ends of the DVM: + +- **HNP (DVM master)** — orchestrates. It scans the job's app contexts + for preload requests, reads the source files, and broadcasts their + contents to every daemon. +- **prted (per-node daemon)** — receives. Each daemon writes the bytes + into its node's top session directory, unpacks archives, and later + symlinks the staged files into the session directory of every local + process in the job. + +### Place in the launch flow + +Pre-positioning happens on the DVM state machine at +`PRTE_JOB_STATE_VM_READY`, **before mapping**: + +``` +… → PRTE_JOB_STATE_VM_READY → (filem preposition) → PRTE_JOB_STATE_MAP → … +``` + +`vm_ready()` in `src/mca/state/dvm/state_dvm.c` calls +`prte_filem.preposition_files(jdata, files_ready, jdata)`. The transfer +is **asynchronous**: `preposition_files` returns immediately, and when +every daemon has acknowledged every file the framework fires the +`files_ready` completion callback, which advances the job to +`PRTE_JOB_STATE_MAP` (or `PRTE_JOB_STATE_FILES_POSN_FAILED` on error). +The same entry point is also reached from the PLM launch-support path +(`src/mca/plm/base/plm_base_launch_support.c`). + +Then, much later, when a daemon is about to fork the local application +processes, `odls` calls the second framework entry point, +`prte_filem.link_local_files(jdata, app)` +(`src/mca/odls/base/odls_base_default_fns.c`), to create the per-proc +symlinks. + +--- + +## Directory layout + +``` +filem/ + filem.h # module/component vtable + the request/file-set/process-set classes + base/ + base.h # framework-global decls, the "none" no-op prototypes, base comm API + filem_base_frame.c # framework open/close; the default "none" prte_filem module + filem_base_select.c # component selection — classic PICK-ONE (single winner) + filem_base_fns.c # the request/file-set/process-set PMIX_CLASS_INSTANCEs + all "none" no-ops + filem_base_receive.c # dormant base RML service: remote-path / node-name query commands + owner.txt # owner/status (INTEL, maintenance) + raw/ # the ONLY component (pri 0): xcast-based chunked staging +``` + +Read `filem.h` first — it defines the data structures (`request`, +`file_set`, `process_set`) and the module vtable. Then read the `raw` +component, which is where all the real work lives; the base provides +mostly the class definitions and a stack of no-op fallbacks. + +--- + +## The module contract + +A `filem` module is a `prte_filem_base_module_t` (alias for +`prte_filem_base_module_1_0_0_t`, in `filem.h`). Its vtable is broad — +it carries a classic put/get/rm/wait file-transfer API **and** the +higher-level preposition/link API — but in practice only two of these +functions do anything today (see the raw component). Every function +pointer: + +| Field | Signature | Meaning | Return | +|-------|-----------|---------|--------| +| `filem_init` | `int (void)` | Module init (called by `select` on the winner). | `PRTE_SUCCESS` | +| `filem_finalize` | `int (void)` | Module teardown (called on framework close). | `PRTE_SUCCESS` | +| `fault_handler` | `void (const prte_rml_recovery_status_t *)` | React to a daemon failure during transfer. | — | +| `put` / `put_nb` | `int (prte_filem_base_request_t *)` | Push file(s) to remote proc(s), blocking / async. | `PRTE_SUCCESS`/`PRTE_ERROR` | +| `get` / `get_nb` | `int (prte_filem_base_request_t *)` | Pull file(s) from remote proc(s), blocking / async. | `PRTE_SUCCESS`/`PRTE_ERROR` | +| `rm` / `rm_nb` | `int (prte_filem_base_request_t *)` | Remove remote file(s), blocking / async. | `PRTE_SUCCESS`/`PRTE_ERROR` | +| `wait` | `int (prte_filem_base_request_t *)` | Block until one async request completes. | `PRTE_SUCCESS`/`PRTE_ERROR` | +| `wait_all` | `int (pmix_list_t *)` | Block until a list of async requests completes. | `PRTE_SUCCESS`/`PRTE_ERROR` | +| `preposition_files` | `int (prte_job_t *, prte_filem_completion_cbfunc_t, void *)` | **Stage a whole job's preload files to every node (async).** | `PRTE_SUCCESS`, callback on completion | +| `link_local_files` | `int (prte_job_t *, prte_app_context_t *)` | **Symlink already-staged files into each local proc's session dir.** | `PRTE_SUCCESS`/error | + +The completion callback type is +`typedef void (*prte_filem_completion_cbfunc_t)(int status, void *cbdata)`. + +**Only `preposition_files` and `link_local_files` are actually invoked +anywhere in PRRTE today** (from the state machine / PLM and from odls, +respectively). The put/get/rm/wait half of the vtable is legacy FileM +API that the `raw` module deliberately wires to the base "none" no-ops. +If you are adding file-staging behavior, you almost certainly want to +work through the preposition/link pair, not put/get. + +### The version macro + +Components declare `PRTE_FILEM_BASE_VERSION_2_0_0` +(`PRTE_MCA_BASE_VERSION_3_0_0("filem", 2, 0, 0)`). Note the module +*struct* is still named `..._1_0_0_t`; the framework version and the +struct version are independent numbers here. + +--- + +## Component selection is "pick one" + +`prte_filem_base_select()` (in `filem_base_select.c`) is a **classic +single-winner selection**, unlike `rmaps`: it calls `pmix_mca_base_select`, +copies the highest-priority component's module into the global +`prte_filem`, and runs its `filem_init`. If **no** component is selected +it is not an error — the framework simply keeps the default **"none"** +module (all no-ops) that `filem_base_frame.c` statically installs into +`prte_filem`. Selection is driven from the HNP and daemon ESS init +(`src/mca/ess/hnp/ess_hnp_module.c`, +`src/mca/ess/base/ess_base_std_prted.c`). + +Today the tree ships exactly one component, `raw`, whose `query` returns +priority **0**. So in a normal build `raw` always wins; the "none" +module is what you get only if `raw` is unbuilt/unselected. + +--- + +## What `base/` provides + +The base is thin. It contributes the data classes, the no-op fallback +module, and a (currently dormant) RML query service. + +### Data structures (`filem.h` + `filem_base_fns.c`) + +Three reference-counted PMIx classes model a transfer request. They are +constructed/destructed in `filem_base_fns.c` via `PMIX_CLASS_INSTANCE`: + +- **`prte_filem_base_process_set_t`** — a `{source, sink}` pair of + `pmix_proc_t`s naming who a file moves *from* and *to*. Wildcards mean + "all procs of a job"; INVALID means "not applicable". Constructed to + `PRTE_NAME_INVALID` on both ends. +- **`prte_filem_base_file_set_t`** — one `{local_target, remote_target}` + file pairing, plus `app_idx`, local/remote **hints** + (`PRTE_FILEM_HINT_NONE`/`SHARED`), and a **`target_flag`** file-type + code (`PRTE_FILEM_TYPE_FILE`/`DIR`/`TAR`/`BZIP`/`GZIP`/`EXE`/`UNKNOWN`). + The destructor frees both path strings. +- **`prte_filem_base_request_t`** — a whole request: a list of process + sets, a list of file sets, plus internal bookkeeping arrays + (`is_done`, `is_active`, `exit_status`, `num_mv`) and a + `movement_type` (`PUT`/`GET`/`RM`/`UNKNOWN`). The destructor drains and + releases both lists and frees the bookkeeping arrays. + +The `raw` component largely ignores `process_set`/`request` and works +directly with `file_set` plus its own private classes; these types +exist because the put/get API is defined in terms of them. + +### The "none" module (`filem_base_fns.c` + `filem_base_frame.c`) + +`filem_base_frame.c` statically initializes the global `prte_filem` to a +module made entirely of the `prte_filem_base_none_*` functions from +`filem_base_fns.c`. Each of these is a no-op that returns `PRTE_SUCCESS` +(the `preposition` no-op politely fires the completion callback with +`PRTE_SUCCESS` so the state machine still advances). This is what runs +when file staging is disabled or unselected — pre-positioning simply +does nothing and the job proceeds. + +### The dormant base RML service (`filem_base_receive.c`) + +`base.h` declares a small RML service — `prte_filem_base_comm_start()`, +`prte_filem_base_comm_stop()`, and the `prte_filem_base_recv()` handler +— that answers two commands on `PRTE_RML_TAG_FILEM_BASE`: + +- `PRTE_FILEM_GET_PROC_NODE_NAME_CMD` — given a `pmix_proc`, reply with + the name of the node that proc is on (looked up via + `prte_get_job_data_object` / the job's proc array). +- `PRTE_FILEM_GET_REMOTE_PATH_CMD` — given a filename, resolve it to an + absolute path (prepending `getcwd` if relative), `stat` it, and reply + with the absolute path plus a file-type code + (`FILE`/`DIR`/`UNKNOWN`). + +This is scaffolding for a put/get-style component that needs to +negotiate remote absolute paths before transferring. **No code in the +tree currently calls `prte_filem_base_comm_start`, and the `raw` +component runs its own receives instead**, so these handlers are +effectively dead today — accurate to note, and a place to be careful: +`raw` and this base service would both claim `PRTE_RML_TAG_FILEM_BASE` +if the base service were ever started. The global +`prte_filem_base_is_active` bool is likewise defined but currently +unused. + +--- + +## Threading & the async transfer model + +`filem` does real asynchronous I/O on the progress thread, so it follows +the caddy/threadshift pattern described in the top-level `AGENTS.md`. The +`preposition_files` API is non-blocking: it kicks off work and returns, +and a **completion callback** (`prte_filem_completion_cbfunc_t`) is +invoked once every daemon has acknowledged every file. Do not block the +progress thread waiting for a transfer; register a callback and let the +state machine advance from there. + +The concrete mechanics — chunked reads driven by libevent write events, +`xcast` broadcast to daemons, per-daemon ack counting — all live in the +`raw` component and are documented in its guide. The key framework-level +contract is just: **`preposition_files` is fire-and-forget with a +completion callback; `link_local_files` is synchronous and runs on the +daemon at fork time.** + +--- + +## Conventions & gotchas + +- **Preload paths are forced relative.** Staged files always land under a + node's session directory; the framework rewrites absolute source paths + to relative remote targets so a stray `/etc/...` can never be + overwritten on a remote node. Preserve that safety property. +- **`preposition_files` MUST fire its callback on every exit path** + (including "nothing to stage" and error), or the job wedges at + `VM_READY` forever. The "none" module and `raw` both take care to do + this. +- **Two entry points, two ends of the DVM.** `preposition_files` runs on + the HNP; `link_local_files` runs on the daemon. Don't assume a single + address space or that HNP-side state is visible daemon-side — the only + channel between them is the RML/xcast traffic. +- **Selection failure is not an error.** Leaving `prte_filem` as the + "none" module is a supported configuration; never treat a missing + component as fatal. +- Standard PRRTE rules still apply: `prte_config.h` first, braces on + every block, `NULL ==`/constant-on-left comparisons, no new compiler + warnings, `PRTE_ERROR_LOG`/`PRTE_ACTIVATE_JOB_STATE` for errors. + +--- + +## Debugging + +```sh +prte --prtemca filem_base_verbose 5 ... # trace staging decisions on both ends +prun --preload-binary ... # stage the executable itself +prun --preload-files a.dat,b.tar.gz ... # stage data files (archives auto-extracted) +``` + +Framework verbosity ≥1 already prints the list of files chosen for +positioning and every chunk sent/received; ≥10 traces per-proc symlink +creation and archive path enumeration. A stuck job at `VM_READY` almost +always means a preposition callback that never fired or an ack that never +arrived from a daemon. + +--- + +## Where to go next + +- [`raw/AGENTS.md`](raw/AGENTS.md) — the only component: how files are + chunked, `xcast`-broadcast to every daemon, written into the session + directory, unarchived, and symlinked into each proc's directory. Read + this next. diff --git a/src/mca/filem/CLAUDE.md b/src/mca/filem/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/filem/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/filem/raw/AGENTS.md b/src/mca/filem/raw/AGENTS.md new file mode 100644 index 0000000000..e64ac61208 --- /dev/null +++ b/src/mca/filem/raw/AGENTS.md @@ -0,0 +1,262 @@ +# AGENTS.md — `filem/raw` (the xcast staging component) + +Component guide for `src/mca/filem/raw/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the +preposition/link entry points, and where this sits in the launch flow +(`VM_READY` → preposition → `MAP`). + +--- + +## Role and priority + +`raw` is the **only** `filem` component in the tree and the one that +actually stages files. Its `query` returns priority **0** +(`filem_raw_component.c`), so it wins by default whenever it is built; +if it is absent, the framework falls back to the no-op "none" module. + +"raw" describes its transport: it does **not** shell out to `scp`/`rsync` +or negotiate remote paths. It reads each source file on the HNP, chops it +into fixed-size chunks, and **`xcast`-broadcasts** the raw bytes to every +daemon over the RML. Each daemon reassembles the file into its node's +session directory, auto-extracts archives, and later symlinks the result +into each local process's session directory. + +`raw` implements exactly two of the framework's vtable slots — +`preposition_files` and `link_local_files` — plus `filem_init`, +`filem_finalize`, and `fault_handler`. Every put/get/rm/wait slot is +deliberately wired to the base `prte_filem_base_none_*` no-ops +(`filem_raw_module.c`, module struct at the top). + +--- + +## When/why selected + +Selection is automatic (priority 0, single-winner). The component only +*does* anything when a job's app contexts carry preload attributes: + +- `PRTE_APP_PRELOAD_BIN` — stage the executable itself (set by + `--preload-binary`). +- `PRTE_APP_PRELOAD_FILES` — a comma-separated list of files to stage + (set by `--preload-files`). + +With neither attribute set, `raw_preposition_files` finds nothing to do, +immediately fires the completion callback with `PRTE_SUCCESS`, and the +job proceeds to mapping. + +--- + +## Files + +| File | Contents | +|------|----------| +| `filem_raw_component.c` | Registration, the single `flatten_directory_trees` MCA param, `query` (priority 0). | +| `filem_raw.h` | The four private classes, `PRTE_FILEM_RAW_CHUNK_MAX` (16384), the `flatten_trees` flag. | +| `filem_raw_module.c` | Everything: the module vtable, HNP send path, daemon receive path, symlinking, and class instances. | + +### MCA parameter + +`filem_raw_flatten_directory_trees` (bool, default false). When true, a +staged file's remote target is just its basename — all files land flat in +the working directory instead of recreating their directory tree. + +--- + +## Private data structures (`filem_raw.h`) + +| Class | Lives on | Role | +|-------|----------|------| +| `prte_filem_raw_outbound_t` | HNP | One preposition request. Holds the list of `xfers`, the aggregate `status`, and the caller's `cbfunc`/`cbdata`. When its `xfers` list drains, the callback fires. | +| `prte_filem_raw_xfer_t` | HNP | One file being sent. Carries the read `fd`, the libevent `ev` (the caddy field — **named `ev`** as required), `src` (local path, for dup detection), `file` (remote-relative path), `type`, `nchunk` (next chunk index), and `nrecvd` (how many daemons have acked). | +| `prte_filem_raw_incoming_t` | daemon | One file being received. Carries the write `fd`, `ev`, `file`/`top`/`fullpath`, `type`, the `outputs` list of pending write buffers, and `link_pts` (paths to symlink for each proc). | +| `prte_filem_raw_output_t` | daemon | One received chunk: `numbytes` + a `PRTE_FILEM_RAW_CHUNK_MAX` data buffer, queued on an incoming file's `outputs` list for the write handler. | + +`xfer` and `incoming` both embed `ev` and use `PRTE_PMIX_THREADSHIFT` / +`prte_event_active` to drive their I/O on the progress thread — they are +long-lived caddies, released only when the transfer finishes. + +--- + +## `raw_init` / `raw_finalize` + +`raw_init` constructs the `incoming_files` list and posts a **persistent +RML recv** on `PRTE_RML_TAG_FILEM_BASE` bound to `recv_files` (this fires +on every node — HNP and daemons — since the HNP is also a target of its +own broadcast). If it is the HNP, it also constructs `outbound_files` and +`positioned_files` and posts a second persistent recv on +`PRTE_RML_TAG_FILEM_BASE_RESP` bound to `recv_ack`. `raw_finalize` drains +and destructs those lists. + +Three file-scoped lists hold all state: +`outbound_files`/`positioned_files` (HNP) and `incoming_files` (every +node). + +--- + +## HNP send path + +### `raw_preposition_files(jdata, cbfunc, cbdata)` + +The framework entry point on the master. Steps: + +1. **Scan app contexts** for preload attributes and build a temporary + `fsets` list of `prte_filem_base_file_set_t`: + - `PRTE_APP_PRELOAD_BIN`: mark the file `PRTE_FILEM_TYPE_EXE`, and + **rewrite the app** to run `./` from the session dir + (`app->app`, `app->argv[0]`, and `PRTE_APP_SSNDIR_CWD` are all + updated) so the staged copy is what actually executes. + - `PRTE_APP_PRELOAD_FILES`: split on `,`; infer the `target_flag` from + the suffix (`.tar`→TAR, `.bz`→BZIP, `.gz`→GZIP, else FILE); compute + the `remote_target` (basename if flattening, else the path made + relative — absolute paths have their leading `/` stripped); then + strip any leading `./`/`../` components so nothing escapes above the + session dir. The app's `PRTE_APP_PRELOAD_FILES` list is rewritten to + the cleaned relative names so the daemon side can match them. +2. If nothing was collected, fire the callback and return `PRTE_SUCCESS`. +3. Create one `outbound` object, stash `cbfunc`/`cbdata`, append it to + `outbound_files`. +4. For each file set, **de-duplicate**: skip anything whose `src` already + appears in `positioned_files` (already sent) or in any in-flight + `outbound->xfers` (already queued). This is why the same file + referenced by multiple apps is broadcast only once. +5. `open()` the source `O_RDONLY`, set it `O_NONBLOCK`, build a + `prte_filem_raw_xfer_t`, and `PRTE_PMIX_THREADSHIFT` it to + `send_chunk`. +6. If every file turned out to be a duplicate (empty `xfers`), release + the outbound and fire the callback immediately. + +Note the return value only reports whether the *setup* succeeded; actual +completion is signalled later through the callback. + +### `send_chunk(fd, argc, xfer)` — the read/broadcast pump + +Runs on the progress thread, re-arming itself until EOF: + +1. `read()` up to `PRTE_FILEM_RAW_CHUNK_MAX` (16 KB) bytes. On `EAGAIN`/ + `EINTR`, re-add the event and retry. On a hard error, force + `numbytes = 0` to flush an EOF downstream. +2. If `prte_dvm_abort_ordered`, drop the xfer and stop. +3. Pack a buffer `{file(string), nchunk(int32), data(numbytes bytes)}`; + on the **first chunk** (`nchunk == 0`) also append the `type` so the + receiver knows how to handle it. +4. `prte_grpcomm.xcast(PRTE_RML_TAG_FILEM_BASE, &chunk)` — broadcast to + **all daemons at once**. Increment `nchunk`. +5. If `numbytes == 0` this was the EOF chunk: close the fd and stop. + Otherwise re-arm the read event (`prte_event_active(..., PRTE_EV_WRITE, + 1)`) to pump the next chunk. + +So a file of *N* chunks produces *N* payload broadcasts plus one final +zero-byte broadcast that tells receivers to close and finalize. + +### `recv_ack` + `xfer_complete` — completion accounting + +Each daemon sends an ack `{file, status}` per file. `recv_ack` finds the +matching `xfer` in `outbound_files`, records any non-success status, and +bumps `xfer->nrecvd`. When `nrecvd == prte_process_info.num_daemons` the +file is fully positioned: `xfer_complete` moves the xfer from +`outbound->xfers` to `positioned_files`. When an outbound's `xfers` list +is empty, its `cbfunc` fires (this is the state machine's `files_ready`) +and the outbound is released. + +--- + +## Daemon receive path + +### `recv_files` — reassemble chunks + +Fires on `PRTE_RML_TAG_FILEM_BASE` for each broadcast chunk: + +1. Unpack `{file, nchunk}`; if `nchunk < 0` treat as EOF (`nbytes = 0`), + else unpack the byte payload; on chunk 0 also unpack `type`. +2. Find or create the matching `prte_filem_raw_incoming_t` in + `incoming_files`. +3. **On chunk 0**: compute `top` (first path component), build `fullpath` + under `prte_process_info.top_session_dir`, create the parent + directory, and `open()` the target for writing — `O_RDWR|O_CREAT| + O_TRUNC`, mode `S_IRWXU` for an EXE (so it stays executable) else + `S_IRUSR|S_IWUSR`. Then threadshift the incoming to `write_handler`. +4. Copy the payload into a fresh `prte_filem_raw_output_t`, append it to + `incoming->outputs`, and (if not already pending) activate the write + event. + +Any failure is reported back to the HNP via `send_complete(file, err)`. + +### `write_handler` — drain to disk, then finalize + +Runs on the progress thread; consumes `incoming->outputs`: + +- For each output with `numbytes > 0`, `write()` it to the fd. Short/`EAGAIN` + writes push the remainder back onto the front of the list and re-arm. +- When it hits the **zero-byte** output (EOF), it closes the fd and + finalizes by `type`: + - `FILE`/`EXE`: register `top` as the single link point, then + `send_complete(file, PRTE_SUCCESS)`. + - `TAR`/`BZIP`/`GZIP`: `chdir` into the target dir, run + `tar xf`/`tar xjf`/`tar xzf` via `system()`, `chdir` back, then call + `link_archive` and ack. + +### `link_archive` — enumerate archive contents + +Runs `tar tf ` via `popen`, reads each path, skips directories +and `.deps` trees, and appends every real file path to `inbnd->link_pts`. +Because different apps may share a directory tree but need different +files, each individual file becomes its own link point. + +### `send_complete(file, status)` + +Packs `{file, status}` and `PRTE_RML_SEND`s it to the HNP on +`PRTE_RML_TAG_FILEM_BASE_RESP` — the ack that `recv_ack` counts. + +--- + +## `raw_link_local_files(jdata, app)` — the daemon-side link phase + +Called by `odls` at fork time, **synchronously**, once per app context: + +1. Gather the app's wanted files: the `PRTE_APP_PRELOAD_FILES` list plus, + if `PRTE_APP_PRELOAD_BIN`, the executable basename. +2. For every local child in this job/app that is not yet alive, compute + its per-proc session dir (`session_dir>/`). +3. For each wanted file, find the matching `incoming` entry and, for each + of its `link_pts`, call `create_link` to `symlink()` the file from the + job session dir into the proc's session dir (creating intermediate + dirs, tolerating an already-existing link). + +This is what makes a staged file appear at the relative path the app +expects, in each rank's own directory. + +--- + +## Fault handling + +`raw_fault_handler` is intentionally minimal (marked TODO for real +resilience): if a daemon fails while `incoming_files` or `outbound_files` +is non-empty — i.e. a transfer is in flight — it activates +`PRTE_JOB_STATE_COMM_FAILED`. It relies on the fact that `xcast` is +already resilient for the not-in-flight case. + +--- + +## Things to watch when editing + +- **`ev` must stay named `ev`** in `xfer` and `incoming` — libevent/the + threadshift macros require it. These objects are caddies that outlive + the function that created them; never stack-allocate them. +- **Always ack, always callback.** Every receive error path must + `send_complete` so the HNP's ack count can complete, and the HNP's + outbound callback must fire on every path — a dropped ack or missed + callback hangs the job at `VM_READY`. This is the classic `raw` bug. +- **De-duplication depends on `src`/`positioned_files`.** The + already-sent checks in `raw_preposition_files` compare against both + `positioned_files` and in-flight `outbound->xfers`; keep both, or the + same file gets broadcast repeatedly across successive jobs in a DVM. +- **Chunk-0 carries the metadata.** File `type` (and the fd-open + decision) rides only on the first chunk; the zero-byte final chunk + triggers finalize. Don't reorder or coalesce these. +- **Paths are forced relative** on both the send side (strip leading + `/`, `./`, `../`) and the write side (rooted at `top_session_dir`). + This is a security property — staging must never let a user overwrite + an absolute path on a remote node. Preserve it. +- **`raw` owns `PRTE_RML_TAG_FILEM_BASE`.** It posts its own recv in + `raw_init` rather than using the base `filem_base_receive.c` service; + don't start the base comm service alongside it or the two will collide + on the tag. diff --git a/src/mca/filem/raw/CLAUDE.md b/src/mca/filem/raw/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/filem/raw/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/grpcomm/AGENTS.md b/src/mca/grpcomm/AGENTS.md new file mode 100644 index 0000000000..336870d0c2 --- /dev/null +++ b/src/mca/grpcomm/AGENTS.md @@ -0,0 +1,255 @@ +# AGENTS.md — The `grpcomm` Framework (Group Communication) + +Orientation for AI agents and human contributors working in +`src/mca/grpcomm/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`grpcomm` (Group Communication) provides the **collective communication +services that span the DVM's daemons**: scalable broadcast (`xcast`), +allgather/barrier (`fence`), and PMIx group operations (`group`). Its own +header says it plainly: + +> The PRTE Group Comm framework provides communication services that span +> entire jobs or collections of processes. It is not intended to be used +> for point-to-point communications (the RML does that), nor should it be +> viewed as a high-performance communication channel for large-scale data +> transfers. + +grpcomm runs **on every daemon** (`prted` and the HNP). It is layered on +top of the RML (Runtime Messaging Layer) and the routing tree: it does not +open its own connections — it sends RML messages along the radix routing +tree that `src/rml/` maintains (`prte_rml_base.children`, +`PRTE_PROC_MY_PARENT`, `PRTE_PROC_MY_HNP`). The framework is the machinery +behind almost every DVM-wide action: + +- The `plm`/`state` code broadcasts launch messages, wireup (nidmap), and + DAEMON commands with `prte_grpcomm.xcast(PRTE_RML_TAG_DAEMON, …)` / + `PRTE_RML_TAG_WIREUP`. +- The PMIx server shim satisfies `PMIx_Fence` / + `PMIx_server_register_resources` for local clients via + `prte_grpcomm.fence(...)` (see `src/prted/pmix/pmix_server_fence.c`). +- The PMIx server shim satisfies `PMIx_Group_construct/destruct/cancel` + via `prte_grpcomm.group(...)` (see + `src/prted/pmix/pmix_server_group.c`). +- The RML fault handler re-broadcasts `PRTE_RML_TAG_DAEMON_DIED` / + `PRTE_RML_TAG_DAEMON_REVIVED` through `xcast` as part of DVM recovery. + +Everything reaches the framework through a single global module instance, +`prte_grpcomm` (declared in `grpcomm.h`, defined in +`base/grpcomm_base_frame.c`). Callers never talk to a component directly; +they call `prte_grpcomm.(...)`. + +--- + +## Directory layout + +``` +grpcomm/ + grpcomm.h # THE module/component vtable + prte_pmix_grp_caddy_t + base/ + base.h # framework-global struct (context_id) + select() proto + grpcomm_base_frame.c # open/close/register; prte_pmix_grp_caddy_t class + grpcomm_base_select.c # single-winner component selection + grpcomm_base_stubs.c # STALE / NOT COMPILED — see warning below + static-components.h # generated: the built-in component list + direct/ # the only component (pri 5): RML-tree collectives +``` + +Read `grpcomm.h` first — the entire framework contract is the one vtable +struct it defines. Then read the `direct` component, which is where all the +real work lives. + +> **Warning — `grpcomm_base_stubs.c` is dead code.** It is **not** listed +> in `base/Makefile.am` and is never compiled. It references an older +> collective-tracking API (`prte_grpcomm_signature_t`, +> `prte_grpcomm_base.actives`, `prte_grpcomm_base.sig_table`, +> `prte_grpcomm_base_get_tracker()`, a `grp_construct` vtable entry, and +> `prte_pmix_grp_caddy_t` fields `sig`/`grpcbfunc` that do not exist in the +> current `grpcomm.h`). Do not treat anything in it as the live design; the +> current tracking model lives entirely inside the `direct` component. If +> you are cleaning up, this file is a candidate for deletion. + +--- + +## The module contract + +Unlike most PRRTE frameworks there is no separate "component vtable" and +"module vtable": `grpcomm.h` defines one struct, +`prte_grpcomm_base_module_t`, tagged **`Ver 4.0`**, and the selected +component fills it in. Every function pointer **MUST** be provided. + +| Field | Signature | Meaning / return protocol | +|-------|-----------|---------------------------| +| `init` | `int (*)(void)` | Called once on the winning module right after selection. Set up trackers, register RML receives. Returns `PRTE_SUCCESS`. | +| `finalize` | `void (*)(void)` | Tear down trackers and cancel RML receives. Called by the framework close. | +| `fault_handler` | `void (*)(const prte_rml_recovery_status_t *status)` | Invoked by the RML/routed layer (`src/rml/routed_radix.c`) when the routing tree changes — a daemon died, revived, or the local node was re-parented/promoted. Repair or abort in-flight collectives. | +| `xcast` | `int (*)(prte_rml_tag_t tag, pmix_data_buffer_t *msg)` | Scalably broadcast `msg` to **every** daemon in the DVM, to be delivered at `tag`. Non-destructive to `msg` (caller still owns it). Returns `PRTE_SUCCESS` when the broadcast has been *accepted*, not when it completes. | +| `xcast_nb` | `int (*)(prte_rml_tag_t tag, pmix_data_buffer_t *msg, prte_grpcomm_xcast_complete_fn_t cbfunc, void *cbdata)` | Same as `xcast`, but when `cbfunc != NULL` it fires on the **master** once the whole DVM has confirmed receipt (all ACKs have rolled back up the tree). `cbfunc`/`cbdata` are ignored on non-master daemons. `xcast` is just `xcast_nb(tag, msg, NULL, NULL)`. | +| `fence` | `int (*)(const pmix_proc_t procs[], size_t nprocs, const pmix_info_t info[], size_t ninfo, char *data, size_t ndata, pmix_modex_cbfunc_t cbfunc, void *cbdata)` | Non-blocking allgather/barrier across the daemons hosting `procs`. Barrier == NULL data. `cbfunc` is invoked with the gathered buffer on completion. Returns `PRTE_SUCCESS` once queued. | +| `group` | `int (*)(pmix_group_operation_t op, char *grpid, const pmix_proc_t procs[], size_t nprocs, const pmix_info_t directives[], size_t ndirs, pmix_info_cbfunc_t cbfunc, void *cbdata)` | PMIx group construct/destruct/cancel. Basically a fence with enough differences (context-id assignment, membership assembly, bootstrap) to warrant its own path. | + +`prte_grpcomm_cbfunc_t` (`void (*)(int status, pmix_data_buffer_t *buf, +void *cbdata)`) is declared in `grpcomm.h` for collective completion, and +`prte_grpcomm_xcast_complete_fn_t` (`void (*)(void *cbdata)`) is the +xcast-completion callback. + +The version macro components must stamp is +**`PRTE_GRPCOMM_BASE_VERSION_4_0_0`** (`grpcomm.h`), chained to +`PRTE_MCA_BASE_VERSION_3_0_0("grpcomm", 4, 0, 0)`. + +--- + +## Component selection is "pick one" + +`prte_grpcomm_base_select()` (in `grpcomm_base_select.c`) is a **standard +single-winner** selection — the opposite of `rmaps`. It calls +`pmix_mca_base_select("grpcomm", …)`, copies the winning module into the +global `prte_grpcomm`, and calls its `init()`. If no component is selected +it returns `PRTE_ERR_NOT_FOUND`. + +Today there is exactly one component, `direct`, with query priority **5** +(it is "always available"). Any replacement would just need to return a +higher priority from its `query`. + +--- + +## What `base/` provides + +The base is deliberately thin — all the collective algorithms live in the +component. The base offers only: + +### Framework plumbing (`grpcomm_base_frame.c`) + +- `PMIX_MCA_BASE_FRAMEWORK_DECLARE(prte, grpcomm, "GRPCOMM", …)` wires up + open/close/register. `prte_grpcomm_base_open()` opens all components; + `prte_grpcomm_base_close()` calls `prte_grpcomm.finalize()` (if set), + then closes the components. +- The global `prte_grpcomm` module (zero-initialized until selection) and + the global `prte_grpcomm_base` struct. + +### The framework-global struct (`base.h`) + +```c +typedef struct { + uint32_t context_id; /* initialized to UINT32_MAX */ +} prte_grpcomm_base_t; +``` + +`context_id` is the **group context-id pool**. When a group construct asks +for `PMIX_GROUP_ASSIGN_CONTEXT_ID`, the HNP hands out +`prte_grpcomm_base.context_id` and **decrements** it (see +`grpcomm_direct_group.c`). It counts *down* from `UINT32_MAX` so that these +DVM-assigned context ids do not collide with ids assigned from the bottom +of the range elsewhere. + +### The group caddy class (`grpcomm_base_frame.c` + `grpcomm.h`) + +`prte_pmix_grp_caddy_t` is the thread-shift caddy that carries a +`PMIx_Group*` request from the PMIx-server callback thread onto the +progress thread. It embeds the mandatory caddy fields (`ev`, `lock`, +`cbfunc`/`cbdata`) plus the group request parameters (`op`, `grpid`, +`procs`/`nprocs`, `directives`/`ndirs`, `info`/`ninfo`). Its constructor/ +destructor (`grpcon`/`grpdes`) manage the lock and free `grpid`/`info`. +The `direct` component allocates and posts these caddies. + +### Selection (`grpcomm_base_select.c`) + +`prte_grpcomm_base_select()` as described above. + +That's the whole base API. There is **no** base-level collective tracking, +signature packing, or xcast plumbing in the live build — the dead +`grpcomm_base_stubs.c` notwithstanding. If you are looking for the +signature/tracker model, it is component-private (see the `direct` guide). + +--- + +## The collective model (implemented in the component) + +Because all algorithms live in `direct`, the concepts below are documented +in depth in [`direct/AGENTS.md`](direct/AGENTS.md); here is the shape so +the framework makes sense: + +- **Routing tree.** Collectives ride the radix routing tree owned by + `src/rml/` (`prte_rml_base.children` / `n_children`, + `PRTE_PROC_MY_PARENT`, `PRTE_PROC_MY_HNP`). The HNP is the root of every + collective. +- **Signatures.** A collective is identified not by a global counter but + by a *signature* — for `fence`, the sorted set of participating procs; + for `group`, the `groupID` + operation; for `xcast`, an HNP-assigned, + globally-unique `op_id`. A signature lets independently-arriving pieces + of the same collective find each other. +- **Trackers.** Each daemon keeps a per-collective tracker (a + `prte_grpcomm_fence_t` / `prte_grpcomm_group_t` / xcast `op_t`) on a + component list, counting how many contributions it `nexpected` vs. + `nreported`. When a tracker completes locally it rolls up to the parent; + when the HNP's tracker completes, it broadcasts the release via `xcast`. +- **Two-phase collective.** `fence`/`group` are an **up-tree allgather** + (children → parent → … → HNP) followed by a **down-tree release** + (`xcast` from the HNP to everyone). `xcast` itself is the down-tree half, + made reliable with an ACK rollup. + +--- + +## Conventions, threading, and gotchas + +- **Everything runs on the single progress thread.** Every entry point + (`fence`, `group`, `xcast_nb`) immediately thread-shifts: it allocates a + heap caddy/op, `prte_event_set(prte_event_base, &cd->ev, …)`, + `PMIX_POST_OBJECT`, `prte_event_active`. The framework-global data + (trackers, `context_id`, xcast op lists) is only touched inside those + handlers. Follow that pattern — never touch tracker lists from a caller + thread. See the top-level guide's *Thread-shifting with caddies* + section; the caddy's event member must be named `ev`. +- **RML receives are persistent.** The component registers + `PRTE_RML_PERSISTENT` receives for each collective tag in `init()` and + cancels them in `finalize()`. The tags are `PRTE_RML_TAG_XCAST`, + `..._XCAST_ACK`, `..._FENCE`, `..._FENCE_RELEASE`, `..._GROUP`, + `..._GROUP_RELEASE`. +- **`xcast` returns on acceptance, not completion.** If you need to know + the whole DVM received a broadcast, use `xcast_nb` with a callback — that + is the hook the elastic DVM-shrink path uses to run its single + routing-tree repair. +- **Non-destructive to the caller's buffer.** `xcast`/`fence` copy the + payload; the caller keeps ownership of the `pmix_data_buffer_t` it + passed. +- **Capability-guarded FT code.** Group fault-tolerance (the + `PMIX_GROUP_CANCEL` operation and cancel routing) is compiled only when + `#if PRTE_PMIX_HAVE_GROUP_FT`. That macro is defined by + `config/prte_setup_pmix.m4` via `PRTE_CHECK_PMIX_CAP([GROUP_FT], …)`, + which succeeds when the installed PMIx advertises `PMIX_CAP_GROUP_FT`. + Any new group-FT code must live behind that guard, and you must build + against a new-enough PMIx (and re-run `autogen.pl`) to exercise it. +- **Standard PRRTE rules apply:** `prte_config.h` first, constant-on-left + comparisons, braces on every block, `PRTE_ERROR_LOG`/`PMIX_ERROR_LOG` + for unexpected errors, no new compiler warnings. + +--- + +## Debugging + +```sh +prte --prtemca grpcomm_base_verbose 5 ... # trace collective progress +prte --prtemca plm_base_verbose 5 ... # daemon launch / xcast of launch msg +prte --prtemca state_base_verbose 5 ... # job-state transitions that drive fences +prte --prtemca routed_base_verbose 5 ... # routing-tree (children/parent) view +``` + +Framework verbosity ≥1 already narrates each `xcast`, `fence`, and `group` +call and its rollup counts (`nexpected` vs `nreported`); ≥5 adds per-child +relay/ack traffic. Because collectives ride the routing tree, a "collective +hang" is almost always a routing-tree problem — check `routed_base_verbose` +and the fault handlers first. + +--- + +## Where to go next + +- [`direct/AGENTS.md`](direct/AGENTS.md) — the one and only component; + read it for the actual xcast/fence/group algorithms, the op-id + sequencing and ACK rollup, and the group construct/cancel/FT handling. diff --git a/src/mca/grpcomm/CLAUDE.md b/src/mca/grpcomm/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/grpcomm/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/grpcomm/direct/AGENTS.md b/src/mca/grpcomm/direct/AGENTS.md new file mode 100644 index 0000000000..07cedbf17f --- /dev/null +++ b/src/mca/grpcomm/direct/AGENTS.md @@ -0,0 +1,315 @@ +# AGENTS.md — `grpcomm/direct` (the RML-tree collective engine) + +Component guide for `src/mca/grpcomm/direct/`. Read the +[framework guide](../AGENTS.md) first for the module vtable, the collective +model (signatures / trackers / two-phase collective), and the threading +rules referenced throughout. + +--- + +## Role and priority + +`direct` is the **only** grpcomm component and the sole implementation of +every collective in PRRTE. Its `query` (`grpcomm_direct_component.c`) +returns priority **5** and declares itself "always available", so it always +wins the single-winner selection. It implements the framework vtable by +running collectives **directly over the RML radix routing tree** — hence +the name — with the HNP as the root of every operation. + +Files: + +| File | Contents | +|------|----------| +| `grpcomm_direct.h` | Component struct `prte_grpcomm_direct_component_t`; the fence/group signature structs; the fence/group tracker structs; the fence caddy; all public entry-point prototypes; `print_signature()` debug helper. | +| `grpcomm_direct_component.c` | Registration, `direct_query` (priority 5), and the `PMIX_CLASS_INSTANCE` definitions for the signature/tracker/caddy classes. | +| `grpcomm_direct.c` | The module vtable (`prte_grpcomm_direct_module`), `init()` (construct trackers + register the six persistent RML receives), `finalize()`, and the combined `fault_handler`. | +| `grpcomm_direct_xcast.c` | The reliable, fault-tolerant broadcast: op-id sequencing, tree forwarding, ACK rollup, late-joiner/promotion handling, and the completion-callback FIFO. | +| `grpcomm_direct_fence.c` | Allgather/barrier: up-tree rollup to the HNP, then a `xcast` release back down. | +| `grpcomm_direct_group.c` | PMIx `PMIx_Group_construct/destruct/cancel`: membership assembly, context-id assignment, bootstrap, add-members, final-order, and the group-FT abort/cancel paths. | + +The module vtable (`grpcomm_direct.c`): + +```c +prte_grpcomm_base_module_t prte_grpcomm_direct_module = { + .init = init, .finalize = finalize, .fault_handler = fault_handler, + .xcast = prte_grpcomm_direct_xcast, .xcast_nb = prte_grpcomm_direct_xcast_nb, + .fence = prte_grpcomm_direct_fence, .group = prte_grpcomm_direct_group +}; +``` + +`init()` constructs the three tracker containers on the component +(`xcast_ops`, `fence_ops`, `group_ops`) and registers **six** persistent +RML receives: `XCAST`, `XCAST_ACK`, `FENCE`, `FENCE_RELEASE`, `GROUP`, +`GROUP_RELEASE`. `finalize()` destructs the trackers and cancels the +receives. `fault_handler()` simply fans the recovery notice out to the +xcast, fence, and group fault handlers in turn. + +--- + +## Key data structures (`grpcomm_direct.h`) + +- **`prte_grpcomm_direct_component_t`** — the component. Holds + `xcast_ops` (a `prte_grpcomm_xcast_t`), `fence_ops` (list of + `prte_grpcomm_fence_t`), and `group_ops` (list of `prte_grpcomm_group_t`). +- **`prte_grpcomm_xcast_t`** — global xcast state: the `ops` list of + in-flight broadcasts, a `pending_completions` FIFO of completion + callbacks awaiting relay-back, and three sequence counters + (`op_id_inited`, `op_id_completed`, `op_id_completed_at_promotion`). +- **`prte_grpcomm_direct_fence_signature_t`** — a fence's identity: the + array of participating `pmix_proc_t` (`signature`) and its size (`sz`). + Two fences are "the same" iff their proc arrays match byte-for-byte. +- **`prte_grpcomm_direct_group_signature_t`** — a group's identity and + payload: `op`, `groupID`, `assignID`/`ctxid`/`ctxid_assigned`, initial + `members`, `bootstrap` count, `follower` flag, `addmembers`, and + `final_order`. +- **`prte_grpcomm_fence_t` / `prte_grpcomm_group_t`** — the per-collective + trackers: the signature, resolved participating daemons (`dmns`/`ndmns`), + the `nexpected`/`nreported` rollup counters (group adds bootstrap + leader/follower counters), a `bucket`/info-lists for gathered data, and + the user `cbfunc`/`cbdata`. +- **`prte_pmix_fence_caddy_t`** — the thread-shift caddy for a fence + request (mirrors `prte_pmix_grp_caddy_t` for groups, which lives in the + framework header). + +`create_dmns()` (duplicated in `fence.c` and `group.c`) turns a signature's +proc set into the set of daemon vpids that must participate: if the target +nspace is the daemon job itself, *all* daemons participate; otherwise it +walks each proc's `proc->node->daemon` (or, for `PMIX_RANK_WILDCARD`, every +daemon in that job's map) and de-dups. `prte_rml_get_num_contributors()` +then tells the tracker how many child contributions to expect. + +--- + +## `xcast` — reliable fault-tolerant broadcast (`grpcomm_direct_xcast.c`) + +This is the most intricate file. The public `prte_grpcomm_direct_xcast()` +is just `xcast_nb(tag, msg, NULL, NULL)`. + +### Message flow + +1. **Originate (`xcast_nb`).** Any daemon can originate. It builds an + `op_t`, stashes the (possibly compressed) user message and tag, records + the optional completion callback, and thread-shifts to `begin_xcast`. +2. **`begin_xcast`.** Packs the op and reliably sends it **to the HNP** + (`PRTE_RML_RELIABLE_SEND(... PRTE_PROC_MY_HNP ... PRTE_RML_TAG_XCAST)`). + The initiating op is then discarded — it is not the tracked op. If the + originator is the master, it also enqueues one entry on the + `pending_completions` FIFO (see *Completion callbacks* below). +3. **HNP assigns the op-id.** The HNP's `xcast_recv` sees `sig.op_id == 0` + and stamps `sig.op_id = ++XCAST.op_id_inited` — a globally-unique, + monotonically-increasing collective id. (A non-zero op-id arriving at + the HNP is a bug → `PRTE_ERR_DUPLICATE_MSG`.) +4. **Forward down the tree (`forward_op`/`forward_op_to`).** Each daemon, + on receiving a new op, forwards the packed op to each of its routing-tree + children (`prte_rml_base.children`) and processes it locally. +5. **Process locally (`process_msg`).** Decompresses if needed and + delivers the payload to *itself* at the user tag via + `PRTE_RML_POST_MESSAGE` (not a real send — it is injected straight into + the local RML message processor). `PRTE_RML_TAG_WIREUP` is special-cased + to `process_wireup()` (decode nidmap). +6. **ACK rollup.** A leaf (`0 == n_children`) immediately `finish_op`s, + sending an ACK to its parent. An interior daemon `finish_op`s only once + `nreported == nexpected` (all children have ACKed). `finish_op` sends + its own ACK upward, advances `op_id_completed`, processes the message if + not already done, fires the completion callback (master only), and + releases the op. + +`send_ack`/`request_ack`/`xcast_ack` carry the ACK protocol on +`PRTE_RML_TAG_XCAST_ACK`, distinguished by an `is_request` bool: a plain +ACK ("my subtree is done"), or a *request* for an ACK (used after a fault +to re-poll a child without resending the payload). + +### Ordering and fault tolerance + +The comments in this file are the real spec — read them. The load-bearing +ideas: + +- **`process_first` set.** Most xcasts forward before processing (to + preserve message ordering), but `PRTE_RML_TAG_WIREUP` and + `PRTE_RML_TAG_DAEMON_DIED` are processed *first* because they change the + child set: a death *grows* our subtree (orphans promote to us), so we + must repair before forwarding. `PRTE_RML_TAG_DAEMON_REVIVED` deliberately + stays on the forward-first path (a return *shrinks* our subtree) — the + comment explicitly says do not move it. +- **Late joiners.** A daemon that has never seen an xcast + (`op_id_inited == 0`) but is handed op N>1 is a grown/rebooted/bootstrap + daemon; it adopts ops `1..N-1` as already complete + (`op_id_completed = op_id_completed_at_promotion = N-1`) so `finish_op` + does not raise `PRTE_ERR_OUT_OF_ORDER_MSG` and force-exit. +- **Promotion / re-parenting.** `xcast_fault_handler` (local-scope only) + reacts to `status->promoted` / `parent_changed` / `children_changed`: + it invalidates upward ack-ids (new parent will re-issue them), resets + `nexpected` to the new child count, starts new ack rounds + (`ack_id_down++`), and holds replays (`replay_pending_parent`) after a + promotion until the parent replays the ops in order. +- **`op_id_completed_at_promotion`** guards the "assume-complete" logic so + a promoted daemon does not wrongly assume its *newly-acquired* subtree + finished ops the daemon itself completed before promotion. + +### Completion callbacks (the `pending_completions` FIFO) + +The op the master ends up *tracking* is a fresh one built on receipt, not +the initiating op — so a callback cannot ride the initiating op. Instead +`begin_xcast` enqueues one `pending_completion_t` per **master-originated** +broadcast (NULL callback included, to keep alignment), in send order. +When the master receives that broadcast back and builds its tracked op, it +pops the FIFO head and attaches the callback. `finish_op` fires it — but +only on the master, where a completed op means the *entire DVM* has +received the broadcast. This is the hook the elastic DVM-shrink path uses. + +--- + +## `fence` — allgather / barrier (`grpcomm_direct_fence.c`) + +`prte_grpcomm_direct_fence()` is the vtable `fence`. It rejects a NULL +`procs` array (`PRTE_ERR_NOT_SUPPORTED`), builds a `prte_pmix_fence_caddy_t`, +and thread-shifts to the static `fence()` handler. + +Message flow: + +1. **`fence` handler.** Computes the fence signature from `cd->procs`, + gets-or-creates the tracker (`get_tracker(..., true)`), packs signature + + info + the local `data` payload into a relay buffer, and **sends it to + itself** on `PRTE_RML_TAG_FENCE`. Sending to self funnels the local + contribution through the same receive path everything else uses. +2. **`fence_recv`** (`PRTE_RML_TAG_FENCE`). Unpacks the signature, finds + the tracker, merges info (`PMIX_TIMEOUT` takes the max; a non-success + `PMIX_LOCAL_COLLECTIVE_STATUS` is sticky), copies the payload into + `coll->bucket`, and bumps `nreported`. When `nreported == nexpected`: + - **HNP:** the allgather is complete → pack signature + status + bucket + and broadcast the result down via + `prte_grpcomm.xcast(PRTE_RML_TAG_FENCE_RELEASE, reply)`. + - **non-HNP:** the local subtree rollup is complete → forward the + bucket up to `PRTE_PROC_MY_PARENT` on `PRTE_RML_TAG_FENCE`. +3. **`fence_release`** (`PRTE_RML_TAG_FENCE_RELEASE`, delivered by the + xcast). Unpacks the signature + status, finds the tracker (missing + tracker == "I had no local participants", not an error), and fires + `coll->cbfunc(status, bytes, size, cbdata, relcb, bytes)` to hand the + gathered data back to the PMIx server. Removes and releases the tracker. + +`nexpected` counts routing-tree child contributors +(`prte_rml_get_num_contributors`) plus one if this daemon is itself a +participant. The tracker lives on `component.fence_ops`, keyed by exact +proc-signature match. + +The fence `fault_handler` is currently **not resilient**: a TODO. If any +fence op is in flight when a daemon fails it activates +`PRTE_JOB_STATE_COMM_FAILED` (kills the job) rather than repairing. + +--- + +## `group` — PMIx group operations (`grpcomm_direct_group.c`) + +`prte_grpcomm_direct_group()` is the vtable `group`, driving +`PMIx_Group_construct` / `destruct` / `cancel`. It builds a +`prte_pmix_grp_caddy_t` (framework header) and thread-shifts to the static +`group()` handler. The rollup/release skeleton mirrors `fence`, but with a +much richer signature and payload. + +### The `group` handler + +- **Cancel short-circuit** (`#if PRTE_PMIX_HAVE_GROUP_FT`): a + `PMIX_GROUP_CANCEL` op is *not* a rollup collective — it routes straight + to the HNP via `request_group_cancel()` and returns. +- Otherwise it builds the group signature from `grpid` + `procs` (a NULL + `procs` marks a bootstrap **follower**), scans the directives + (`PMIX_GROUP_ASSIGN_CONTEXT_ID`, `PMIX_GROUP_BOOTSTRAP`, `PMIX_TIMEOUT`, + `PMIX_GROUP_ADD_MEMBERS`, `PMIX_GROUP_INFO`, `PMIX_PROC_DATA` endpoints, + `PMIX_GROUP_FINAL_MEMBERSHIP_ORDER`, `PMIX_LOCAL_COLLECTIVE_STATUS`), + gets-or-creates the tracker, and relays. +- **Bootstrap** ops send **directly to the HNP** (there is no rollup tree — + each daemon reports straight to the controller); non-bootstrap ops send + to self on `PRTE_RML_TAG_GROUP`, entering the same up-tree rollup as + fence. + +### `grp_recv` (rollup) and `grp_release` (down-tree) + +- **`grp_recv`** (`PRTE_RML_TAG_GROUP`). Handles the FT cancel first + (HNP-only: find the in-flight construct by groupID and abort it). Then + it merges the incoming contribution (status, timeout, grpinfo, endpoints) + into the tracker and bumps the appropriate counter: bootstrap **leaders** + (`nleaders_reported`), bootstrap **followers** (`nfollowers_reported`), + or ordinary participants (`nreported`). Completion is + `nleaders_reported == nleaders && nfollowers_reported == nfollowers` for + bootstrap, else `nreported == nexpected`. + - **HNP at completion:** for a construct it assigns the context id (if + requested, from the decrementing `prte_grpcomm_base.context_id`), + assembles the **final membership** (union of members + add-members, + wildcard-preserving), applies `final_order` if given (else `qsort` for a + stable order), packs signature + status + membership + grpinfo + + endpoints, and broadcasts the result with + `prte_grpcomm.xcast(PRTE_RML_TAG_GROUP_RELEASE, reply)`. + - **non-HNP at completion:** roll the aggregated results up to + `PRTE_PROC_MY_PARENT`. +- **`grp_release`** (`PRTE_RML_TAG_GROUP_RELEASE`, via the xcast). For a + **destruct** it removes the group from the server's pset list and + completes the local participants. For a **construct** it unpacks the + final membership / context-id / grpinfo / endpoints, calls + `PMIx_server_register_resources` (blocking on a caddy lock), records the + new group in `prte_pmix_server_globals.groups`, and returns the assembled + info to local clients via `coll->cbfunc`. Finally it deletes the tracker + (`find_delete_tracker`, keyed by groupID). + +### Group fault tolerance (`#if PRTE_PMIX_HAVE_GROUP_FT`) + +Guarded by `PRTE_PMIX_HAVE_GROUP_FT` (from +`PRTE_CHECK_PMIX_CAP([GROUP_FT])`, i.e. the installed PMIx advertises +`PMIX_CAP_GROUP_FT`). Two related paths, both converging on +`abort_group_op()`: + +- **Explicit cancel.** A client's `PMIx_Group_cancel` reaches + `prte_grpcomm_direct_group()` with `op == PMIX_GROUP_CANCEL`. + `request_group_cancel()` routes a signature-only message + (`op == PMIX_GROUP_CANCEL` + groupID) to the HNP and acks the requester. + The HNP's `grp_recv` (before `get_tracker`, so it never creates a + spurious cancel tracker) calls `find_construct_op(groupID)` and, if the + construct is still in flight, `abort_group_op(coll, PMIX_GROUP_CONSTRUCT_ABORT)`. +- **Participant failure.** `prte_grpcomm_direct_group_fault_handler` runs + **only on the HNP, only in the GLOBAL-scope pass** (the global + notification carries the consistent `failed_ranks` on every rank). It + aborts every in-flight group op that a failed daemon participated in + (constructs get `PMIX_GROUP_CONSTRUCT_ABORT`, destructs get + `PMIX_SUCCESS` since they are tearing down anyway). An op whose daemon set + was never resolved (e.g. a bootstrap) is aborted defensively. + +`abort_group_op()` broadcasts a `PRTE_RML_TAG_GROUP_RELEASE` carrying only +the signature + a completion status; the normal `grp_release` non-success +path then completes each daemon's local participants with that status and +deletes the tracker — so a cancel/abort tears down the collective cleanly +**without tearing down the DVM**, which is the whole point of this work. + +--- + +## Things to watch when editing + +- **`grp_construct`/`begin_xcast`/`prte_grpcomm_API_*` in the base are + dead.** `grpcomm_base_stubs.c` is not compiled (see the framework + guide). Do not model new work on it. The live tracking model is entirely + in this component. +- **Never touch trackers off the progress thread.** Every entry point + thread-shifts through `prte_event_set`/`prte_event_active` for exactly + this reason. The tracker lists, `context_id`, and xcast counters are + progress-thread-only. +- **xcast ordering is a correctness invariant, not a nicety.** The + `process_first` set, the late-joiner catch-up, and the promotion replay + hold are what keep the reliable broadcast correct across DVM + grow/shrink/unheal. Read the in-file comments before changing any of it; + a mistake here manifests as `PRTE_ERR_OUT_OF_ORDER_MSG` force-exits or + silent message loss during recovery. +- **Signatures must round-trip exactly.** Fence trackers match on a + byte-for-byte `memcmp` of the proc array; group trackers match on + groupID + op. If you add a field to a signature, update *both* its + pack/unpack and its constructor/destructor, or trackers will fail to + coalesce (hang) or leak. +- **Group-FT code must stay behind `#if PRTE_PMIX_HAVE_GROUP_FT`.** The + cancel op, `find_construct_op`, and `request_group_cancel` are all + guarded; keep it that way so PRRTE still builds against a pre-FT PMIx. + Note the fault-handler abort loop itself is *unguarded* (it uses only + status fields and `abort_group_op`), but the `PMIX_GROUP_CANCEL` handling + is guarded — preserve that split. +- **The fence fault handler is a known gap.** It kills the job on any + in-flight fence when a daemon fails (TODO to make it resilient). If you + are adding fence resilience, that is the place — do not weaken it into a + silent no-op. +- Standard PRRTE rules: `prte_config.h` first, constant-on-left, braces + everywhere, `PMIX_ERROR_LOG`/`PRTE_ERROR_LOG`, no new warnings. diff --git a/src/mca/grpcomm/direct/CLAUDE.md b/src/mca/grpcomm/direct/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/grpcomm/direct/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/iof/AGENTS.md b/src/mca/iof/AGENTS.md new file mode 100644 index 0000000000..a3013b8eae --- /dev/null +++ b/src/mca/iof/AGENTS.md @@ -0,0 +1,414 @@ +# AGENTS.md — The `iof` Framework (I/O Forwarding) + +Orientation for AI agents and human contributors working in +`src/mca/iof/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`iof` (I/O Forwarding) connects the `stdin`/`stdout`/`stderr` of every +launched application process back to the user. An application proc runs +on some node, forked by that node's `prted`; its terminal is not the +user's terminal. `iof` is the machinery that: + +- **captures** each local proc's `stdout`/`stderr` (the read side), and +- **delivers** `stdin` down to the proc(s) that asked for it (the write / + sink side), + +routing every byte through the HNP so it lands on the user's terminal or +output files. + +The framework is **role-split** into exactly two components that never +run together in one process: + +| Component | Runs in | Priority | Role | +|-----------|---------|----------|------| +| `hnp` | the HNP / DVM master (`prte`, `prun`, `mpirun`) | 100 | The hub. Collects all output (local + remote), hands it to the PMIx server for terminal/file output, and injects stdin. | +| `prted` | every per-node daemon (`prted`) | 80 | The relay. Captures its local procs' output and forwards it to the HNP; receives stdin from the HNP and writes it to local procs. | + +Because the two roles are mutually exclusive (a process is either the +master or a daemon), the "priorities" never actually compete — the query +functions gate on process type, so only one ever returns a module. See +[Component selection](#component-selection). + +### The end-to-end picture + +``` + application proc (on node N) + │ stdout/stderr fd stdin fd ▲ + ▼ (pipe/pty read end) (pipe write end) │ + ┌──────────────────────────────────────────────────┐ + │ prted iof (node N) │ + │ read_handler: read(fd) ──► PMIx_server_IOF_deliver (local echo) │ + │ └──► pack+RML ─► HNP (PRTE_RML_TAG_IOF_HNP) │ + │ recv (PRTE_RML_TAG_IOF_PROXY): stdin ─► write_output ─► proc pipe │ + └──────────────────────────────────────────────────┘ + │ RML ▲ RML + ▼ │ + ┌──────────────────────────────────────────────────┐ + │ hnp iof (DVM master) │ + │ recv (PRTE_RML_TAG_IOF_HNP): ─► PMIx_server_IOF_deliver │ + │ read_local_handler (my own children): ─► PMIx_server_IOF_deliver │ + │ push_stdin: ─► RML to hosting daemon / local proc pipe │ + └──────────────────────────────────────────────────┘ + │ + ▼ + PMIx server library ──► user's terminal / --output files / pulling tools +``` + +**Key architectural fact:** the iof components do **not** write +`stdout`/`stderr` to the user's terminal themselves, and they do **not** +implement `--output` tagging, per-rank files, or the copy-to-a-tool +("pull") logic. They hand captured bytes to +`PMIx_server_IOF_deliver()` (via the `prte_iof_deliver_t` carrier), and +the **PMIx server library** does the terminal/file emission and honors +the user's output directives. The iof framework's own write machinery +(sinks, `prte_iof_base_write_output`, write handlers) is used almost +entirely for the **stdin** direction — writing bytes down a local proc's +stdin pipe. Keep that split in mind; it is the single biggest thing the +historical [`README.txt`](README.txt) and the stale docstrings in +[`iof.h`](iof.h) get wrong relative to today's code. + +There is **no proxy-to-proxy traffic**: a daemon never sends output to +another daemon. Everything funnels HNP-ward and the HNP fans back out. + +--- + +## Directory layout + +``` +iof/ + iof.h # the module vtable (init/push/pull/close/complete/finalize/push_stdin) + iof_types.h # prte_iof_tag_t + the PRTE_IOF_* stream/flow/tool tag bitmask + README.txt # HISTORICAL notes (2007-era design) — largely stale, read with care + base/ + base.h # framework struct, the sink/read/write/proc/deliver structs, all the macros + iof_base_frame.c # open/close/register; MCA param; prte_iof_base_output; all class instances + iof_base_select.c # pick-ONE selection (highest priority query wins) + iof_base_output.c # prte_iof_base_write_output + prte_iof_base_write_handler (the sink engine) + iof_base_setup.[ch] # pre-fork pipe/pty creation, child fd dup2, parent push/pull wiring + static-components.h # generated: the hnp + prted component table + hnp/ # HNP hub component (pri 100) — see hnp/AGENTS.md + prted/ # daemon relay component (pri 80) — see prted/AGENTS.md +``` + +Read `iof_types.h` and the struct block at the top of `base.h` first — +the tag bitmask and the five core structs are the whole vocabulary of the +framework. Then read one component end-to-end (`prted` is the simpler +read→forward story). + +--- + +## The module contract + +Every component fills in a `prte_iof_base_module_t` (defined in +[`iof.h`](iof.h) as `prte_iof_base_module_2_0_0_t`). It is a small, +fixed vtable — there is no return-code "try the next component" protocol +like `rmaps` has, because only one iof module is ever selected: + +```c +struct prte_iof_base_module_2_0_0_t { + prte_iof_base_init_fn_t init; /* void -> int */ + prte_iof_base_push_fn_t push; /* (peer, tag, fd) -> int */ + prte_iof_base_pull_fn_t pull; /* (peer, tag, fd) -> int */ + prte_iof_base_close_fn_t close; /* (peer, tag) -> int */ + prte_iof_base_complete_fn_t complete; /* (jdata) -> void */ + prte_iof_base_finalize_fn_t finalize; /* void -> int */ + prte_iof_base_push_stdin_fn_t push_stdin; /* (dst, data, sz) -> int (hnp only) */ +}; +``` + +| Function | Meaning | Typical caller | +|----------|---------|----------------| +| `init` | Post the framework's persistent RML receive and construct the per-process `procs` list. Called by `prte_iof_base_select` right after the module is chosen. | selection | +| `push` | **Capture output.** Tie a local read fd (the read end of a proc's `stdout` or `stderr` pipe, distinguished by `src_tag`) to a read event that forwards whatever appears. | `prte_iof_base_setup_parent` (from `odls`) | +| `pull` | **Register a stdin sink.** Tie a local write fd (the write end of a proc's `stdin` pipe) to a sink so data addressed to that proc's stdin gets written down it. Only `PRTE_IOF_STDIN` is supported. | `prte_iof_base_setup_parent` | +| `close` | Tear down the read events and/or sink for the named peer for the streams in `source_tag`; drop the proc from `procs` once all three are gone. | teardown paths | +| `complete` | Job finished: purge any lingering `prte_iof_proc_t`s belonging to `jdata->nspace`. | `state` machine on `JOB`/`PROC` completion | +| `finalize` | Cancel the RML receive and destruct the `procs` list. | framework close | +| `push_stdin` | **Inject stdin.** HNP-only: route a chunk of stdin to a target proc (or wildcard) — to the hosting daemon over RML, or to a local proc's sink. `NULL` in the `prted` module. | PMIx server glue (`pmix_server_gen.c`) | + +Note the naming is a little counter-intuitive and the [`iof.h`](iof.h) +docstrings are stale: **`push` handles the OUTPUT (read) side**, **`pull` +handles the STDIN (write/sink) side**. Trust the implementations, not the +header prose. + +`init`/`finalize`/`push_stdin`/`complete` may legitimately be `NULL` in a +module; the base and callers all guard with `if (NULL != prte_iof.xxx)`. + +--- + +## Component selection + +Unlike `rmaps`, `iof` is a classic **pick-one** framework. +`prte_iof_base_select()` (in +[`iof_base_select.c`](base/iof_base_select.c)) calls +`pmix_mca_base_select()`, which queries every component and keeps the +single highest-priority module. The winner is copied into the global +`prte_iof` module struct and its `init()` is called immediately. + +Selection is really driven by **process role**, not by the numeric +priority: + +- `hnp`'s query (in + [`iof_hnp_component.c`](hnp/iof_hnp_component.c)) returns priority `100` + **only if `PRTE_PROC_IS_MASTER`**, else `-1`/`PRTE_ERROR`. +- `prted`'s query (in + [`iof_prted_component.c`](prted/iof_prted_component.c)) returns priority + `80` **only if `PRTE_PROC_IS_DAEMON`**, else `-1`/`PRTE_ERROR`. + +So in any given process exactly one of them offers a module, and that one +wins. A tool that is neither master nor daemon gets no iof module and +interacts with the HNP's iof purely through the PMIx server. + +--- + +## The base machinery in detail + +The base is where all the shared data structures, event macros, and the +sink write engine live. A component is mostly glue that allocates these +structs and arms these events. + +### The five core structs (`base.h`) + +| Struct | What it models | +|--------|----------------| +| `prte_iof_proc_t` | Per-proc **endpoint bundle**: the proc `name`, its `stdinev` sink, and its `revstdout` / `revstderr` read events. Each component keeps a `pmix_list_t procs` of these. | +| `prte_iof_read_event_t` | One **read side**: an fd, its libevent `ev`, the `tag` (stdout/stderr), `active`/`activated` flags, `always_readable`, a back-pointer to the owning `proc`, and an optional `sink`. Destructor closes the fd. | +| `prte_iof_sink_t` | One **output endpoint**: proc `name`, owning `daemon`, `tag`, a `prte_iof_write_event_t *wev`, and `xoff`/`exclusive`/`closed` flags. | +| `prte_iof_write_event_t` | One **write side**: an fd, its libevent `ev`, `pending` (is the write event armed?), `always_writable`, and a `pmix_list_t outputs` of queued chunks. | +| `prte_iof_write_output_t` | One **queued write chunk**: a fixed `data[PRTE_IOF_BASE_TAGGED_OUT_MAX]` (8192) buffer and `numbytes`. A `numbytes == 0` chunk is the sentinel meaning "flush then close this fd." | +| `prte_iof_deliver_t` | Carrier for handing bytes to the PMIx server: a `source` proc and a `pmix_byte_object_t bo`. Freed by the `PMIx_server_IOF_deliver` completion callback. | + +All are PMIx classes (`PMIX_CLASS_INSTANCE` in +[`iof_base_frame.c`](base/iof_base_frame.c)); construct/destruct them with +`PMIX_NEW`/`PMIX_RELEASE`. The `prte_iof_proc_t` destructor releases its +sink and both read events; the read-event destructor `close()`s its fd; +the write-event destructor closes fds `> 2` (never stdout/stderr of the +daemon itself). + +### Read-event macros (`base.h`) + +- `PRTE_IOF_READ_EVENT(&slot, proc, fd, tag, cbfunc, activate)` — + allocate a `prte_iof_read_event_t`, retain the proc, set up its + libevent handler on `fd` (a timer event if the fd is "always readable," + i.e. a regular file / non-tty char dev / block dev; a real + `PRTE_EV_READ` fd event otherwise), and optionally activate it. +- `PRTE_IOF_READ_ACTIVATE(rev)` / `PRTE_IOF_READ_ADDEV(rev)` — mark the + read event active and add it to the event base. + +The `always_readable` branch exists because regular files never signal +readiness through the event loop; they are driven by a zero-length timer +instead. `prte_iof_base_fd_always_ready(fd)` is the predicate. + +### The sink write engine (`iof_base_output.c`) + +This is the heart of the **stdin / output-to-fd** path: + +- **`prte_iof_base_write_output(name, stream, data, numbytes, channel)`** — + append a copy of `data` (into a fresh `prte_iof_write_output_t`) to the + write event's `outputs` list, and if the write event isn't already + armed, arm it with `PRTE_IOF_SINK_ACTIVATE`. Returns the current + backlog size (list length). A `numbytes == 0` call still enqueues a + sentinel so the fd is flushed and closed. Callers compare the return + against `PRTE_IOF_MAX_INPUT_BUFFERS` (50) to detect back-pressure. + +- **`prte_iof_base_write_handler(fd, event, cbdata)`** — the generic + libevent write callback. It drains `wev->outputs`, `write()`-ing each + chunk to `wev->fd`. It handles the three realities of non-blocking + writes: + - `EAGAIN`/`EINTR` → prepend the chunk back and leave the event armed to + retry; + - **partial write** → `memmove` the unwritten tail to the front, fix + `numbytes`, prepend, retry; + - `numbytes == 0` chunk → close the stream by releasing the sink. + + If the backlog ever exceeds `prte_iof_base_output_limit` it declares IOF + hopelessly behind and fires `PRTE_JOB_STATE_FORCED_EXIT`. To avoid + starving other fds, an "always writable" (regular-file) sink yields + after `PRTE_IOF_SINK_BLOCKSIZE` (1024) bytes and re-arms. + + This generic handler is wired up by the **PMIx server glue** + (`src/prted/pmix/pmix_server_gen.c`) for sinks it creates. The two iof + components each define their **own** near-identical + `stdin_write_handler` (with subtly different close/`xoff` semantics) and + pass it to `PRTE_IOF_SINK_DEFINE` — so when editing write semantics, + check all three copies. + +### Sink macros (`base.h`) + +- `PRTE_IOF_SINK_DEFINE(&slot, name, fd, tag, wrthndlr)` — allocate a + `prte_iof_sink_t`, load its name/tag, and (if `fd >= 0`) set up its + write event's libevent handler on `fd`, choosing timer vs. + `PRTE_EV_WRITE` by `always_writable`. +- `PRTE_IOF_SINK_ACTIVATE(wev)` — mark the write event `pending` and add + it to the event base (with a timer for always-writable fds). + +### `prte_iof_base_output()` (`iof_base_frame.c`) + +A convenience used by **other** frameworks (`rmaps`, `ras`, `state`) to +emit a formatted string as though it were `stdout` from a given source +proc — e.g. `--display-map` / allocation dumps. It wraps the string in a +`prte_iof_deliver_t` and calls `PMIx_server_IOF_deliver` so the output +threads through the same PMIx output path as real proc output. It does +not touch the sink engine. + +### Fork-time setup helpers (`iof_base_setup.[ch]`) + +Called by `odls` around the `fork()` of each app proc: + +- **`prte_iof_base_setup_prefork(opts)`** — before fork: create the + `stdout` pipe (or a pty if `usepty` and PTY support is compiled in), + the `stderr` pipe, and — only if `opts->connect_stdin` — the `stdin` + pipe. `connect_stdin` is set true only for the proc that receives stdin + (normally rank 0); everyone else gets `/dev/null`. +- **`prte_iof_base_setup_child(opts, env)`** — in the child after fork: + `dup2` the pipe ends onto fds 0/1/2, disable echo on a pty, and wire + `stdin` to `/dev/null` when not connected. +- **`prte_iof_base_setup_parent(name, opts)`** — in the daemon/HNP after + fork: call `prte_iof.pull(name, PRTE_IOF_STDIN, p_stdin[1])` (if + connecting stdin) and then `prte_iof.push(name, PRTE_IOF_STDOUT, …)` and + `push(name, PRTE_IOF_STDERR, …)`. This is where the abstract module + vtable meets real file descriptors. + +### The `prte_iof` global and RML tags + +- `prte_iof` (in [`iof_base_frame.c`](base/iof_base_frame.c)) is the + selected module; everything outside the framework calls through it + (`prte_iof.push_stdin(...)`, `prte_iof.complete(...)`). +- `PRTE_RML_TAG_IOF_HNP` — daemons → HNP (forwarded output and XON/XOFF). +- `PRTE_RML_TAG_IOF_PROXY` — HNP → daemons (stdin, and xcast stdin to all). + +--- + +## The tag model (`iof_types.h`) + +Streams and control signals share one `prte_iof_tag_t` (a `uint16_t`) +bitmask: + +| Tag | Value | Meaning | +|-----|-------|---------| +| `PRTE_IOF_STDIN` | `0x0001` | stdin stream | +| `PRTE_IOF_STDOUT` | `0x0002` | stdout stream | +| `PRTE_IOF_STDERR` | `0x0004` | stderr stream | +| `PRTE_IOF_STDMERGE` | `0x0006` | stdout+stderr combined | +| `PRTE_IOF_STDDIAG` | `0x0008` | internal diagnostic stream | +| `PRTE_IOF_STDOUTALL` | `0x000e` | stdout+stderr+diag | +| `PRTE_IOF_STDALL` | `0x000f` | every stream | +| `PRTE_IOF_EXCLUSIVE` | `0x0100` | exclusive-access flag | +| `PRTE_IOF_XON` / `PRTE_IOF_XOFF` | `0x1000` / `0x2000` | flow control | +| `PRTE_IOF_PULL` / `PRTE_IOF_CLOSE` | `0x4000` / `0x8000` | tool requests | + +Because tags are bit flags, tests are always `tag & PRTE_IOF_STDOUT`, and +`close`/`push` handle multiple stream bits in one call. The old +`iof.h` comment listing `STDIN=0, STDOUT=1, …` describes a **retired** +enumeration — ignore it; `iof_types.h` is authoritative. + +--- + +## Flow control + +stdin can outrun a slow reader. The framework applies simple XON/XOFF +back-pressure keyed on `PRTE_IOF_MAX_INPUT_BUFFERS` (50 queued chunks): + +- On a daemon, when `prte_iof_base_write_output` reports the stdin sink + backlog has crossed 50 (or a write errors out), + `prte_iof_prted_send_xonxoff(PRTE_IOF_XOFF)` tells the HNP to stop + sending, latched by `prte_mca_iof_prted_component.xoff`. When the + backlog drains below 50, the daemon sends `PRTE_IOF_XON`. +- On the HNP, `push_stdin` to a *local* proc returns + `PRTE_ERR_OUT_OF_RESOURCE` when its own sink passes the same threshold, + which propagates back to stop the read that is producing stdin. + +`prte_iof_base_output_limit` (MCA param `iof_base_output_limit`, default +`INT_MAX`) is the harder ceiling: if a sink's backlog exceeds it, the +write handler concludes something is permanently wedged and forces the +job to exit. + +--- + +## Threading + +Everything here runs on the **progress thread** via libevent fd/timer +handlers — read handlers, write handlers, and the RML receives are all +event callbacks. There is no locking inside the framework because there is +no other thread touching this state. Consequences: + +- All `read()`/`write()` calls are on **non-blocking** fds (the components + `fcntl(O_NONBLOCK)` every fd before arming its event). Never issue a + blocking I/O call from a handler. +- `PMIX_ACQUIRE_OBJECT` / `PMIX_POST_OBJECT` bracket handler entry/exit so + the object's memory is coherent when the event fires — keep them when + you add a handler. +- Reads are bounded to `PRTE_IOF_BASE_MSG_MAX` (4096) bytes per fire; the + handler re-arms itself (`PRTE_IOF_READ_ACTIVATE`) to come back for more, + rather than looping the fd dry, so one chatty proc can't starve the + progress thread. + +--- + +## Conventions and gotchas + +- **`push` is output, `pull` is stdin.** Repeat it until it sticks. The + docstrings in [`iof.h`](iof.h) and the [`README.txt`](README.txt) are + historical and describe a design that no longer matches the code. +- **The PMIx server does the actual output.** `stdout`/`stderr` bytes are + handed to `PMIx_server_IOF_deliver`; terminal writing, `--output` + tagging, per-rank files, and tool "pull" copies are the PMIx server's + job, not this framework's. Do not try to add tagging here. +- **`numbytes == 0` is a sentinel, not a no-op.** A zero-byte chunk / + zero-byte read means "flush and close this stream." Preserve that + meaning on every write path. +- **Activate both read events together.** `push` defines `revstdout` and + `revstderr` separately but only *activates* them once both exist — + otherwise one firing early (e.g. immediate EOF) can drive the proc to + `IOF_COMPLETE` before the other stream is even wired. The `activated` + flag guards double-activation. +- **IOF completion drives proc state.** When both `revstdout` and + `revstderr` are gone (EOF or close), the read handler fires + `PRTE_PROC_STATE_IOF_COMPLETE` — the state machine waits on this before + fully reaping a proc. Don't null a read-event slot without going through + that check. +- **Vestigial declarations.** `prte_iof_base_flush()` (declared in + `base.h`), and `prte_iof_hnp_stdin_cb` / `prte_iof_hnp_stdin_check` / + the `stdinsig` field (declared in `hnp/iof_hnp.h`) have **no live + definitions/uses** — leftovers from when `mpirun` read its own terminal + stdin directly. Today stdin arrives via the PMIx server calling + `prte_iof.push_stdin`. Don't wire new code to these ghosts. +- **The version macro is `PRTE_IOF_BASE_VERSION_2_0_0`.** Match it in any + new component's struct. +- Standard PRRTE rules still apply: `prte_config.h` first, braces on every + block, `NULL ==`/constant-on-left comparisons, no new compiler warnings, + `PRTE_ERROR_LOG` for unexpected errors. + +--- + +## Debugging + +```sh +prte --prtemca iof_base_verbose 5 ... # trace read/forward/write decisions +prun --output tag ... # prefix each line with its source rank (PMIx server) +prun --output timestamp ... # timestamp each line +prun --output-filename DIR ... # per-rank files instead of the terminal +``` + +`iof_base_verbose` ≥1 already narrates every fd read, byte count, sink +activation, and forward; ≥20 traces fd closes and sink teardown. Because +the split between "captured by iof" and "emitted by the PMIx server" is +where output bugs usually hide, correlate iof verbosity with the fact that +the actual terminal write happens downstream in the PMIx server. + +--- + +## Where to go next + +Each component directory has its own `AGENTS.md`: + +- [`hnp/AGENTS.md`](hnp/AGENTS.md) — the HNP hub: collects all output, + emits it, injects stdin. Read this first. +- [`prted/AGENTS.md`](prted/AGENTS.md) — the per-daemon relay: captures + local output and forwards it; delivers stdin locally. diff --git a/src/mca/iof/CLAUDE.md b/src/mca/iof/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/iof/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/iof/hnp/AGENTS.md b/src/mca/iof/hnp/AGENTS.md new file mode 100644 index 0000000000..e863a46333 --- /dev/null +++ b/src/mca/iof/hnp/AGENTS.md @@ -0,0 +1,170 @@ +# AGENTS.md — `iof/hnp` (the HNP I/O hub) + +Component guide for `src/mca/iof/hnp/`. Read the +[framework guide](../AGENTS.md) first for the module vtable, the +sink/read/write/proc structs, the tag model, and the base sink engine +referenced throughout. + +--- + +## Role and selection + +`hnp` is the **hub** of I/O forwarding. It runs **only in the HNP / DVM +master** — the `prte`, `prun`, or `mpirun` process that owns the user's +terminal. *All* IOF traffic converges here: output from the HNP's own +local children, and output relayed over the RML from every `prted`. There +is no proxy-to-proxy path — a daemon that wants another daemon's output +gets it as `prted → HNP → tool`. + +Selection is role-gated. `prte_iof_hnp_query()` in +[`iof_hnp_component.c`](iof_hnp_component.c) returns priority **100** and +the module **only when `PRTE_PROC_IS_MASTER`**; otherwise it returns +`-1`/`PRTE_ERROR` and declines. Since a process is either master or +daemon, this module and `prted` never compete for real. + +The component keeps two pieces of state (in +[`iof_hnp.h`](iof_hnp.h)'s `prte_mca_iof_hnp_component_t`): a +`pmix_list_t procs` of `prte_iof_proc_t` endpoint bundles, and an +(unused, vestigial) `stdinsig` event. + +--- + +## Files + +| File | Contents | +|------|----------| +| `iof_hnp_component.c` | Registration + `query` (gate on `PRTE_PROC_IS_MASTER`, priority 100). | +| `iof_hnp.c` | The module vtable: `init`, `hnp_push`, `hnp_pull`, `hnp_close`, `hnp_complete`, `finalize`, `push_stdin`, and the local `stdin_write_handler`. | +| `iof_hnp_read.c` | `prte_iof_hnp_read_local_handler` — reads the HNP's *own* children's stdout/stderr and emits them via the PMIx server. | +| `iof_hnp_receive.c` | `prte_iof_hnp_recv` — the `PRTE_RML_TAG_IOF_HNP` handler: unpacks daemon-forwarded output and emits it via the PMIx server. | +| `iof_hnp_send.c` | `prte_iof_hnp_send_data_to_endpoint` — packs stdin and sends it to a daemon (or xcasts to all) over `PRTE_RML_TAG_IOF_PROXY`. | +| `iof_hnp.h` | Component struct + prototypes for the above. | + +The module vtable (`prte_iof_hnp_module`) sets `push_stdin` — the HNP is +the **only** component that implements it, because injecting stdin into +the job is inherently a master-side operation. + +--- + +## The two output paths converge on the PMIx server + +Whether output originates locally or remotely, the HNP's job is to hand +it to `PMIx_server_IOF_deliver()`, which performs the actual terminal / +`--output`-file / tool-pull emission. Both paths build a +`prte_iof_deliver_t` (source proc + `pmix_byte_object_t`) and translate +the iof stream bits into PMIx channel bits (`PMIX_FWD_STDOUT_CHANNEL`, +`PMIX_FWD_STDERR_CHANNEL`, `PMIX_FWD_STDDIAG_CHANNEL`). + +### Path 1 — local children (`iof_hnp_read.c`) + +`hnp_push` (called from `prte_iof_base_setup_parent` for procs the HNP +forked itself) arms a `PRTE_IOF_READ_EVENT` on the read end of the child's +stdout/stderr pipe, with `prte_iof_hnp_read_local_handler` as the callback. +On each fire the handler: + +1. `read(fd, …)` up to `PRTE_IOF_BASE_MSG_MAX` (4096) bytes. +2. `numbytes < 0` with `EAGAIN`/`EINTR` → re-arm and return; other + `numbytes <= 0` → EOF, jump to `CLEAN_RETURN`. +3. Otherwise wrap the bytes in a `prte_iof_deliver_t` and call + `PMIx_server_IOF_deliver` — the data goes **straight out**, never over + the RML, because these are the HNP's own children. +4. Re-arm the read event. + +At `CLEAN_RETURN` (EOF/error) it releases the finished read event +(`revstdout` or `revstderr`) and, when both are gone, fires +`PRTE_PROC_STATE_IOF_COMPLETE` for the proc. It carefully `PMIX_RETAIN`s +the proc across the release to avoid a recursive free. + +### Path 2 — remote daemons (`iof_hnp_receive.c`) + +`prte_iof_hnp_recv` is the persistent `PRTE_RML_TAG_IOF_HNP` receive posted +by `init()`. A daemon forwards output as a packed buffer of +`{ tag (uint16), origin proc, numbytes (int32), bytes }`. The handler +unpacks those, finds-or-creates the `prte_iof_proc_t` for `origin`, maps +the tag to PMIx channel bits, and calls `PMIx_server_IOF_deliver`. Same +destination as Path 1 — the PMIx server — just sourced from a remote node. +XON/XOFF flow-control messages arrive on this same tag (tag-only buffers); +they are consumed here as part of the stdin back-pressure protocol. + +--- + +## Stdin injection (`push_stdin`, `hnp_pull`, `iof_hnp_send.c`) + +The HNP is where stdin *enters* the DVM. The PMIx server calls +`prte_iof.push_stdin(dst_name, data, sz)` (from +`src/prted/pmix/pmix_server_gen.c`) when the user's terminal (or a tool) +produces input. `push_stdin` routes it: + +- **Wildcard rank** (`PMIX_RANK_WILDCARD`) → `prte_iof_hnp_send_data_to_endpoint` + with a wildcard host, which `xcast`s the buffer to every daemon over + `PRTE_RML_TAG_IOF_PROXY`. +- Otherwise it looks up the daemon hosting `dst_name` + (`prte_get_proc_daemon_vpid`). If that daemon **isn't** the HNP, it + sends the buffer to that daemon over `PRTE_RML_TAG_IOF_PROXY` (a + zero-byte payload tells the daemon to close the proc's stdin). +- If the target proc is **local** to the HNP, it writes directly into that + proc's stdin sink via `prte_iof_base_write_output(&name, PRTE_IOF_STDIN, + data, sz, proct->stdinev->wev)`. Crossing `PRTE_IOF_MAX_INPUT_BUFFERS` + (50) returns `PRTE_ERR_OUT_OF_RESOURCE` to apply back-pressure. + +`hnp_pull` is how a local proc's stdin *sink* gets registered: called from +`prte_iof_base_setup_parent` with `PRTE_IOF_STDIN` and the write end of the +proc's stdin pipe, it `PRTE_IOF_SINK_DEFINE`s a `prte_iof_sink_t` on +`proct->stdinev` (handler = the local `stdin_write_handler`), tags it with +the HNP as the owning `daemon`, and activates it. Only `PRTE_IOF_STDIN` is +accepted; anything else returns `PRTE_ERR_NOT_SUPPORTED`. + +`iof_hnp_send.c` also short-circuits: if the destination is a daemon in +the HNP's own job family and `prte_dvm_abort_ordered` is set, it drops the +send (but still forwards to non-daemon tools that may be watching an abort). + +### `stdin_write_handler` (in `iof_hnp.c`) + +The HNP's own sink write callback drains `wev->outputs` to the proc's +stdin fd with the standard non-blocking dance (EAGAIN/EINTR → prepend and +re-arm; partial write → `memmove` + prepend + re-arm; `numbytes == 0` → +close). It differs from the base `prte_iof_base_write_handler` in two +ways: it dumps pending data immediately if `prte_abnormal_term_ordered` +(the DVM is aborting), and it honors the sink's `closed` flag, releasing +the sink once the last queued byte is written. + +--- + +## Close and completion + +- `hnp_close(peer, source_tag)` releases the sink and/or read events named + by the tag bits, and drops the `prte_iof_proc_t` from `procs` once all + three (`stdinev`, `revstdout`, `revstderr`) are gone. +- `hnp_complete(jdata)` sweeps `procs` for any entry whose nspace matches + the finished job and releases it — a safety net for endpoints that + outlived their proc. +- `finalize()` destructs the `procs` list. (`init()` had posted the + `PRTE_RML_TAG_IOF_HNP` receive and constructed the list.) + +--- + +## Gotchas when editing + +- **Everything routes through the PMIx server for output.** Don't add + terminal-writing or tagging logic here; emit via `PMIx_server_IOF_deliver` + and let the server honor `--output`. The `prte_iof_deliver_t` you pass is + freed by the delivery completion callback (`lkcbfunc`) — on a failed + submit you must `PMIX_RELEASE` it yourself, as the code does. +- **Retain-before-release on completion.** `read_local_handler`'s + `CLEAN_RETURN` and the several places that null `revstdout`/`revstderr` + can recursively free the proc; keep the `PMIX_RETAIN(proct)` guard. +- **Both read events must be defined before either is activated.** `hnp_push` + only flips the `activated` flags once `revstdout && revstderr` exist, + so an immediate EOF on one stream can't declare the proc IOF-complete + before the other is wired. +- **The `query` has a copy-paste quirk:** it tests `!PRTE_PROC_IS_MASTER && + !PRTE_PROC_IS_MASTER` (the same predicate twice). It is functionally + correct (only the master runs here) but if you touch it, fix it to a + single clean condition rather than propagating the duplication. +- **Zero-byte stdin means close.** `push_stdin` deliberately forwards + zero-length payloads so a preceding buffer is flushed and the proc's + stdin fd is then closed. Preserve that. +- **`stdinsig`, `prte_iof_hnp_stdin_cb`, `prte_iof_hnp_stdin_check` are + dead.** Declared in `iof_hnp.h` but unimplemented/unused — remnants of + direct terminal-stdin reading. Ignore them; stdin now enters via + `push_stdin`. diff --git a/src/mca/iof/hnp/CLAUDE.md b/src/mca/iof/hnp/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/iof/hnp/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/iof/prted/AGENTS.md b/src/mca/iof/prted/AGENTS.md new file mode 100644 index 0000000000..7e57a32dec --- /dev/null +++ b/src/mca/iof/prted/AGENTS.md @@ -0,0 +1,179 @@ +# AGENTS.md — `iof/prted` (the per-daemon I/O relay) + +Component guide for `src/mca/iof/prted/`. Read the +[framework guide](../AGENTS.md) first for the module vtable, the +sink/read/write/proc structs, the tag model, and the base sink engine +referenced throughout. + +--- + +## Role and selection + +`prted` is the **relay**. It runs in **every per-node daemon** (`prted`), +where the application procs actually live. Its two jobs are: + +1. **Capture** each local proc's `stdout`/`stderr` and forward the bytes + to the HNP over the RML (while also handing a copy to the local PMIx + server), and +2. **Deliver** `stdin` that arrives from the HNP down to the local + proc(s) that pulled it. + +A daemon never talks IOF to another daemon — output goes up to the HNP, +stdin comes down from the HNP. + +Selection is role-gated. `prte_iof_prted_query()` in +[`iof_prted_component.c`](iof_prted_component.c) returns priority **80** +and the module **only when `PRTE_PROC_IS_DAEMON`**; otherwise it declines +with `-1`/`PRTE_ERROR`. + +Component state (in [`iof_prted.h`](iof_prted.h)'s +`prte_mca_iof_prted_component_t`): a `pmix_list_t procs` of endpoint +bundles and a single `bool xoff` latch for stdin flow control toward the +HNP. + +--- + +## Files + +| File | Contents | +|------|----------| +| `iof_prted_component.c` | Registration + `query` (gate on `PRTE_PROC_IS_DAEMON`, priority 80). | +| `iof_prted.c` | The module vtable: `init`, `prted_push`, `prted_pull`, `prted_close`, `prted_complete`, `finalize`, and the local `stdin_write_handler`. **No `push_stdin`** — that's HNP-only. | +| `iof_prted_read.c` | `prte_iof_prted_read_handler` — reads a local proc's stdout/stderr, echoes locally via the PMIx server, and forwards to the HNP. | +| `iof_prted_receive.c` | `prte_iof_prted_recv` (the `PRTE_RML_TAG_IOF_PROXY` handler for incoming stdin) + `prte_iof_prted_send_xonxoff` (flow control back to the HNP). | +| `iof_prted.h` | Component struct + prototypes. | + +The vtable leaves `push_stdin` unset (`NULL`): a daemon receives stdin +passively over the RML rather than being asked to inject it. + +--- + +## Output capture and forward (`prted_push`, `iof_prted_read.c`) + +`prted_push` (called from `prte_iof_base_setup_parent` after the daemon +forks an app proc) sets the fd non-blocking, finds-or-creates the +`prte_iof_proc_t` for the proc, verifies the proc's job data exists +locally (`prte_get_job_data_object`), and arms a `PRTE_IOF_READ_EVENT` on +the read end of the stdout or stderr pipe (per `src_tag`) with +`prte_iof_prted_read_handler` as the callback. As in the HNP, both read +events are *defined* first and only *activated* (guarded by `activated`) +once `revstdout && revstderr` both exist, so an early EOF on one stream +can't prematurely trip IOF completion. + +On each fire, `prte_iof_prted_read_handler`: + +1. `read(rev->fd, …)` up to `PRTE_IOF_BASE_MSG_MAX` (4096) bytes. +2. `numbytes < 0` with `EAGAIN`/`EINTR` → re-arm and return; other + `numbytes <= 0` → EOF, jump to `CLEAN_RETURN`. +3. **Local echo:** wrap the bytes in a `prte_iof_deliver_t`, map the tag + to PMIx channel bits, and call `PMIx_server_IOF_deliver` — this lets a + local PMIx server / tool see the output without a round-trip. +4. **Forward to HNP:** pack `{ tag (uint16), proc name, numbytes (int32), + bytes }` into a `pmix_data_buffer_t` and + `PRTE_RML_RELIABLE_SEND(..., PRTE_PROC_MY_HNP->rank, buf, + PRTE_RML_TAG_IOF_HNP)`. The tag is packed first so a pure flow-control + message can be just the tag. +5. Re-arm the read event. + +At `CLEAN_RETURN` (EOF/error) it releases the finished read event and, +when both `revstdout` and `revstderr` are gone, fires +`PRTE_PROC_STATE_IOF_COMPLETE` for the proc — the same completion signal +the HNP uses, so the state machine can reap the proc. + +Note the daemon **does not buffer** output waiting for HNP acks: each read +is immediately reliable-sent. The wire format is decoded by the HNP's +`prte_iof_hnp_recv`. + +--- + +## Stdin delivery (`prted_pull`, `iof_prted_receive.c`) + +`prted_pull` registers where local stdin goes. Called from +`prte_iof_base_setup_parent` with `PRTE_IOF_STDIN` and the write end of the +proc's stdin pipe, it sets the fd non-blocking, finds-or-creates the +proc's endpoint (matching by full name via +`prte_util_compare_name_fields`), and `PRTE_IOF_SINK_DEFINE`s a +`prte_iof_sink_t` on `proct->stdinev` with the local `stdin_write_handler`. +Only `PRTE_IOF_STDIN` is supported; anything else returns +`PRTE_ERR_NOT_SUPPORTED`. + +`prte_iof_prted_recv` is the persistent `PRTE_RML_TAG_IOF_PROXY` receive +posted by `init()`. Incoming buffers carry `{ stream (uint16), target +proc, bytes }`: + +1. Unpack the stream; if it isn't `PRTE_IOF_STDIN` it's a protocol error + (`PRTE_ERR_COMM_FAILURE`). (Flow-control tags are handled by the HNP, + not here.) +2. Unpack the target proc and the data. +3. Walk `procs`, matching the target by nspace and by rank + (`PMIX_CHECK_RANK` honors wildcard, so a broadcast reaches every local + proc that pulled stdin). For each match with a live `stdinev`, call + `prte_iof_base_write_output(&target, stream, data, numbytes, + proct->stdinev->wev)`. +4. If that write backs up past `PRTE_IOF_MAX_INPUT_BUFFERS` (50) and we + haven't already, latch `xoff = true` and + `prte_iof_prted_send_xonxoff(PRTE_IOF_XOFF)` to tell the HNP to stop + sending stdin. + +A zero-byte payload is forwarded through `write_output` too, so it flushes +preceding data and then closes the proc's stdin fd. + +### `stdin_write_handler` (in `iof_prted.c`) + +The daemon's sink write callback drains `wev->outputs` to the local proc's +stdin fd with the usual non-blocking handling (EAGAIN/EINTR → prepend + +re-arm; partial write → `memmove` + prepend + re-arm; `numbytes == 0` → +release the write event and null `sink->wev` to close). Its distinctive +behavior is **flow-control recovery**: on a fatal write error it sends +`PRTE_IOF_XOFF`, and at the `CHECK` label, whenever `xoff` is latched and +the backlog has fallen below `PRTE_IOF_MAX_INPUT_BUFFERS`, it clears the +latch and sends `PRTE_IOF_XON` to resume stdin from the HNP. The inline +`RHC:` comment flags the unsolved case of several procs fighting over +XON/XOFF at different consumption rates. + +### `prte_iof_prted_send_xonxoff` (`iof_prted_receive.c`) + +Builds a tag-only `pmix_data_buffer_t` (just the `PRTE_IOF_XON`/`XOFF` +tag) and reliable-sends it to the HNP on `PRTE_RML_TAG_IOF_HNP` — the same +tag used for forwarded output, distinguished by the leading tag value. + +--- + +## Close, completion, teardown + +- `prted_close(peer, source_tag)` releases the sink and/or read events for + the tag bits and drops the proc from `procs` once all three streams are + gone. For a daemon this "just" closes local fds — there is no remote + state to unwind. +- `prted_complete(jdata)` sweeps `procs` and releases any endpoint whose + nspace matches the finished job. +- `finalize()` `PMIX_LIST_DESTRUCT`s `procs` **and** + `PRTE_RML_CANCEL`s the `PRTE_RML_TAG_IOF_PROXY` receive — unlike the HNP + module, the daemon explicitly cancels its RML receive on shutdown. + +--- + +## Gotchas when editing + +- **Two consumers per read: local PMIx server + the HNP.** Every + successful read both delivers locally (`PMIx_server_IOF_deliver`) and + reliable-sends to the HNP. Dropping either breaks a use case (local + tools attached to the daemon vs. the user's terminal). The + `prte_iof_deliver_t` is freed by the delivery callback; on a failed + submit, `PMIX_RELEASE` it as the code does. +- **Forwarded output is never acked/buffered here.** The daemon fires and + forgets over the RML; backpressure only exists on the *stdin* side via + XON/XOFF. Don't assume symmetric flow control. +- **Guard the `xoff` latch.** It's a single component-wide bool, so only + toggle it through the send-XOFF (on backup) / send-XON (on drain) pair. + Spurious toggles cause stdin stalls or floods. +- **`prte_get_job_data_object` must succeed in `prted_push`.** A proc + whose job data isn't present locally is an error (`PRTE_ERR_NOT_FOUND`); + don't relax that — it means the daemon was asked to forward for a proc + it doesn't own. +- **Match on full name for stdin, nspace+rank for delivery.** `prted_pull` + uses `prte_util_compare_name_fields(PRTE_NS_CMP_ALL, …)`; `recv` uses + `PMIX_CHECK_NSPACE` + `PMIX_CHECK_RANK` (wildcard-aware). Keep those + matching rules — they're what make wildcard-stdin fan-out work. +- **Zero-byte stdin means close.** Preserve the zero-length forward path + through `write_output`. diff --git a/src/mca/iof/prted/CLAUDE.md b/src/mca/iof/prted/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/iof/prted/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/odls/AGENTS.md b/src/mca/odls/AGENTS.md new file mode 100644 index 0000000000..02e86acc26 --- /dev/null +++ b/src/mca/odls/AGENTS.md @@ -0,0 +1,431 @@ +# AGENTS.md — The `odls` Framework (Daemon Local Launch Subsystem) + +Orientation for AI agents and human contributors working in +`src/mca/odls/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`odls` is the **PRTE Daemon's Local Launch Subsystem**. It is the code +that actually **forks and execs the application processes** on each node, +tracks them, signals them, reaps them via `waitpid`, and reports their +state transitions back into the job state machine. Where `rmaps` decides +*which* proc goes *where* (on the HNP), `odls` is what *does the launch* +(on every daemon). + +Unusually for an MCA framework, `odls` has two very different jobs +depending on where the process runs: + +| Role | Where | What odls does | +|------|-------|----------------| +| **HNP / DVM master** | one process | Serializes the computed placement into a single **launch message** (`get_add_procs_data`) that is broadcast to all daemons. | +| **prted (daemon)** | every node | Parses that message (`construct_child_list`), works out which procs are *local*, and fork/execs them (`launch_local_procs`). | + +The HNP is itself a daemon (vpid 0), so it also launches any local procs +assigned to its own node — but its launch message construction is the +part unique to it. + +### Place in the launch state machine + +``` +… → MAP → MAP_COMPLETE → SYSTEM_PREP → LAUNCH_DAEMONS → … → LAUNCH_APPS → RUNNING → … + ▲ + └── odls runs here +``` + +The flow that drives odls (see `src/mca/plm/base/plm_base_launch_support.c` +and `src/prted/prted_comm.c`): + +1. A job reaches `PRTE_JOB_STATE_LAUNCH_APPS`. On the HNP, + `prte_plm_base_launch_apps()` packs the daemon command + (`PRTE_DAEMON_ADD_LOCAL_PROCS`, or `PRTE_DAEMON_DVM_ADD_PROCS` for a + fixed DVM) into `jdata->launch_msg`, then calls + `prte_odls.get_add_procs_data()` to append the placement/regex/setup + payload. +2. That call ends (asynchronously, after `PMIx_server_setup_application` + returns) by activating `PRTE_JOB_STATE_SEND_LAUNCH_MSG`, which xcasts + `jdata->launch_msg` to every daemon over the RML. +3. Each daemon's `prted_comm.c` dispatch sees `PRTE_DAEMON_ADD_LOCAL_PROCS` + and calls `prte_odls.launch_local_procs(buffer)`. +4. `launch_local_procs` → `construct_child_list` (decode) → + `PRTE_ACTIVATE_LOCAL_LAUNCH` → `launch_local` (per-app fork/exec) → + each child transitions to `PRTE_PROC_STATE_RUNNING`. + +The same module also fields the kill/signal daemon commands +(`PRTE_DAEMON_KILL_LOCAL_PROCS`, `PRTE_DAEMON_SIGNAL_LOCAL_PROCS`) and +the errmgr's restart path. + +--- + +## Directory layout + +``` +odls/ + odls.h # module vtable (5 fn ptrs) + component typedef + version macro + odls_types.h # PRTE_DAEMON_* command flags; child-error pipe struct + base/ + base.h # framework globals struct, base-fn prototypes, the two caddy classes, + # PRTE_ACTIVATE_LOCAL_LAUNCH / PRTE_ODLS_SET_ERROR macros + odls_base_frame.c # open/close/register; MCA params; the spawn-thread pool; class instances + odls_base_select.c # component selection (pick ONE, highest priority) + odls_base_default_fns.c # THE big one: build msg, parse msg, wireup, env setup, spawn, waitpid, + # kill, restart — everything a component reuses + odls_base_bind.c # prte_odls_base_set(): apply cpu/memory binding in the child pre-exec, + # proxy binding errors up the pipe + help-prte-odls-base.txt # xterm-related error text + pdefault/ # the only component (pri 10): real fork()/execve() launcher +``` + +Read `odls.h` and `base/base.h` first (the contract and the shared data +structures), then `base/odls_base_default_fns.c`, which is where almost +all real work lives. The `pdefault` component is a thin shell around the +base helpers — read it last. + +--- + +## The module contract + +Every odls component fills in a `prte_odls_base_module_t` (declared in +`odls.h`) with five function pointers: + +```c +typedef struct prte_odls_base_module_1_3_0_t { + prte_odls_base_module_get_add_procs_data_fn_t get_add_procs_data; + prte_odls_base_module_launch_local_processes_fn_t launch_local_procs; + prte_odls_base_module_kill_local_processes_fn_t kill_local_procs; + prte_odls_base_module_signal_local_process_fn_t signal_local_procs; + prte_odls_base_module_restart_proc_fn_t restart_proc; +} prte_odls_base_module_t; +``` + +| Function | Signature | Runs on | Meaning | +|----------|-----------|---------|---------| +| `get_add_procs_data` | `(pmix_data_buffer_t *data, pmix_nspace_t job)` | HNP | Serialize the whole job (proc→node map, regex nodemap/procmap, personality, uid/gid, app-setup info) into `data` for broadcast. Returns `PRTE_SUCCESS`/error. | +| `launch_local_procs` | `(pmix_data_buffer_t *data)` | daemon | Decode the message, build this node's child list, fork/exec the local procs. Returns `PRTE_SUCCESS`/error. | +| `kill_local_procs` | `(pmix_pointer_array_t *procs)` | daemon | Kill the listed procs (`NULL` ⇒ all local procs). Escalates SIGCONT→SIGTERM→SIGKILL. | +| `signal_local_procs` | `(const pmix_proc_t *proc, int32_t signal)` | daemon | Deliver `signal` to one proc (`NULL` ⇒ all local procs). | +| `restart_proc` | `(prte_proc_t *child)` | daemon | Re-fork a single already-known child (fault recovery / comm-spawn restart). | + +The return protocol is the ordinary PRRTE one: `PRTE_SUCCESS` or a +`PRTE_ERR_*`. Unlike `rmaps`, there is **no** "take next option" — one +component wins and owns every call. Errors on the daemon side almost +always end by activating a proc or job **error state** rather than +returning up the stack, because the launch runs asynchronously on the +event loop (`PRTE_ACTIVATE_PROC_STATE(..., PRTE_PROC_STATE_FAILED_TO_LAUNCH)`, +`PRTE_ACTIVATE_JOB_STATE(..., PRTE_JOB_STATE_NEVER_LAUNCHED)`). + +The version macro is `PRTE_ODLS_BASE_VERSION_2_0_0`. + +--- + +## Component selection is "pick one" + +`prte_odls_base_select()` (`odls_base_select.c`) is the standard MCA +"select the single best component" pattern: it calls `pmix_mca_base_select`, +copies the winning module into the global `prte_odls`, and everything in +the tree calls through `prte_odls.()`. There is currently exactly one +component — `pdefault`, priority **10** — deliberately low so a +site-specific launcher could override it. The framework's open/select +logic only runs the launch machinery inside a daemon; a tool never +selects an odls module for launching. + +--- + +## What `base/` provides — the heart of the framework + +Because there is only one component and it delegates almost everything, +**the base is the framework.** A component supplies just the primitive +`fork_local_proc` (and the raw `kill`/`signal` syscalls); the base does +message construction, parsing, wireup, environment assembly, threading, +`waitpid` interpretation, and cleanup. Walk these in order. + +### 1. Framework globals and the spawn-thread pool (`odls_base_frame.c`) + +`prte_odls_globals` (`prte_odls_globals_t` in `base.h`) holds: + +- **`ev_bases` / `ev_threads` / `num_threads` / `max_threads` / `cutoff` / + `next_base`** — a pool of libevent progress threads used to *parallelize + forking* when a node hosts many local procs. `prte_odls_base_start_threads()` + decides how many to spin up: a persistent DVM uses `max_threads` (default + 16); otherwise, below `cutoff` (default 32) local procs it uses **zero** + dedicated threads (fork straight on `prte_event_base`), and above it + scales to `num_local_procs / 8` capped at `max_threads`. + `prte_odls_base_harvest_threads()` tears them down. +- **`xterm_ranks` / `xtermcmd`** — support for `--xterm`: a list of ranks + whose output should be shown in separate `xterm` windows, parsed at + framework open from the `prte_xterm` global. +- **`signal_direct_children_only`** — MCA flag controlling whether signals + go to the child only or its whole process group. +- **`exec_agent`** — an optional wrapper command to exec instead of the app. + +MCA params, all under `prte odls base`: `max_threads`, `num_threads`, +`cutoff`, `signal_direct_children_only`, `exec_agent`. Framework open also +**unblocks `SIGCHLD`** (odls must see child deaths) and builds the xterm +command vector. Framework close reaps the thread pool and releases the +global `prte_local_children` array. + +This file also defines the two caddy classes (below) via +`PMIX_CLASS_INSTANCE`. + +### 2. Building the launch message — `prte_odls_base_default_get_add_procs_data()` + +Runs **on the HNP only**. Packs into the supplied buffer, in a strict +order that the parser below must mirror (there is a literal comment in the +source warning about this): + +1. An `int8` flag: were new daemons launched for this job? If so, pack a + nested buffer containing **every other active job** (`prte_job_pack`) + plus each proc's `parent` daemon vpid — so a freshly added daemon + learns about pre-existing jobs and can route collectives correctly. +2. The job being launched (`prte_job_pack`). +3. A **nodemap** regex and a **procmap** (ppn) regex, generated by asking + the PMIx server (`PMIx_generate_regex` / `PMIx_generate_ppn`) to + compress the node names and the per-node rank lists. +4. Job info: personality, a per-job network allocation request, the + launching user's `uid`/`gid`, and — if envars have not yet been + harvested — a `PMIX_SETUP_APP_ENVARS` directive. + +It then calls `PMIx_server_setup_application()` **asynchronously**. The +completion callback `setup_cbfunc()` packs whatever setup blob PMIx +returned (as a `PMIX_BYTE_OBJECT`) onto `jdata->launch_msg`, then activates +`PRTE_JOB_STATE_SEND_LAUNCH_MSG` and wakes the waiting thread. The function +blocks on a `prte_pmix_lock_t` until that callback fires. + +### 3. Parsing the message and wiring up — `prte_odls_base_default_construct_child_list()` + +Runs **on every daemon** (including the HNP). This is the mirror image of +`get_add_procs_data`, and the single most important function to understand: + +- Unpacks the "new daemons" flag; on a **non-master** daemon it unpacks the + prior-jobs blob, adds each unknown job to the local `prte_job_data` + array, reconnects each of its procs to the owning daemon's node, and + registers the nspace with the local PMIx server. (The **master already + has** all of this, so it discards its copy — `jdata->index = -1; + PMIX_RELEASE`.) +- Unpacks the job to launch. On the master it throws away the unpacked copy + and fetches the fully-populated local `prte_job_t`; on a daemon it keeps + the unpacked copy, creates a `map` if needed, and resolves the job's + **schizo** personality via `prte_schizo_base_detect_proxy()`. +- Unpacks the optional app-setup byte object, and folds any + `PMIX_SET/ADD/UNSET/PREPEND/APPEND_ENVAR` items into the job attributes + (prepended, so they apply before launch). +- **Wireup loop:** for every proc in the job, connect it to its node via + the parent daemon (`daemons->procs[pptr->parent]->node`), add the node to + the job map once (guarded by `PRTE_NODE_FLAG_MAPPED`, which is then reset), + and — crucially — decide **locality**: if `pptr->parent == + PRTE_PROC_MY_NAME->rank`, the proc is *mine*. Local procs are retained + onto the global **`prte_local_children`** array, flagged + `PRTE_PROC_FLAG_LOCAL`, counted into `jdata->num_local_procs`, and their + app is flagged `PRTE_APP_FLAG_USED_ON_NODE`. Restart jobs get + `PRTE_PROC_NOBARRIER` set. +- Registers the nspace with the PMIx server (`prte_pmix_server_register_nspace`), + runs `PMIx_server_setup_local_support` if setup info was present, starts + the spawn threads, and blocks until local support is ready. +- On any failure it activates `PRTE_JOB_STATE_NEVER_LAUNCHED` so the HNP + doesn't hang waiting for a daemon that silently died. + +### 4. Kicking off the fork — `PRTE_ACTIVATE_LOCAL_LAUNCH` and `prte_odls_base_default_launch_local()` + +The component's `launch_local_procs` finishes by invoking the +`PRTE_ACTIVATE_LOCAL_LAUNCH(job, fork_local_proc)` macro (in `base.h`), +which allocates a `prte_odls_launch_local_t` caddy, stashes the component's +`fork_local` primitive on it, and posts `prte_odls_base_default_launch_local` +to `prte_event_base`. + +`prte_odls_base_default_launch_local()` is the per-node launch driver: + +- Records a baseline `getcwd` (it will `chdir` around per app and must + return here). +- Enforces the **system limits** on total children and open file + descriptors; if over budget it retries via a `PRTE_DETECT_TIMEOUT` timer + (up to a few times) rather than failing outright. +- For each **app used on this node**: sets up the working directory + (`setup_path`, honoring `PRTE_APP_SSNDIR_CWD` / `PRTE_APP_USER_CWD`), + merges `prte_launch_environ` into `app->env`, applies env directives + (`process_envars` — the SET/ADD/UNSET/PREPEND/APPEND handling, with app + attributes trumping job attributes), calls the schizo's `setup_fork`, + links prepositioned files (`prte_filem`), checks the executable, and + applies resource limits. +- For each **local child of that app** in `INIT`/`RESTART` state: registers + the `waitpid` callback (`prte_wait_cb` → `prte_odls_base_default_wait_local_proc`), + sets `PRTE_PROC_FLAG_ALIVE`, allocates a **`prte_odls_spawn_caddy_t`**, + sets up IOF (`prte_iof_base_setup_prefork` / `setup_parent`), picks the + next event base from the thread pool, and posts + `prte_odls_base_spawn_proc` to it. + +**STOP_ON_EXEC caveat:** if `PRTE_JOB_STOP_ON_EXEC` is set (debugger +attach), the fork is forced onto `prte_event_base` rather than a worker +thread, because the ptrace tracer must be the same thread that later +detaches — see the long comment near the thread-selection code. + +### 5. The spawn step — `prte_odls_base_spawn_proc()` + +Runs on the chosen event base. This is the last common code before the +component's raw fork: + +- Honors `PRTE_JOB_DO_NOT_SPAWN` (mapping-only "donotlaunch" jobs): just + mark the child `TERMINATED` and return. +- Calls `PMIx_server_setup_fork()` to inject the PMIx client environment. +- Resolves the actual command/argv: normal app, or `--xterm` wrapper, or a + per-job `PRTE_JOB_EXEC_AGENT`, or the global `exec_agent`; optionally + index-suffixes `argv[0]` with the rank (`PRTE_JOB_INDEX_ARGV`). +- Calls the component's **`cd->fork_local(cd)`** — the actual `fork`/`execve`. +- On success stores the pid (on the master) and activates + `PRTE_PROC_STATE_RUNNING`; on failure activates a failure state. + +### 6. Applying binding in the child — `prte_odls_base_set()` (`odls_base_bind.c`) + +Called from *inside the forked child* (by the component's `do_child`) +before `execve`. It reads the proc's computed `child->cpuset` (the hwloc +bitmap string the mapper produced) and calls `hwloc_set_cpubind` / +`prte_hwloc_base_set_process_membind_policy`. Because the child is not a +real PRTE process, **it cannot use normal error reporting** — instead it +writes rendered `show_help` messages back up the pipe to the parent +(`send_warn_show_help` / `send_error_show_help`, using the +`prte_odls_pipe_err_msg_t` struct from `odls_types.h`). Whether a binding +failure is fatal or a warning depends on `PRTE_BINDING_REQUIRED` and +`PRTE_BINDING_POLICY_IS_SET` (a *required, explicitly-requested* binding +that fails kills the child; a defaulted one degrades to a warning). If the +proc has no cpuset but the daemon itself is bound, the child is "freed" to +all allowed cpus. + +### 7. Reaping children — `prte_odls_base_default_wait_local_proc()` + +The `waitpid` callback, registered per child and fired by +`src/runtime/prte_wait.c` when SIGCHLD is reaped. It decodes +`proc->exit_code` (the raw wait status) into a proc state: + +- `WIFEXITED` + zero ⇒ `PRTE_PROC_STATE_WAITPID_FIRED`. +- `WIFEXITED` + nonzero, with `PRTE_JOB_ERROR_NONZERO_EXIT` set ⇒ + `PRTE_PROC_STATE_TERM_NON_ZERO`. +- Exited "normally" but never did the required PMIx init/finalize sync ⇒ + `PRTE_PROC_STATE_TERM_WO_SYNC` (checked against `PRTE_PROC_FLAG_REG` / + `PRTE_PROC_FLAG_HAS_DEREG` and `prte_allowed_exit_without_sync`). +- `WIFSIGNALED` ⇒ `PRTE_PROC_STATE_ABORTED_BY_SIG`, and the exit code is + rewritten to `signo + 128` (shell convention, so `prog` and `prun prog` + agree). +- Proc that called `prte_abort` ⇒ `PRTE_PROC_STATE_CALLED_ABORT`; a proc + ordered dead (`KILLED_BY_CMD`) is passed straight through. +- **STOP_ON_EXEC** (`WIFSTOPPED` + `SIGTRAP` under `PRTE_JOB_STOP_ON_EXEC`): + this is the debugger-attach stop. Detach with SIGSTOP so the child stays + parked for the debugger, re-register the waitpid, fire + `PRTE_PROC_STATE_READY_FOR_DEBUG`, and **do not** fall through to exit + handling. This detach must run on `prte_event_base` (see the fork + thread-affinity note above). + +It ends at `MOVEON:` by cancelling the wait tracker and activating the +computed proc state. + +### 8. Kill / signal / restart + +- **`prte_odls_base_default_kill_local_procs()`** — walks the requested + procs against `prte_local_children`, closes stdin IOF, cancels the + waitpid (to avoid races), then escalates **SIGCONT → SIGTERM → SIGKILL** + with `nanosleep` gaps, marking each `KILLED_BY_CMD`. It calls the + component's raw `kill_local(pid, signum)`. +- **`prte_odls_base_default_signal_local_procs()`** — finds the target + child (or all) and calls the component's raw `signal_local(pid, signum)`. +- **`prte_odls_base_default_restart_proc()`** — resets a single known + child's state/flags and re-dispatches it through `prte_odls_base_spawn_proc` + (same caddy/thread/IOF machinery as a first launch). + +--- + +## Key data structures + +| Type | Where | Purpose | +|------|-------|---------| +| `prte_odls_base_module_t` | `odls.h` | The 5-pointer vtable; the selected one lives in the global `prte_odls`. | +| `prte_odls_globals_t` / `prte_odls_globals` | `base.h` / `frame.c` | Framework-wide state: the spawn-thread pool, xterm ranks, exec agent, signal policy. | +| `prte_local_children` | `src/runtime/prte_globals` (a `pmix_pointer_array_t`) | The daemon's authoritative list of the procs it launched — every base fn iterates it. Allocated at framework open, released at close. | +| `prte_odls_spawn_caddy_t` | `base.h` | Per-child fork caddy: `cmd`, `wdir`, `argv`, `env`, `jdata`, `app`, `child`, IOF `opts`, and the `fork_local` fn ptr. Carries `ev` for thread-shifting. Heap-allocated, released after spawn. | +| `prte_odls_launch_local_t` | `base.h` | Per-node "start launching job J" caddy carried by `PRTE_ACTIVATE_LOCAL_LAUNCH`; holds `job`, `fork_local`, and a `retries` counter for the sys-limit backoff. | +| `prte_odls_pipe_err_msg_t` | `odls_types.h` | Fixed struct written up the child→parent pipe to proxy a `show_help` error (fatal flag + exit status + three string lengths). | +| `PRTE_DAEMON_*` command flags | `odls_types.h` | The daemon command byte that leads every RML control message to a prted (ADD_LOCAL_PROCS, KILL, SIGNAL, EXIT, …). | + +--- + +## Threading model + +- **Message build/parse, wireup, waitpid interpretation, kill/signal, and + state activation** all run on the **progress thread** (`prte_event_base`) + — the normal PRRTE event-driven model. +- **Only the fork/exec spawn step** may be off-loaded to the **odls + worker-thread pool** (`prte_odls_globals.ev_bases`) to parallelize + launching many procs. Each spawn is a self-contained caddy handed to one + worker base; it touches only its own child, so no shared-state locking is + needed on the hot path. +- `SIGCHLD` must stay unblocked (done at framework open); child death is + delivered through `src/runtime/prte_wait.c`, which fires the registered + `wait_local_proc` callback on `prte_event_base`. +- The two blocking base functions (`get_add_procs_data`, + `construct_child_list`) use a `prte_pmix_lock_t` to wait for the async + PMIx server callbacks, per the house caddy/lock pattern. + +--- + +## Gotchas when editing + +- **Pack/parse symmetry is sacred.** `get_add_procs_data` and + `construct_child_list` are a hand-matched serializer/parser pair. Any + change to the packed order/type in one **must** be mirrored in the other, + or daemons will mis-decode the launch message and the job hangs or + crashes. The source says so in capitals — heed it. +- **Locality is `parent == my vpid`.** A proc is "local" iff its `parent` + daemon vpid equals this daemon's rank. Getting the wireup wrong silently + launches procs on the wrong node or not at all. +- **`prte_local_children` is the single source of truth** on a daemon. + Adding/removing a child there, and its `PRTE_PROC_FLAG_*` flags + (`LOCAL`, `ALIVE`, `WAITPID`, `IOF_COMPLETE`, `REG`), gate the whole + lifecycle. A child is only fully released once **both** `WAITPID` and + `IOF_COMPLETE` are set. +- **Failure means activating a state, not returning.** On the daemon side + the launch is asynchronous; report errors with + `PRTE_ACTIVATE_PROC_STATE(FAILED_TO_LAUNCH/FAILED_TO_START)` or + `PRTE_ACTIVATE_JOB_STATE(NEVER_LAUNCHED)` so the HNP can react — don't + just bubble an `rc` up into the event loop. +- **The child cannot log normally.** Between `fork` and `execve`, error + reporting goes up the pipe as a rendered `show_help` string; never call + ordinary PRRTE logging there. +- **STOP_ON_EXEC pins the tracer thread.** Both the fork (in + `launch_local`/`restart_proc`) and the ptrace detach (in + `wait_local_proc`) must happen on `prte_event_base`; do not "optimize" + them onto a worker thread. +- **`chdir` bookkeeping.** `launch_local`/`restart_proc` bounce the daemon's + cwd per app and must always `chdir` back to `basedir` before returning. +- Standard PRRTE rules apply: `prte_config.h` first, braces on every block, + `NULL ==`/constant-on-left comparisons, `PRTE_ERROR_LOG` for unexpected + errors, no new compiler warnings. + +--- + +## Debugging + +```sh +prte --prtemca odls_base_verbose 5 ... # trace child-list build, dispatch, waitpid +prte --prtemca odls_base_verbose 10 ... # + sys-limit checks, per-thread dispatch +prte --prtemca odls_base_verbose 20 ... # >15 dumps the exact argv/env being exec'd +prte --prtemca state_base_verbose 5 ... # see LAUNCH_APPS / RUNNING transitions odls drives +prun --xterm 0,1 ... # route ranks 0,1 to xterm windows (see frame.c) +prun --report-bindings ... # print each child's applied binding (odls_base_bind.c) +``` + +Useful tuning params: `--prtemca odls_base_num_threads N` / +`odls_base_cutoff N` (spawn-thread pool sizing), +`--prtemca odls_base_exec_agent CMD` (wrap every exec), +`--prtemca odls_base_signal_direct_children_only 1` (don't signal the +child's whole process group). + +--- + +## Where to go next + +The launch primitive itself lives in the component: + +- [`pdefault/AGENTS.md`](pdefault/AGENTS.md) — the default (and only) + local-launch component: the real `fork()`/`execve()`, the parent/child + pipe protocol, and how it plugs `fork_local_proc` into the base. diff --git a/src/mca/odls/CLAUDE.md b/src/mca/odls/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/odls/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/odls/pdefault/AGENTS.md b/src/mca/odls/pdefault/AGENTS.md new file mode 100644 index 0000000000..33aa1d548f --- /dev/null +++ b/src/mca/odls/pdefault/AGENTS.md @@ -0,0 +1,187 @@ +# AGENTS.md — `odls/pdefault` (the fork/exec local launcher) + +Component guide for `src/mca/odls/pdefault/`. Read the +[framework guide](../AGENTS.md) first for the module contract, the launch +state machine, and the base helpers (`construct_child_list`, +`launch_local`, `spawn_proc`, `wait_local_proc`, `kill/signal/restart`, +`prte_odls_base_set`) that this component leans on for nearly everything. + +--- + +## Role and priority + +`pdefault` is the **default — and currently only — odls component**. It is +the code that performs the actual **`fork()` + `execve()`** of application +processes on a node using POSIX process primitives. Priority **10**, +deliberately low ("let others override us — we are the default", per +`component_query`), so a site could drop in a specialized launcher without +touching the base. + +It is only built where `fork` exists: `configure.m4` does +`AC_CHECK_FUNC([fork])` and skips the component otherwise. Because the base +select logic only engages odls inside a daemon, `pdefault` is what every +`prted` (and the HNP for its own local procs) uses to launch apps. + +Files: + +| File | Contents | +|------|----------| +| `odls_pdefault_component.c` | Component struct + `component_query` (returns priority 10 and the module). | +| `odls_pdefault_module.c` | The module: the five vtable fns, the `fork`/`execve` primitive (`fork_local_proc`), and the parent/child pipe protocol (`do_parent`/`do_child`). | +| `odls_pdefault.h` | Extern declarations for the component and module structs. | +| `configure.m4` | Gates the build on `fork` support. | +| `help-prte-odls-default.txt` | Rendered error text for binding/exec/iof failures. | + +--- + +## The module: mostly delegation + +`prte_odls_pdefault_module` wires four of its five entry points straight +to base helpers, passing in the component's own syscall primitives: + +```c +prte_odls_base_module_t prte_odls_pdefault_module = { + .get_add_procs_data = prte_odls_base_default_get_add_procs_data, // base, verbatim + .launch_local_procs = launch_local_procs, // → base + fork_local_proc + .kill_local_procs = kill_local_procs, // → base + odls_default_kill_local + .signal_local_procs = signal_local_procs, // → base + send_signal + .restart_proc = restart_proc, // → base + fork_local_proc +}; +``` + +- **`get_add_procs_data`** is the base function itself — the HNP-side + message builder has nothing OS-specific, so the component doesn't wrap it. +- **`launch_local_procs(data)`** calls + `prte_odls_base_default_construct_child_list()` to decode the message and + build the local child list, then fires + `PRTE_ACTIVATE_LOCAL_LAUNCH(job, fork_local_proc)` — handing the base's + per-node launch driver this component's `fork_local_proc` as the fork + primitive. +- **`kill_local_procs(procs)`** → `prte_odls_base_default_kill_local_procs(procs, + odls_default_kill_local)`. The base does the SIGCONT→SIGTERM→SIGKILL + escalation; the component supplies only the raw delivery. +- **`signal_local_procs(proc, signal)`** → + `prte_odls_base_default_signal_local_procs(proc, signal, send_signal)`. +- **`restart_proc(child)`** → + `prte_odls_base_default_restart_proc(child, fork_local_proc)`. + +So the component's *real* content is three primitives — +`fork_local_proc`, `odls_default_kill_local`, `send_signal` — plus the +child-side pre-exec sequence. + +--- + +## The fork/exec primitive — `fork_local_proc()` + +This is the function the base calls (as `cd->fork_local(cd)`) from inside +`prte_odls_base_spawn_proc`, once per child, on whichever event base the +base picked. The design, spelled out in the long header comment, is a +**pipe-synchronized fork**: + +1. Open a pipe `p[2]`. +2. `fork()`. Record `child->pid` (in *both* parent and child copies). +3. **Child** (`pid == 0`): close the read end, call `do_child(cd, p[1])` — + which never returns (it either `execve`s or `_exit`s). +4. **Parent**: close the write end, return `do_parent(cd, p[0])`. + +The pipe is the child→parent error channel: the child sets it +close-on-exec, so if `execve` succeeds the pipe simply **closes with no +data** and the parent reads EOF ⇒ success. If anything fails before exec, +the child writes a rendered `show_help` message up the pipe and the parent +prints it. + +`pipe()` or `fork()` failure sets `child->state = +PRTE_PROC_STATE_FAILED_TO_START` and returns `PMIX_ERR_SYS_LIMITS_PIPES` / +`PMIX_ERR_SYS_LIMITS_CHILDREN`. + +### `do_child()` — everything between fork and exec + +Runs in the forked child; `__prte_attribute_noreturn__`. In order: + +1. `setpgid(0,0)` — new process group so later signals reach grandchildren. +2. Make the pipe write-fd close-on-exec. +3. If this is a real child with output forwarding: `prte_iof_base_setup_child` + to hook up stdout/stderr, then **`prte_odls_base_set(cd, write_fd)`** — + the base binding routine (cpu + memory affinity from `child->cpuset`), + which proxies any binding error up the pipe. (If there is no child and + no output forwarding, stdio is tied to `/dev/null`.) +4. `pmix_close_open_file_descriptors()` — close everything except + stdio and the pipe. +5. Restore default signal handlers (`SIGTERM/INT/HUP/PIPE/CHLD`) and + unblock all signals — the event library may have left them altered, and + an app must not inherit a blocked SIGTERM. +6. `chdir(cd->wdir)` to the app's working directory. +7. If `PRTE_JOB_STOP_ON_EXEC`: `ptrace(PRTE_TRACEME, …)` so the app stops at + `execve` for a debugger to attach. +8. **`execve(cd->cmd, cd->argv, cd->env)`.** On return (always an error) it + distinguishes a bad interpreter (`ENOENT` but the file exists) from other + errno values and sends the `"execve error"` help up the pipe, then + `_exit`s. + +Errors here go through `send_error_show_help()` (fatal, exits) or +`send_warn_show_help()` (non-fatal, returns), which serialize a +`prte_odls_pipe_err_msg_t` header + file/topic/message strings via +`write_help_msg` — the format `do_parent` expects. + +### `do_parent()` — block until the child reports + +Runs on the event base. Closes the child ends of the IOF pipes, then loops +reading `prte_odls_pipe_err_msg_t` records: + +- **Pipe closed / read timeout** (`PMIX_ERR_TIMEOUT`) ⇒ child exec'd + successfully: set `child->state = RUNNING`, flag `ALIVE`, return + `PRTE_SUCCESS`. +- **A message arrives** ⇒ read the file/topic/msg strings, render with + `pmix_show_help_norender`. If `msg.fatal`, set `child->state = + FAILED_TO_START`, unset `ALIVE`, and return `PRTE_ERR_SILENT` (the string + was already shown). If it was only a warning, keep looping. +- **Read error** ⇒ set `child->state = UNDEF` and return a converted error. + +The `PRTE_ERR_SILENT` return propagates back through the base's +`spawn_proc`, which then activates `PRTE_PROC_STATE_FAILED_TO_START`. + +--- + +## The signal/kill primitives + +- **`odls_default_kill_local(pid, signum)`** — used by the base kill path. + When `HAVE_SETPGID`, it targets `-pgrp` (the process group's lead) so the + signal reaches any children the app spawned. `ESRCH` (already gone) is + treated as success. +- **`send_signal(pd, signal)`** — used by the base signal path. Honors the + `prte_odls_globals.signal_direct_children_only` MCA flag: if set, signals + only `pd`; otherwise signals the whole group (`-pd`). Maps `kill(2)` errno + to PRRTE codes (`ESRCH` ⇒ ignored, `EPERM` ⇒ `PRTE_ERR_PERM`, etc.). + +Both are static helpers passed as function pointers into the base — the +base owns the *policy* (which procs, what escalation), the component owns +the *mechanism* (the actual `kill`). + +--- + +## Things to watch when editing + +- **`do_child` is post-fork: async-signal-safety rules apply.** It runs in + a forked child that has not yet exec'd. Keep it to the existing minimal + syscall/`show_help`-over-pipe idiom; do not add malloc-heavy or + lock-taking PRRTE calls, and never log through the normal channels — the + pipe is the only safe way to report. +- **Preserve the pipe protocol on both ends.** `write_help_msg` (child) and + the read loop in `do_parent` must agree on the `prte_odls_pipe_err_msg_t` + layout and the file/topic/msg ordering; the base's `odls_base_bind.c` also + writes this same format, so the three must stay in lockstep. +- **`execve` is the point of no return.** Everything the app needs — cwd, + env (`cd->env`), argv, binding, closed fds, restored handlers — must be + in place *before* it. The base assembles env/argv/cmd in `spawn_proc`; the + child only finalizes cwd, binding, fds, and signals. +- **STOP_ON_EXEC threading.** The `PTRACE_TRACEME` here pairs with the + detach in the base's `wait_local_proc`; both must stay on + `prte_event_base` (the base forces this). Don't move the fork onto a + worker thread for a stop-on-exec job. +- **Don't reimplement base policy.** New behavior for *which* procs to + launch/kill/signal, retries, IOF wiring, waitpid interpretation, or state + transitions belongs in `base/odls_base_default_fns.c`, shared by any + future component. Keep `pdefault` to the OS primitives. +- **`_exit`, not `exit`, in the child.** The child uses `_exit`/ + `send_error_show_help` (which `_exit`s) so it never runs the parent's + atexit handlers or flushes shared buffers. diff --git a/src/mca/odls/pdefault/CLAUDE.md b/src/mca/odls/pdefault/CLAUDE.md new file mode 120000 index 0000000000..47dc3e3d86 --- /dev/null +++ b/src/mca/odls/pdefault/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/src/mca/plm/AGENTS.md b/src/mca/plm/AGENTS.md new file mode 100644 index 0000000000..ca7e1a8eea --- /dev/null +++ b/src/mca/plm/AGENTS.md @@ -0,0 +1,412 @@ +# AGENTS.md — The `plm` Framework (Process Launch Manager) + +Orientation for AI agents and human contributors working in +`src/mca/plm/`. This is a map, not the rulebook: the authoritative +project guidance lives in the top-level [`AGENTS.md`](../../../AGENTS.md) +and under [`docs/`](../../../docs/). When this file and those disagree, +**the docs win** — and please fix this file. + +--- + +## What this framework does + +`plm` (Process Launch Manager) answers one question: **how do we get a +`prted` daemon running on every node of the DVM?** It is the switchyard +for daemon launch. It runs on the HNP (DVM master): the HNP orchestrates +the launch, the daemons phone home, and the state machine advances. The +components differ only in the *mechanism* used to start the remote +daemons — `ssh` tree-spawn, `srun`, `lsb_launch`, `aprun` — and in +whether the launcher (SLURM/LSF/PALS) or PRRTE itself decides which +daemon lands on which node. + +`plm` sits in the DVM/job-launch state machine at `LAUNCH_DAEMONS`: + +``` +INIT → INIT_COMPLETE → ALLOCATE → ALLOCATION_COMPLETE → LAUNCH_DAEMONS + → DAEMONS_LAUNCHED → DAEMONS_REPORTED → VM_READY → MAP → MAP_COMPLETE + → SYSTEM_PREP → LAUNCH_APPS → SEND_LAUNCH_MSG → RUNNING → TERMINATED + ▲ ▲ + plm launches prteds here daemons phone home / apps launch +``` + +The `state` framework fires the component's `launch_daemons` handler +when the daemon job enters `PRTE_JOB_STATE_LAUNCH_DAEMONS`. That handler +calls `prte_plm_base_setup_virtual_machine()` to compute the daemon map +(which nodes need a new daemon, and what vpid each gets), then spawns +those daemons by whatever mechanism the component implements. Crucially, +**launch is asynchronous**: the handler returns after *starting* the +launch and sets the job to `DAEMONS_LAUNCHED`. The job does **not** +advance until every launched daemon calls back to the HNP +(`prte_plm_base_daemon_callback`), at which point the framework advances +to `DAEMONS_REPORTED` and on to `VM_READY`. Only then does mapping and +application launch proceed. + +Two very different flows share this framework: + +1. **DVM formation** — the daemon job (`PRTE_PROC_MY_NAME->nspace`) is + launched once to stand up the daemons. This is the launcher-heavy + path. +2. **Application jobs** — every later `prun`/`comm_spawn` job flows + through the same state machine, but its `LAUNCH_DAEMONS` step usually + finds `map->num_new_daemons == 0` ("no new daemons required") and + fast-forwards straight to `DAEMONS_REPORTED`. Application launch + itself is not a `plm` component's job — it is the `odls` on each + daemon, driven by the base `launch_apps`/`send_launch_msg` handlers. + +--- + +## Directory layout + +``` +plm/ + plm.h # the module/component vtable (function-pointer struct) + plm_types.h # **job / proc / node / app STATE codes** + PLM command codes (tree-wide!) + base/ + base.h # public base API (state-machine handlers, spawn_response, ...) + plm_private.h # framework-internal API + prte_plm_globals_t + proxy/prted-cmd helpers + plm_base_frame.c # framework open/close/register; the DEFAULT "local-only" module + plm_base_select.c # pick-ONE-component selection (highest priority wins) + plm_base_receive.c # the HNP command processor: tools/daemons → HNP (PRTE_RML_TAG_PLM) + plm_base_launch_support.c # the heart: state handlers, daemon callback/wireup, setup_vm, arg building + plm_base_prted_cmds.c # xcast-based terminate/kill/signal commands to daemons + plm_base_jobid.c # HNP nspace + per-job jobid assignment + help-plm-base.txt # user-facing error text + ssh/ # DEFAULT/fallback (pri 10): rsh/ssh tree-spawn — the reference impl + slurm/ # SLURM (pri 75): one srun launches all prteds + lsf/ # LSF (pri 75): lsb_launch() API call + pals/ # Cray PALS (pri 100, only built where PALS exists): aprun +``` + +Read `plm_types.h` first — it is consumed **tree-wide**. It carries the +authoritative `PRTE_JOB_STATE_*`, `PRTE_PROC_STATE_*`, +`PRTE_NODE_STATE_*`, and `PRTE_APP_STATE_*` numeric codes plus the +`prte_plm_cmd_flag_t` command codes. Then read +`plm_base_launch_support.c`, where control actually lives. + +### `plm_types.h` — the state codes (repo-critical) + +These `#define`d integers are hand-assigned and **every value must stay +unique within its family** (see the top-level AGENTS.md rule on status +codes). A few load-bearing boundaries and values: + +| Family | Boundary / key values | +|--------|-----------------------| +| Proc state | `UNTERMINATED = 15`; `RUNNING = 4`, `REGISTERED = 5`; `TERMINATED = 20`; `ERROR = 50` (error codes are offsets from it — `FAILED_TO_START = ERROR+3`, `COMM_FAILED = ERROR+6`, `ABORTED = ERROR+2`, …). Anything `< UNTERMINATED` means still-running. | +| Job state | `LAUNCH_DAEMONS = 8`, `DAEMONS_LAUNCHED = 9`, `DAEMONS_REPORTED = 10`, `VM_READY = 11`, `RUNNING = 14`; `UNTERMINATED = 30`, `TERMINATED = 31`; `ERROR = 50` (`FAILED_TO_START = ERROR+3`, `NEVER_LAUNCHED = ERROR+10`, `MAP_FAILED = ERROR+19`, …). | +| Node state | `UP = 3`, `DOWN = 2`, `DO_NOT_USE = 5`, `NOT_INCLUDED = 6`, `ADDED = 7`. | +| PLM commands | `LAUNCH_JOB_CMD = 1`, `UPDATE_PROC_STATE = 2`, `REGISTERED_CMD = 3`, `TOOL_ATTACHED_CMD = 4`, `READY_FOR_DEBUG_CMD = 5`, `LOCAL_LAUNCH_COMP_CMD = 6`. | + +Note the sequences deliberately **skip** some offsets (e.g. job error +`ERROR+15`) — do not reuse a gap assuming it is free. + +--- + +## The module contract + +Every component fills in the same vtable, declared in `plm.h` as +`prte_plm_base_module_t` (version macro +`PRTE_PLM_BASE_VERSION_2_0_0`). **All entries are mandatory** in +principle, but in practice most components reuse the base implementations +for everything except `spawn`, `init`, and `finalize`. The selected +module is copied wholesale into the global `prte_plm`. + +| Field | Signature | Meaning / return | +|-------|-----------|------------------| +| `init` | `int (*)(void)` | One-time setup: start the PLM recvs, register the component's `launch_daemons` handler on `PRTE_JOB_STATE_LAUNCH_DAEMONS`, set `daemon_nodes_assigned_at_launch`. Returns `PRTE_SUCCESS`/error. | +| `set_hnp_name` | `int (*)(void)` | Create the DVM's base nspace + HNP procID. **Every** component points this at `prte_plm_base_set_hnp_name`. | +| `spawn` | `int (*)(prte_job_t *jdata)` | **Non-blocking.** Kick a job into the launch state machine (`ACTIVATE_JOB_STATE INIT`, or `MAP` for a restart). The actual daemon launch happens later in the `LAUNCH_DAEMONS` handler, not here. Returns immediately. | +| `remote_spawn` | `int (*)(void)` | Called *on a daemon* to launch that daemon's own children — the tree-spawn fan-out. Only `ssh` implements it; everyone else leaves it `NULL`. | +| `terminate_job` | `int (*)(pmix_nspace_t)` | Kill all procs of a job. Base impl `prte_plm_base_prted_terminate_job` (xcast a kill-local-procs command). | +| `terminate_orteds` | `int (*)(void)` | Tear down the daemons themselves. Base impl `prte_plm_base_prted_exit` (xcast `PRTE_DAEMON_EXIT_CMD` / `HALT_VM`). SLURM/PALS wrap it to also reconcile their launcher-process bookkeeping. | +| `terminate_procs` | `int (*)(pmix_pointer_array_t *procs)` | Kill a specific proc set. Base impl `prte_plm_base_prted_kill_local_procs`. | +| `signal_job` | `int (*)(pmix_nspace_t, int32_t)` | Signal a job's procs. Base impl `prte_plm_base_prted_signal_local_procs`. | +| `finalize` | `int (*)(void)` | Stop recvs, free launcher state. | + +Unlike `rmaps` (whose module return codes are a per-job "is this mine?" +protocol), `plm` selects **one** module and it owns everything. The +"non-blocking spawn, wait for callback" contract is the thing to +internalize: never block the progress thread waiting for a daemon to +come up. + +--- + +## Component selection — pick ONE (unlike rmaps) + +`prte_plm_base_select()` (in `plm_base_select.c`) uses the standard +`pmix_mca_base_select()`: it queries every component, and the **single +highest-priority** component that returns a module wins. Its module is +copied into `prte_plm`. If *no* component selects (e.g. purely local +operation with no launcher), selection quietly leaves the **default +"local-only" module** defined in `plm_base_frame.c` in place and returns +success — an error is only raised later if someone actually tries to +launch daemons ("no-available-pls"). + +Selection is driven by the **resource-manager environment**, not by a +fixed priority ladder — each component's `query` inspects the +environment and either offers itself or bows out: + +| Component | Priority | Selected when… | +|-----------|----------|----------------| +| `pals` | 100 (MCA `plm_pals_priority`) | Always offers itself — **but only built where Cray PALS is detected** (`PRTE_CHECK_PALS`), so it is simply absent elsewhere. | +| `slurm` | 75 | `SLURM_JOBID` is set in the environment (and `srun --version` runs). | +| `lsf` | 75 | `LSB_JOBID` set, IBM CSM **not** enabled (`CSM_ALLOCATION_ID` unset), and `lsb_init()` succeeds. Only built when LSF headers/libs are found. | +| `ssh` | 10 (MCA `plm_ssh_priority`) | Always available once a launch agent (`ssh`/`rsh`, or `qrsh`/`llspawn`/`pbs_tmrsh`) is found in PATH. The **default/fallback**. | + +Because it is pick-one, in a SLURM allocation `slurm` (75) beats `ssh` +(10); on a bare cluster only `ssh` is present. `ssh` is deliberately the +lowest-priority, always-there catch-all — read it first, it is the +reference implementation. + +The `rsh` name is a registered **alias** for `ssh` (see +`mca_plm_base_register` in `plm_base_frame.c`): `--prtemca plm rsh` still +works and maps to the `ssh` component, and its MCA vars alias too. + +--- + +## What `base/` provides — walk the important pieces + +A component is mostly glue around the base. The base owns the entire +state machine, the daemon callback/wireup, the orted command-line +construction, and all the xcast-based termination. Understand these +before touching a component. + +### The state-machine handlers (`plm_base_launch_support.c`) + +The `state` framework calls these as a job walks the launch states. Each +is a libevent handler taking `(int fd, short args, void *cbdata)` where +`cbdata` is a `prte_state_caddy_t *` carrying the `jdata`. They form the +spine of launch: + +| Handler | State it services → what it does | +|---------|----------------------------------| +| `prte_plm_base_setup_job` | `INIT` → assign a jobid (`prte_plm_base_create_jobid`), arm spawn/job timeout timers, then `INIT_COMPLETE`. | +| `prte_plm_base_setup_job_complete` | `INIT_COMPLETE` → `ALLOCATE`. | +| `prte_plm_base_allocation_complete` | `ALLOCATION_COMPLETE` → `LAUNCH_DAEMONS` (the component's handler). Has a bootstrap-DVM special case that stands up the VM directly. | +| `prte_plm_base_daemons_launched` | `DAEMONS_LAUNCHED` → **deliberately a no-op**; we wait for daemons to phone home rather than advancing. | +| `prte_plm_base_daemons_reported` | `DAEMONS_REPORTED` → set node slots (unmanaged allocations), compute `total_slots_alloc`, then `VM_READY`. | +| `prte_plm_base_vm_ready` | `VM_READY` → check topology limits, preposition files (`filem`), then `MAP`. | +| `prte_plm_base_mapping_complete` | `MAP_COMPLETE` → `SYSTEM_PREP`. | +| `prte_plm_base_complete_setup` | `SYSTEM_PREP` → `LAUNCH_APPS`. | +| `prte_plm_base_launch_apps` | `LAUNCH_APPS` → pack the `PRTE_DAEMON_ADD_LOCAL_PROCS` (or `DVM_ADD_PROCS` for a fixed DVM) command plus the `odls` add-procs payload into `jdata->launch_msg`. | +| `prte_plm_base_send_launch_msg` | `SEND_LAUNCH_MSG` → xcast `jdata->launch_msg` to all daemons on `PRTE_RML_TAG_DAEMON`. This is what actually starts application procs. | +| `prte_plm_base_post_launch` | `RUNNING` → cancel spawn timer, wire up IOF, optionally dump the proctable, send the spawn response. | +| `prte_plm_base_registered` | `REGISTERED` → mark the job registered. | + +`prte_plm_base_spawn_response()` notifies the original spawn requestor +(tool via PMIx event, or another daemon via `PRTE_RML_TAG_LAUNCH_RESP`) +that the job launched. + +### The daemon callback / wireup — the "report back" + +This is the crux of the whole framework. After a component starts a +`prted`, that daemon connects back to the HNP and sends its identity on +`PRTE_RML_TAG_PRTED_CALLBACK`. The recv is +**`prte_plm_base_daemon_callback`**. For each daemon in the buffer it: + +1. Looks up the daemon's `prte_proc_t` by rank in the daemon job, sets + `daemon->state = PRTE_PROC_STATE_RUNNING`, sets `PRTE_PROC_FLAG_ALIVE`. +2. Unpacks and stores the daemon's **contact URI** (`daemon->rml_uri`, + stashed as `PMIX_PROC_URI`) — this is how the HNP learns to talk to + the daemon. +3. Unpacks the **node name** (+ aliases), reconciling it with the + allocation's name (the daemon's `gethostname` result wins; the + original becomes an alias). Sets `PRTE_NODE_FLAG_DAEMON_LAUNCHED` and + node state `UP`. +4. Unpacks the node **topology** (possibly compressed), de-duplicating + against `prte_node_topologies` and recording an hwloc diff when it + matches an existing one. Under `prte_homo_nodes` only daemon rank 1 + sends a topology and everyone else inherits it. +5. Bumps `jdatorted->num_reported`. When the count reaches + `num_procs`, `progress_daemons()` sets the daemon job to + `DAEMONS_REPORTED` and activates that state for every application job + parked in `DAEMONS_LAUNCHED` — releasing the whole DVM to proceed to + `VM_READY`. + +The failure counterpart is **`prte_plm_base_daemon_failed`** (recv on +`PRTE_RML_TAG_REPORT_REMOTE_LAUNCH`): a daemon (or a proxy launcher) +reports that a specific daemon vpid failed to start; it marks that proc +`FAILED_TO_START` and activates the proc-failure state. `ssh`'s +`ssh_wait_daemon` and the tree-spawn children send to this tag. + +Both recvs are registered by **`prte_plm_base_comm_start()`** +(`plm_base_receive.c`), which every component calls from `init`. On the +master it also registers the stack-trace recv; the base +`prte_plm_base_recv` on `PRTE_RML_TAG_PLM` is registered on all procs. + +### The command processor (`plm_base_receive.c`) + +`prte_plm_base_recv` is the HNP's inbound command handler for +tools/daemons on `PRTE_RML_TAG_PLM`. It runs inside an event (so it is +thread-safe on the progress thread) and switches on +`prte_plm_cmd_flag_t`: + +- **`PRTE_PLM_LAUNCH_JOB_CMD`** — the big one. Unpacks a `prte_job_t`, + records the originator, assigns a `schizo` personality + (`prte_schizo_base_detect_proxy`), resolves the target **session(s)** + (spawn-target list via `resolve_spawn_targets`, else session-id / + alloc-id / ref-id / parent session, with ownership checks), links the + child to its parent, processes `add-host`/`add-hostfile`, and finally + calls `prte_plm.spawn(jdata)` (or caches the job if the DVM isn't ready + yet). +- **`PRTE_PLM_UPDATE_PROC_STATE`** — daemons report per-proc pid/state/ + exit-code; the handler activates the corresponding proc state. +- **`PRTE_PLM_REGISTERED_CMD`** — procs registered for sync; advances to + `REGISTERED` when all report. +- **`PRTE_PLM_READY_FOR_DEBUG_CMD`** — debugger-stop bookkeeping. +- **`PRTE_PLM_LOCAL_LAUNCH_COMP_CMD`** — a daemon reports its local app + procs launched (pid + state); advances to `STARTED` on the first and + `RUNNING` when `num_launched == num_procs`. +- **`PRTE_PLM_TOOL_ATTACHED_CMD`** — register a connecting tool as a job. + +### orted command-line construction + +Every launcher has to build the argv that starts a remote `prted`. The +base provides the shared pieces: + +- **`prte_plm_base_setup_prted_cmd(&argc, &argv)`** — splits + `prte_launch_agent` (default `"prted"`) into argv and returns the + index of the `prted` word (so a wrapper like `valgrind ... prted` can + be handled). +- **`prte_plm_base_prted_append_basic_args(&argc, &argv, ess, + &proc_vpid_index)`** — appends the standard daemon options: debug + flags, `--prtemca ess `, `ess_base_nspace`, `ess_base_vpid +