The MPI_T events interface is inert (symbols present but no mca_base_event framework; no source/event ever registered)

[jsquyres]

The MPI_T events interface is inert (symbols present but no mca_base_event framework; no source/event ever registered)

Conformance status: PARTIAL | MPI-5.0 chapter: 15 - Tool Support | Parent: #13998 | Epic: #13987

Summary

Open MPI ships every C symbol of the MPI-4.0/5.0 MPI_T events facility (event sources, event types, event-registration handles, callbacks, dropped-event handling, and in-callback data accessors), but the facility is permanently inert: MPI_T_event_get_num and MPI_T_source_get_num always report zero, every index/handle/registration query returns an error, and there is no mca_base_event framework anywhere in the tree, no producer that registers an event, and no configure flag, MCA component, or library that can enable event delivery. Reporting an empty event space is itself standard-conformant, which is why this is PARTIAL rather than UNSUPPORTED, but the marquee events feature delivers nothing and cannot be turned on. This issue consolidates five §15.3.8 conformance rows (event sources, event types, callbacks, dropped events, reading event data) because all five share one root cause -- the missing event infrastructure -- and must be fixed as one body of work.

Why one issue for five §15.3.8 facets. The chapter analysis tracks the events sub-interface as five separate PARTIAL rows: (a) event sources, (b) event types + registration handles, (c) callbacks + safety-level dispatch, (d) dropped-event handling, and (e) in-callback data accessors. They are not five independent bugs: each row is inert for the same reason -- no mca_base_event framework exists and no producer ever registers a source or event type. Splitting them into five issues would create five tickets blocked on one prerequisite and invite partial/incoherent fixes. They are consolidated here and tracked as one feature with five facets.

What MPI-5.0 requires

MPI-5.0 §15.3.8 "Events" (printed pp.744-759) defines the events facility. The introduction (printed p.744, PDF p.796) states the core obligation:

"During the execution of an MPI application, the MPI implementation can raise events of a specific type to inform the user of a state change in the implementation. Event types describe specific state changes within the MPI implementation. ... The MPI implementation is said to raise an event when it invokes a callback function previously registered by the user for the corresponding event type."

Crucially, the standard explicitly permits an implementation to export zero event types. From "Event Type Query Functions" (printed p.748, PDF p.800):

"An MPI implementation exports a set of N event types through the MPI tool information interface. If N is zero, then the MPI implementation does not export any event types; otherwise, the provided event types are indexed from 0 to N - 1."

The same permission applies to event sources. MPI_T_SOURCE_GET_NUM (printed p.745, PDF p.797) "returns number of event sources (integer)" and "An MPI implementation is allowed to increase the number of sources during the execution of an MPI process." The standard mandates no minimum count of sources or events.

This is the key to the PARTIAL (not UNSUPPORTED) verdict: an implementation that registers zero sources and zero event types, and returns the appropriate error for every index/handle, is individually conformant for each entry point. The standard does not require any event to ever be delivered. So the gap is not "violates a MUST" -- it is "the headline facility is structurally incapable of ever being non-empty, with no path to enable it," which the chapter taxonomy classifies as PARTIAL (symbols present, empty-space behavior legal, but the feature is never actually provided and there is no enable path).

For completeness, the per-call error contracts the entry points must honor once non-empty are also specified here. MPI_T_SOURCE_GET_TIMESTAMP (printed p.746, PDF p.798):

"The call returns MPI_SUCCESS and a current timestamp in the argument timestamp if the source supports ad-hoc generation of timestamps. The call returns MPI_T_ERR_INVALID_INDEX if the index does not identify a valid source. The call returns MPI_T_ERR_NOT_SUPPORTED if the source does not support timestamps."

Callback safety levels are mandated by Table 15.5 "Hierarchy of safety requirement levels for event callback routines" (printed p.747, PDF p.799), defining MPI_T_CB_REQUIRE_NONE, MPI_T_CB_REQUIRE_MPI_RESTRICTED, MPI_T_CB_REQUIRE_THREAD_SAFE, and MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE; the implementation "provides the safety requirement as an argument to the callback when it is invoked."

What Open MPI currently does

Every events entry point is a hard-coded stub that reports an empty event space. All paths are confirmed against the live tree at 408ced4. (Each file unconditionally compiles into libmpi via ompi/mpi/tool/Makefile.am; there is no build flag gating them.)

(a) Event sources -- always zero, every index rejected. ompi/mpi/tool/source_get_num.c, function MPI_T_source_get_num, lines 26-38 as of 408ced4:

int MPI_T_source_get_num (int *num_source)
{
    if (!mpit_is_initialized ()) {
        return MPI_T_ERR_NOT_INITIALIZED;
    }
    if (MPI_PARAM_CHECK && NULL == num_source) {
        return MPI_ERR_ARG;
    }
    *num_source = 0;
    return MPI_SUCCESS;
}

This is the only assignment to num_source in the tree. Consequently ompi/mpi/tool/source_get_info.c (MPI_T_source_get_info, returns MPI_T_ERR_INVALID_INDEX at line 33) and ompi/mpi/tool/source_get_timestamp.c (MPI_T_source_get_timestamp, returns MPI_T_ERR_INVALID_INDEX at line 36) reject every index, so the ordering, ticks_per_second, and max_ticks outputs are never produced.

(b) Event types -- always zero, lookups always fail. ompi/mpi/tool/event_get_num.c, function MPI_T_event_get_num, lines 26-38 as of 408ced4:

int MPI_T_event_get_num (int *num_event)
{
    if (!mpit_is_initialized ()) {
        return MPI_T_ERR_NOT_INITIALIZED;
    }
    if (MPI_PARAM_CHECK && NULL == num_event) {
        return MPI_ERR_ARG;
    }
    *num_event = 0;
    return MPI_SUCCESS;
}

Therefore event_get_info.c (MPI_T_event_get_info) returns MPI_T_ERR_INVALID_INDEX (line 37); event_handle_alloc.c (MPI_T_event_handle_alloc) returns MPI_T_ERR_INVALID_INDEX (line 34); event_get_index.c (MPI_T_event_get_index) returns MPI_T_ERR_INVALID_NAME (line 36); and event_handle_free.c (MPI_T_event_handle_free) returns MPI_T_ERR_INVALID_HANDLE (line 35) -- there are no handles to free.

(c) Callbacks -- registration always fails; safety constants are dead. ompi/mpi/tool/event_register_callback.c, function MPI_T_event_register_callback, lines 27-36 as of 408ced4:

int MPI_T_event_register_callback (MPI_T_event_registration event_registration,
                                   MPI_T_cb_safety cb_safety, MPI_Info info, void *user_data,
                                   MPI_T_event_cb_function event_cb_function)
{
    if (!mpit_is_initialized ()) {
        return MPI_T_ERR_NOT_INITIALIZED;
    }
    return MPI_T_ERR_INVALID_HANDLE;
}

event_callback_get_info.c, event_callback_set_info.c, event_handle_get_info.c, and event_handle_set_info.c likewise return MPI_T_ERR_INVALID_HANDLE (line 35 / 36). The four safety-level constants are defined in ompi/include/mpi.h.in lines 949-952 as of 408ced4, but no dispatch ever reads them:

    MPI_T_CB_REQUIRE_NONE = OPAL_MCA_BASE_CB_REQUIRE_NONE,
    MPI_T_CB_REQUIRE_MPI_RESTRICTED = OPAL_MCA_BASE_CB_REQUIRE_MPI_RESTRICTED,
    MPI_T_CB_REQUIRE_THREAD_SAFE = OPAL_MCA_BASE_CB_REQUIRE_THREAD_SAFE,
    MPI_T_CB_REQUIRE_ASYNC_SIGNAL_SAFE = OPAL_MCA_BASE_CB_REQUIRE_ASYNC_SIGNAL_SAFE

(d) Dropped events -- no handler can be set. ompi/mpi/tool/event_set_dropped_handler.c, function MPI_T_event_set_dropped_handler, returns MPI_T_ERR_INVALID_HANDLE unconditionally after the init check (line 34 as of 408ced4).

(e) Reading event data -- every accessor fails. event_read.c (MPI_T_event_read, line 33), event_copy.c (MPI_T_event_copy, line 34), event_get_timestamp.c (MPI_T_event_get_timestamp, line 33), and event_get_source.c (MPI_T_event_get_source, line 33) all return MPI_T_ERR_INVALID_HANDLE -- there is no live MPI_T_event_instance for any of them to operate on.

Root cause -- the framework does not exist. A tree-wide search for mca_base_event (excluding 3rd-party / openpmix) finds exactly one hit, a comment in ompi/include/mpi.h.in line 942 ("Values are set in configure.ac for consistency with mca_base_event.h"). There is no opal/mca/base/mca_base_event.c or .h, no mca_base_event_t struct, no registration/enumeration/dispatch API, and no mca_base_event_raise() path. Contrast the fully built performance-variable framework: opal/mca/base/mca_base_pvar.c and mca_base_pvar.h both exist and back the working pvar entry points. No MCA component (PML, BTL, coll, runtime, ...) registers any event source or event type, and there is no configure switch that enables event delivery.

Why this is a gap

The standard's events model is "register a callback for an event type, and the implementation raises the event by invoking that callback" (§15.3.8 intro, p.744). Open MPI can never reach that model: zero sources and zero event types are reported, so no handle can be allocated, so no callback can be registered, so no event can be raised, so the in-callback accessors are never reachable. The safety-level constants the standard mandates (Table 15.5, p.747) are defined but consumed by nothing. The empty-event-space behavior is legal per p.748 ("If N is zero, then the MPI implementation does not export any event types"), but there is no mca_base_event framework and no enable path, so the facility is structurally incapable of ever being non-empty. The feature exists in name (symbols, headers, typedefs) but is permanently non-functional -- present-but-inert = PARTIAL.

Reproducers

C reproducer (mpicc)

This program documents the current inert state. As of 408ced4 it prints RESULT: INERT (the BEFORE / failing signal) because both counts are zero and there is no enable path. Once the events framework lands and at least one producer registers a source and an event type, a build that exposes events will print RESULT: HAS_EVENTS and exercise handle allocation + callback registration (the AFTER / passing signal).

Honesty notes: this is a singleton run (mpirun -np 1), runtime behavior, no special memory requirements; the entry points return error codes (they do not crash). The reproducer cannot, by itself, prove a specific event fires -- that requires a known producer and is left to the implementor's functional tests (see the note below the Fortran section). It only proves the facility is empty and that, when non-empty, the alloc/callback path is reachable.

/* mpicc mpit_events_inert.c -o mpit_events_inert && mpirun -np 1 ./mpit_events_inert */
#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

static volatile int g_callback_fired = 0;

static void event_cb(MPI_T_event_instance event, MPI_T_event_registration handle,
                     MPI_T_cb_safety cb_safety, void *user_data)
{
    (void) event; (void) handle; (void) cb_safety; (void) user_data;
    g_callback_fired = 1;
}

int main(void)
{
    int provided = 0, rc;
    rc = MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);
    if (MPI_SUCCESS != rc) { fprintf(stderr, "MPI_T_init_thread failed: %d\n", rc); return 1; }

    int num_sources = -1, num_events = -1;
    rc = MPI_T_source_get_num(&num_sources);
    if (MPI_SUCCESS != rc) { fprintf(stderr, "MPI_T_source_get_num failed: %d\n", rc); MPI_T_finalize(); return 1; }
    rc = MPI_T_event_get_num(&num_events);
    if (MPI_SUCCESS != rc) { fprintf(stderr, "MPI_T_event_get_num failed: %d\n", rc); MPI_T_finalize(); return 1; }

    printf("num_sources=%d num_events=%d\n", num_sources, num_events);

    if (0 == num_sources && 0 == num_events) {
        /* BEFORE (as of 408ced4faa): facility is empty and cannot be enabled. */
        printf("RESULT: INERT\n");
        MPI_T_finalize();
        return 77;  /* nonzero: the events facility delivers nothing */
    }

    /* AFTER: events exist. Exercise the alloc + callback-registration path on event 0. */
    char name[256], desc[256];
    int name_len = sizeof(name), desc_len = sizeof(desc), verbosity = 0, num_elements = 0;
    MPI_Datatype dts[64];
    MPI_Aint disps[64];
    MPI_T_enum et;
    MPI_Info info_used;
    int bind = 0;
    name_len = sizeof(name); desc_len = sizeof(desc); num_elements = (int)(sizeof(dts)/sizeof(dts[0]));
    rc = MPI_T_event_get_info(0, name, &name_len, &verbosity, dts, disps, &num_elements,
                              &et, &info_used, desc, &desc_len, &bind);
    if (MPI_SUCCESS != rc) { fprintf(stderr, "MPI_T_event_get_info(0) failed: %d\n", rc); MPI_T_finalize(); return 1; }

    MPI_T_event_registration reg;
    rc = MPI_T_event_handle_alloc(0, NULL, MPI_INFO_NULL, &reg);
    if (MPI_SUCCESS != rc) { fprintf(stderr, "MPI_T_event_handle_alloc(0) failed: %d\n", rc); MPI_T_finalize(); return 1; }

    rc = MPI_T_event_register_callback(reg, MPI_T_CB_REQUIRE_NONE, MPI_INFO_NULL, NULL, event_cb);
    if (MPI_SUCCESS != rc) { fprintf(stderr, "MPI_T_event_register_callback failed: %d\n", rc); MPI_T_finalize(); return 1; }

    (void) g_callback_fired;  /* implementor's functional test should now trigger the producer and assert this becomes 1 */

    MPI_T_event_handle_free(reg, NULL, NULL);
    printf("RESULT: HAS_EVENTS\n");
    MPI_T_finalize();
    return 0;
}

Fortran reproducer (use mpi_f08, mpifort)

Fortran f08: N/A -- MPI-5.0 defines no Fortran binding for the MPI_T interface. Chapter 15 §15.3 specifies C bindings only for every MPI_T procedure (the MPI_T_* routines, including all of §15.3.8 Events, have a "C binding" block and no Fortran or Fortran 2008 binding). The chapter analysis confirms this (for-claude/mpi-standard-5.0-apis.json: MPI_T procedures carry f08_expressible=false). The only Tool-Support routine with Fortran bindings is MPI_Pcontrol (§15.2), which is not part of this finding. A Fortran reproducer is therefore not applicable.

These reproducers are starting points. The implementor must add exhaustive singleton unit tests and/or mpirun/mpiexec-launched functional tests to fully prove the gap is closed -- in particular, a test that registers a callback against a real producer's event type, triggers the producer, and asserts the callback fires and that MPI_T_event_read / event_copy / event_get_timestamp / event_get_source return sensible data, plus a dropped-event overflow test.

Success criteria

Tie-out is against MPI-5.0 §15.3.8 (pp.744-759) and Table 15.5 (p.747). The gap is closed when, in a default build with at least one event producer enabled:

• An mca_base_event framework exists in opal/mca/base/ (header + implementation), mirroring mca_base_pvar, providing event-type and event-source registration, enumeration, handle management, callback dispatch, and a raise path (§15.3.8 intro, p.744).
• At least one producer registers an event source and an event type, so MPI_T_source_get_num and MPI_T_event_get_num can report a nonzero count (p.745 / p.748; "If N is zero, ..." -- nonzero is now reachable).
• MPI_T_source_get_info returns valid ordering (MPI_T_SOURCE_ORDERED / MPI_T_SOURCE_UNORDERED), ticks_per_second, and max_ticks for a valid source_index, and MPI_T_ERR_INVALID_INDEX for an out-of-range one (p.745-746).
• MPI_T_source_get_timestamp returns MPI_SUCCESS + a timestamp for a source that supports ad-hoc timestamps, MPI_T_ERR_INVALID_INDEX for a bad index, and MPI_T_ERR_NOT_SUPPORTED for a source that does not support timestamps (p.746).
• MPI_T_event_get_info / MPI_T_event_get_index resolve a registered event type's metadata; MPI_T_event_handle_alloc / MPI_T_event_handle_free create and destroy a registration on a valid event_index (p.748-750).
• MPI_T_event_register_callback stores the callback at the requested MPI_T_cb_safety level, and the implementation passes the actual safety level to the callback at invocation time, honoring the Table 15.5 hierarchy (p.747).
• MPI_T_event_set_dropped_handler registers a dropped-event handler that the framework invokes with the dropped count when events overflow (p.755).
• Inside a fired callback, MPI_T_event_read / MPI_T_event_copy / MPI_T_event_get_timestamp / MPI_T_event_get_source return the event's element data, timestamp, and source index (p.756-759).
• The C reproducer prints RESULT: HAS_EVENTS (exit 0) instead of RESULT: INERT (exit 77), and the implementor's functional test confirms the registered callback fires.
• The empty-event-space path remains conformant when no producer is enabled (zero sources/events, correct error codes), so no regression for tools that tolerate an empty space (p.748).

Where to fix (code pointers)

As of 408ced4:

• New framework: opal/mca/base/mca_base_event.c and opal/mca/base/mca_base_event.h (do not exist; model on opal/mca/base/mca_base_pvar.{c,h}). Add to opal/mca/base/Makefile.am; initialize/finalize alongside the pvar framework in OPAL's MCA-base lifecycle.
• At least one producer registering a source + event type (e.g. PML/BTL message events, coll events, or runtime/process-management events).
• Rewire the stubs in ompi/mpi/tool/:
  • sources: source_get_num.c, source_get_info.c, source_get_timestamp.c
  • types/handles: event_get_num.c, event_get_info.c, event_get_index.c, event_handle_alloc.c, event_handle_free.c
  • callbacks: event_register_callback.c, event_callback_get_info.c, event_callback_set_info.c, event_handle_get_info.c, event_handle_set_info.c
  • dropped: event_set_dropped_handler.c
  • read: event_read.c, event_copy.c, event_get_timestamp.c, event_get_source.c
• Header alignment: ompi/include/mpi.h.in already defines the typedefs (MPI_T_event_instance, MPI_T_event_registration, the three callback function typedefs at lines 968-979), the MPI_T_cb_safety enum (lines 948-953), and MPI_T_source_order (lines 958-963). The comment at line 942 references a not-yet-existing mca_base_event.h; that header must be created.

Recommended approach

This is a major, multi-component feature, not a quick fix -- build it incrementally.

1. Create the mca_base_event framework in opal/mca/base/, mirroring mca_base_pvar.{c,h}. Define mca_base_event_t (name, description, verbosity, element datatypes/displacements, bind type, enum, source), mca_base_event_registration_t (per-handle state plus a callback list keyed by mca_base_cb_safety), and mca_base_event_source_t (name, ordering, ticks_per_second, max_ticks, timestamp callback). Provide mca_base_event_register(), _get_count(), _get_by_index(), _get_by_name(), _source_register(), _source_get_*(), _handle_alloc()/free(), _register_callback() honoring OPAL_MCA_BASE_CB_REQUIRE_*, an mca_base_event_raise() dispatch, and dropped-event accounting feeding the dropped handler. Wire init/finalize into the OPAL MCA-base lifecycle and add the files to opal/mca/base/Makefile.am.
2. Register real events from at least one producer so the facility is non-empty.
3. Wire the MPI_T entry points to the framework facet by facet, replacing each hard-coded stub above.
4. Also update MPI_T_category_get_events does not validate cat_index (accepts out-of-range, returns MPI_SUCCESS) #14017 so MPI_T_category_get_events populates indices[] from each category's event list once events exist, and validates cat_index.
5. Tests per the success criteria.

Risk: event raising can sit on hot paths (PML/BTL), so the no-event case must stay zero-overhead, and the THREAD_SAFE / ASYNC_SIGNAL_SAFE safety levels constrain what the dispatch may do. Keep the empty-event-space fallback intact so tools that already tolerate zero events do not regress.

Cross-references

• MPI_T_pvar_readreset is an unimplemented stub (always returns MPI_T_ERR_INVALID_HANDLE) #14016 -- MPI_T_pvar_readreset is an unimplemented stub (always returns MPI_T_ERR_INVALID_HANDLE). Sibling §15.3.7 deviation in the same chapter.
• MPI_T_category_get_events does not validate cat_index (accepts out-of-range, returns MPI_SUCCESS) #14017 -- MPI_T_category_get_events does not validate cat_index. Directly downstream: once events exist, that routine must populate per-category event indices; coordinate the fixes.
• MPI_T_ERR_NOT_ACCESSIBLE / MPI_T_ERR_NOT_SUPPORTED constants are undefined in the public headers #14018 -- MPI_T_ERR_NOT_ACCESSIBLE / MPI_T_ERR_NOT_SUPPORTED are undefined in the public headers. Related: MPI_T_source_get_timestamp (p.746) must return MPI_T_ERR_NOT_SUPPORTED once sources exist, and that constant must first be defined.
• Framework to mirror: opal/mca/base/mca_base_pvar.{c,h}.
• Parent chapter issue: MPI-5.0 Chapter 15 - Tool Support: conformance gaps #13998; epic: [EPIC] MPI-5.0 conformance: close the remaining gaps #13987.
• In-flight ABI work PR Updated ABI generation code and new libraries #13280 (MPI-5.0 Chapter 20) adds no event infrastructure; this is not covered there.
• Dedup search (gh issue list --repo open-mpi/ompi --search "MPI_T events" / "MPI_T_event_get_num" / "MPI_T event interface mca_base_event" --state all): no existing issue tracks the inert events facility. (MPI_T_pvar_get_num() returns extraneous number of variables and causes MPI_T_pvar_get_info() to return with error later #11492 is a pvar-count bug, unrelated.)