A lifecycle node cannot answer `get_state` while its own transition callback runs

[YuanYuYuan]

Part of #282 — instance of the re-entrancy defect class; fixed by #253. The class, the shared fix shape and the merge order are stated there.

What happens

Two distinct defects, both reachable from the documented API.

1. During a transition the node cannot answer questions about itself — for anybody

The mutex is held for the whole callback, and it is the same one every state query takes.

who asks	what happens
the callback itself, via get_current_state() or ~/get_state	hangs forever — blocks on a mutex its own thread holds, and std::sync::Mutex is not reentrant
an external lifecycle manager polling ~/get_state, ~/change_state or ~/get_available_transitions	blocks until the transition completes

The second row is the one that surprises people: the manager is in a different process and is doing nothing unusual.

2. While a transition runs, the state reported over the wire is wrong

The node reports Unconfigured rather than Configuring / Activating. A manager reading that is told the node reset itself — indistinguishable from the real thing.

Quieter than the hang, but it is why the first defect stayed invisible: an observer could not see the intermediate state even once the lock was released.

sequenceDiagram
    autonumber
    participant M as Manager<br/>(other process)
    participant T as Node thread
    participant SM as state_machine<br/>mutex
    participant CB as user on_configure

    T->>SM: lock()
    activate SM
    Note over SM: guard held for the<br/>WHOLE transition
    T->>CB: trigger(Configure, callback)
    activate CB
    CB->>SM: get_current_state() → lock()
    Note over CB,SM: ✗ same thread, non-reentrant<br/>blocks forever
    M->>T: ~/get_state
    Note over M,SM: ✗ handler needs the same<br/>mutex — manager stalls too
    deactivate CB
    deactivate SM

Loading

The activation bar on state_machine spanning the callback is the defect.

Root cause

crates/hiroz/src/lifecycle/node.rs — ZLifecycleNode::trigger_transition invoked the user callback from inside the guard:

self.state_machine.lock().unwrap().trigger(transition, callback)

StateMachine::trigger takes &mut self, so the callback necessarily ran under the state_machine mutex — the same mutex the ~/get_state, ~/change_state and ~/get_available_transitions handlers and get_current_state() all take. Non-reentrant, so re-entry from the callback cannot succeed.

Separately, crates/hiroz/src/lifecycle/client.rs's state_from_lc mapped every transition state onto Unconfigured, which is why the intermediate state stayed unobservable even once the lock was released.

Reproduction

// node side
let mut lc = ctx_a.create_lifecycle_node("worker").build()?;
// manager side, separate context, as a real lifecycle manager would be
let client = ZLifecycleClient::new(&mgr_node, "worker")?;

let seen = Arc::new(Mutex::new(None));
let (seen_c, client_c) = (seen.clone(), client.clone());
lc.on_configure = Box::new(move |_| {
    *seen_c.lock().unwrap() = Some(get_state_blocking(client_c.clone()));  // hangs
    CallbackReturn::Success
});

lc.configure()?;
assert_eq!(seen.lock().unwrap().take().unwrap()?, LifecycleState::Configuring);

Run it on a worker thread with a wall-clock deadline. On an unfixed build the inner query times out; on a fixed build it returns Configuring.

Evidence, both directions

Found by an audit of hiroz's locking, not by a user report, and then given the reproduction above.

Re-verified independently on 2026-08-05 against a rebase onto current main: with the branch as-is, 4 tests pass; with the three lifecycle/ production files reverted and the tests kept, all four fail. The failure path is the reproduction — the inner ~/get_state query does not return within its 8 s call timeout, and the harness converts that into a named panic rather than an unhelpful whole-scenario deadline.

The state_from_lc correction is load-bearing for the detector, not cosmetic: without it the intermediate state is unobservable over the wire, so a test cannot distinguish "answered correctly" from "answered Unconfigured".

Fixed by

#253.