Skip to content

cuOptSetLogCallback registration is process-global; concurrent solves can drop or leak a callback #1752

Description

@ramakrishnap-nv

Follow-up to review feedback on #1636 (raised by CodeRabbit, endorsed by @mlubin: "this seems like a real issue").

Problem

cuOptSetLogCallback stores a callback per settings object, but it reaches the
logger through a single process-global slot in cpp/src/utilities/logger.cpp:

static log_callback_with_data_t g_pending_callback = nullptr;
static void* g_pending_callback_data               = nullptr;

Flow: cuOptSolve drops the settings' callback into these globals via
set_pending_log_callback(), and init_logger_t picks it up, copies it into the
logger guard, and installs a callback_sink_mt.

Two failure modes follow, both only with concurrent or overlapping solves.

1. Clobbering. The mutex makes each write atomic, but not the
drop-then-consume pair. Thread A calls set_pending_log_callback(cbA), thread B
calls set_pending_log_callback(cbB) before A's init_logger_t runs, and A
silently gets B's callback.

2. Silent drop, with a stale pointer left behind. init_logger_t returns
early when a guard is already alive:

auto existing_guard = g_active_guard.lock();
if (existing_guard) { guard_ = existing_guard; return; }

So if solve A is running, solve B's init_logger_t never reads the mailbox: B's
callback is never installed, and B's registration stays in the globals. A later
solve can then pick it up and invoke B's function pointer with B's user_data,
which may already have been freed by then.

The second is the more serious one — it is not "some log lines go missing" but a
stale callback plus stale user data surviving the solve that registered them.

Root cause

default_logger() is a process-wide singleton with one global sink list, so
there is no per-solve context for user_log_bridge to route on. Per-solve
callback delivery needs solve identity threaded through the logging path.

Options

  1. Fail loudly. If a solve begins while a guard is active with a different
    callback, return an error from cuOptSolve rather than silently dropping it.
    Small and honest; rejects a legitimate use case.
  2. Thread-local registration. The bridge looks up a thread_local callback,
    so each solving thread gets its own. Covers lines emitted on the solve thread;
    lines from internal worker threads would reach no callback.
  3. Per-solve logger context. Thread solve identity through the logging path
    so the bridge can route correctly. Correct, and the largest change.

Whichever is chosen, clearing the pending slot on the early-return path would
remove the stale-pointer hazard on its own, and is worth doing regardless.

Scope

Single-solve use — the common case — is correct today, which is why #1636 ships
with this documented as a known limitation rather than blocked on it.

Metadata

Metadata

Labels

awaiting responseThis expects a response from maintainer or contributor depending on who requested in last comment.bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions