Skip to content

fix(os/gproc): restore default signal handling when shutdown listenin… - #4868

Open
jiang-bx wants to merge 1 commit into
gogf:masterfrom
jiang-bx:fix/gproc-restore-default-signal-handling
Open

fix(os/gproc): restore default signal handling when shutdown listenin…#4868
jiang-bx wants to merge 1 commit into
gogf:masterfrom
jiang-bx:fix/gproc-restore-default-signal-handling

Conversation

@jiang-bx

Copy link
Copy Markdown
Contributor

listen() is the only reader of signalChan, and it returns as soon as the shutdown
handlers of the first shutdown signal are done:

func listen() {
	defer close(waitChan)
	for {
		sig = <-signalChan            // the only reader of signalChan
		// ... run the handlers ...
		if _, ok := shutdownSignalMap[sig]; ok {
			wg.Wait()
			return                    // returns, but signal.Notify stays armed
		}
	}
}

signal.Notify is never stopped, so from that moment on every SIGINT, SIGTERM, SIGQUIT
and SIGABRT is delivered into a channel that nobody reads any more, and is silently
discarded.

If a shutdown handler blocks, or the process does not exit right after the handlers, the
process becomes immune to every signal but SIGKILL. And because SIGQUIT and SIGABRT are
shutdown signals as well, kill -QUIT can no longer dump the goroutine stacks, so the
hang cannot even be diagnosed.

Why this is a bug rather than a convention

The contract of the loop is "handle the shutdown signal, then stop listening". Stopping
the listening should hand the signals back to their default behavior; leaving them
captured by a channel with no reader is not a policy that any caller can observe, choose
or undo. It is simply an unreachable state.

The blast radius is wide, because nothing here is opt-in: ghttp.Run, ghttp.Wait and
g.Listen all end up in this loop, so it covers essentially every GoFrame server. The
gf CLI is affected too — gf run registers its own shutdown handler through
gproc.AddSigHandlerShutdown and then waits up to 30s for the child process in
app.End. So when a service is started with gf run, one Ctrl+C leaves the CLI and the
service deaf at the same time, and the terminal has to be abandoned.

Reproduction

go run . — a shutdown handler that blocks
package main

import (
	"fmt"
	"os"

	"github.com/gogf/gf/v2/os/gproc"
)

func main() {
	gproc.AddSigHandlerShutdown(func(sig os.Signal) {
		fmt.Printf("shutdown handler received %v, and it never returns\n", sig)
		select {}
	})
	fmt.Printf("pid %d: listening\n", os.Getpid())
	gproc.Listen()
}

Send SIGTERM three times, one second apart. (SIGTERM rather than SIGINT: a shell sets
SIGINT and SIGQUIT to SIG_IGN for background jobs, which would blur the result.)

Before — the process outlives every signal:

  SIGTERM #1 -> still running
  SIGTERM #2 -> still running
  SIGTERM #3 -> still running
  --> survived 3 x SIGTERM; only SIGKILL is left
pid 93368: listening
shutdown handler received terminated, and it never returns

After — the second one terminates it:

  SIGTERM #1 -> still running
  SIGTERM #2 -> process terminated
pid 93489: listening
shutdown handler received terminated, and it never returns

The change

listen() calls endSignalListening() when it takes a shutdown signal, before running
the handlers, so the default behavior is already back in place while they run.

notifySignals is guarded as well: once the listening has ended, a handler added
afterwards would re-arm signal.Notify on a channel that has no reader, and swallow
signals again. The flag is set under signalHandlerMu, the same lock notifySignals
runs under — otherwise a concurrent AddSigHandler could read the flag as false and call
signal.Notify right after signal.Stop, which only narrows the window instead of
closing it.

Compatibility

Once a shutdown signal has been handled, the listened signals go back to the behavior they
had before signal.Notify, so a second shutdown signal terminates the process instead of
being ignored. This is the conventional escape hatch — pressing Ctrl+C again to force quit,
or Kubernetes following SIGTERM with SIGKILL. (When a signal was already ignored at process
start, that is what gets restored instead: a shell sets SIGINT and SIGQUIT to SIG_IGN for
background jobs, for instance.)

signal.Stop covers every signal registered on the channel, custom ones included, such as
the SIGUSR1 ghttp uses for graceful reload. Those handlers could not run after the loop
returned anyway, since nothing reads the channel any more; the difference is that such a
signal now takes its default behavior rather than being silently discarded.

No code can be relying on shutdown being uninterruptible by design: that state is not
reachable deliberately, it only exists because the loop leaves signal.Notify armed after
its reader is gone, which also disables SIGQUIT stack dumps.

Test

Test_Signal feeds signalChan directly, so it never exercises signal.Notify and
cannot observe any of this. The two new tests raise real signals, in a child process —
which is also the only way to assert that a process does get terminated:

  • Test_Signal_SecondShutdownSignalTerminatesProcess — a blocking shutdown handler, and a
    second signal that has to terminate the process.
  • Test_Signal_HandlersAddedAfterListenEndedDoNotReArmNotify — the same, with the handler
    registering another handler before it blocks.

Reverting either half of the change fails the matching test, and Test_Signal keeps
passing with the change (it is //go:build linux, and was also run on darwin to confirm).

…g ends

`listen()` is the only reader of `signalChan`, and it returns as soon as the
shutdown handlers of the first shutdown signal are done. `signal.Notify` is
never stopped, so from that point on every SIGINT/SIGTERM/SIGQUIT/SIGABRT is
delivered to a channel nobody reads and is silently discarded.

If a shutdown handler blocks, or the process fails to exit right after them,
the process becomes immune to every signal but SIGKILL. Because SIGQUIT and
SIGABRT are shutdown signals too, `kill -QUIT` can no longer dump the goroutine
stacks either, so the hang cannot even be diagnosed.

This affects every server built on `ghttp.Run` / `ghttp.Wait` / `g.Listen`, and
the `gf run` CLI itself, which registers its shutdown handler the same way.

Restore the default behavior for the listened signals before running the
shutdown handlers. A second shutdown signal then terminates the process even
when a handler blocks, which is the conventional escape hatch.

`notifySignals` is guarded as well: once the listening has ended, re-arming
`signal.Notify` for a handler added afterwards would swallow signals again. The
flag is set under `signalHandlerMu`, the same lock `notifySignals` runs under,
so a concurrent `AddSigHandler` cannot re-arm right after `signal.Stop`.

The existing `Test_Signal` feeds `signalChan` directly and never exercises
`signal.Notify`, so it cannot observe any of this. The new tests raise real
signals in a child process, which is also the only way to assert that a process
does get terminated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant