Troubleshooting "Shutdown Signal" Error in EKS GitHub Actions Runner

[oliverpark999]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.11.0

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install the `gha-runner-scale-set-controller` Helm chart (version 0.11.0) in the `arc-systems` namespace.
2. Install the gha-runner-scale-set Helm chart (version 0.11.0) in the arc-runners namespace.
3. ArgoCD Deploy

Describe the bug

Build failures occur in the EKS gha Runner due to the error: "The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled."
Out of 10 builds, this error occurs 1-2 times.
The nodes use 'm5a.2xlarge' and 'm5a.xlarge' instances, which have high specifications with vCPU: 8 and MEM: 32Gi.
Additionally, the Runner's resources have been increased to:

  resources:
    requests:
      cpu: "2"
      memory: "12Gi"
    limits:
      cpu: "4"
      memory: "16Gi"

Despite these configurations, the error still occurs.
What could be the cause?

Describe the expected behavior

I expected all 10 GitHub Actions workflow runs to complete successfully without any runner shutdown errors. The runners should handle the jobs consistently, leveraging the high-spec EKS nodes (m5a.2xlarge, m5a.xlarge) and the allocated resources (requests: 2 CPU/12Gi, limits: 4 CPU/16Gi).

Additional Context

replicaCount: 1
resources:
  limits:
    cpu: "2"
    memory: "12Gi"
  requests:
    cpu: "4"
    memory: "16Gi"

Controller Logs

null

Runner Pod Logs

{"time":"2025-05-21T07:49:27.545989999Z","stream":"stdout","_p":"F","log":"Runner listener exit with terminated error, stop the service, no retry needed.

[RUNNER 2025-05-21 07:49:27Z ERR Terminal] WRITE ERROR: Failed to create a session. The runner registration has been deleted from the server, please re-configure. Runner registrations are automatically deleted for runners that have not connected to the service recently.

[RUNNER 2025-05-27 09:33:15Z ERR BrokerServer] at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)

[shichengripple001]

any update on this? i am having the same issue for long time

[WyriHaximus]

Any change whatever you use for cluster autoscaling is consolidating nodes?

[shichengripple001]

no. i have safe to evict set to false

[matthenry87]

I just overcame this. Turns out it was happening on a specific set of nodes in my cluster. Eliminated those and issue went away.

[nikola-jokic]

Hey, I don't think this issue is related to ARC. The controller only reacts to the termination signals. It is responsible for creating runner registration and spinning up the runner. After that, the controller only reacts to the termination signals.

Since this issue doesn't seem to be related to ARC, I will close it here. Please inspect the kubelet log. If everything looks okay, please submit the issue to the runner repo.

Feel free to comment on this issue if you want to share the log or if you have any questions.

[oliverpark999]

I was able to resolve the issue on my own shortly after posting this question.
I don't believe it was related to the Resource QoS of the Runner Pod.
After running several tests, I became confident that the problem was with the Worker Node.
Please take a closer look at the Worker Node.