Self hosted runner jobs being cancelled after 24 hours when request to broker fails

[InaniMatt1]

Describe the bug

A few weeks ago our actions that run for longer than 24 hours on our self hosted runners suddenly started being cancelled,

The operation was canceled.

After getting the runner and worker logs after rerunning with debug logging it looks like after roughly 24 hours in the runner log a request to the broker fails with a not found error,

[2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound

Which prior to that all had been swimming along,

[2025-05-05 20:57:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:07:05 [2025-05-05 20:58:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:08:05 [2025-05-05 20:59:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:09:05 [2025-05-05 21:00:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:10:05 [2025-05-05 21:01:05Z INFO JobDispatcher] Successfully renew job request 89751, job is valid till 05/05/2025 21:11:05 [2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound [2025-05-05 21:01:08Z INFO JobDispatcher] Shutting down JobDispatcher. Make sure all WorkerDispatcher has finished. [2025-05-05 21:01:08Z INFO JobDispatcher] Ensure WorkerDispatcher for job ca395085-040a-526b-2ce8-bdc85f692774 run to finish, cancel any running job. [2025-05-05 21:01:08Z INFO JobDispatcher] Send job cancellation message to worker for job ca395085-040a-526b-2ce8-bdc85f692774. [2025-05-05 21:01:08Z INFO ProcessChannel] Sending message of length 0, with hash 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

Any help would be greatly appreciated!

To Reproduce

Run action for longer than 24 hours on self hosted runner

Expected behavior
Actions not cancelled abruptly after 24 hours

Runner Version and Platform

Latest 2.323 runner version and Ubuntu 24

What's not working?

[2025-05-05 21:01:08Z ERR GitHubActionsService] GET request to https://broker.actions.githubusercontent.com/message?sessionId=ddca1804-2eed-416f-aff7-caa4eb6a7623&status=Busy&runnerVersion=2.323.0&os=Linux&architecture=X64&disableUpdate=false failed. HTTP Status: NotFound

[enescakir]

GitHub Actions had a few incidents last week, and it could be related.

[enescakir]

This workaround also helped us with a few instances, especially for the repository that was recently enabled.
#3823 (comment)