Skip to content

Commit a206242

Browse files
htekdevCopilot
htekdev
and
Copilot
authored
feat: add 3 new error entries (runner-environment x2, silent-failures x1) (#356)
* feat: add 3 new error entries (runner-environment x2, silent-failures x1) * fix: correct IDs to re-433 and re-434 (re-431/re-432 already used in round 211) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 4ea9ec3 commit a206242

3 files changed

Lines changed: 477 additions & 0 deletions

‎errors/runner-environment/checkout-sparse-checkout-cone-mode-root-files-excluded.yml‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
id: runner-environment-434
2+
title: 'actions/checkout sparse-checkout Cone Mode Does Not Include Root Files by Default'
3+
category: runner-environment
4+
severity: silent-failure
5+
tags:
6+
- checkout
7+
- sparse-checkout
8+
- cone-mode
9+
- monorepo
10+
- root-files
11+
- missing-files
12+
patterns:
13+
- regex: 'sparse-checkout-cone-mode:\s*true'
14+
flags: 'i'
15+
- regex: 'sparse-checkout:\s*true'
16+
flags: 'i'
17+
- regex: 'sparse.checkout.*cone'
18+
flags: 'i'
19+
error_messages:
20+
- "sparse-checkout cone mode: true — root files not found"
21+
- "Repository root files missing after checkout with sparse-checkout-cone-mode: true"
22+
- "File not found: package.json (or README.md, .gitignore, etc.) after sparse checkout"
23+
root_cause: |
24+
When using `actions/checkout` with `sparse-checkout: true` and
25+
`sparse-checkout-cone-mode: true`, root-level files (files located directly in the
26+
repository root, NOT inside any subdirectory) may not be included in the checkout
27+
unless they are explicitly specified in `sparse-checkout-patterns`.
28+
29+
Git's cone mode documentation states: "files in the root directory are always included."
30+
However, the `actions/checkout` action generates cone-mode patterns programmatically
31+
from `sparse-checkout-patterns`, and the generated patterns may be overly restrictive —
32+
effectively "locking out" root-level files when the pattern list only names subdirectories
33+
(`src/`, `libs/`, etc.) without also including a root wildcard.
34+
35+
**Example:** If `sparse-checkout-patterns` lists only `src/` and `packages/`, the
36+
generated cone-mode patterns may exclude root files such as:
37+
- `package.json`, `yarn.lock`, `pnpm-lock.yaml` — build tool manifests
38+
- `.gitignore`, `.gitattributes`, `.npmrc` — repository configuration
39+
- `README.md`, `LICENSE` — documentation
40+
- `Makefile`, `Dockerfile`, `docker-compose.yml` — build/deploy configs
41+
42+
This causes silent build failures downstream when tools (npm install, yarn, make, etc.)
43+
can't find their configuration files in the expected root location.
44+
45+
**Why it's a silent failure:** The checkout step succeeds without error. Files are simply
46+
absent from the working directory. Build tools then fail with "file not found" errors
47+
that appear unrelated to the sparse checkout configuration.
48+
49+
Source: actions/checkout#2366 (Jan 2026, 2 reactions) — self-hosted machine, cone mode
50+
actively contradicts documented Git behavior for root files.
51+
fix: |
52+
Add `/*` to your `sparse-checkout-patterns` list to explicitly include all root-level files.
53+
The `/*` pattern matches all files in the root directory (depth 1, non-recursive) while
54+
leaving subdirectory traversal controlled by your other patterns.
55+
56+
```yaml
57+
- uses: actions/checkout@v4
58+
with:
59+
sparse-checkout: |
60+
/*
61+
src/
62+
packages/
63+
sparse-checkout-cone-mode: true
64+
```
65+
66+
This ensures root files are always checked out, while still limiting deep checkout to
67+
only the subdirectories you specify.
68+
69+
**Alternative: use non-cone mode** if you need precise file selection without cone
70+
mode's subdirectory semantics:
71+
```yaml
72+
- uses: actions/checkout@v4
73+
with:
74+
sparse-checkout: |
75+
package.json
76+
yarn.lock
77+
src/
78+
sparse-checkout-cone-mode: false
79+
```
80+
Non-cone mode uses standard gitignore-style patterns rather than cone-mode's
81+
directory-boundary semantics.
82+
fix_code:
83+
- language: yaml
84+
label: 'Broken: cone mode silently excludes root files when only subdirs are listed'
85+
code: |
86+
- uses: actions/checkout@v4
87+
with:
88+
sparse-checkout: |
89+
src/
90+
packages/
91+
sparse-checkout-cone-mode: true
92+
# ❌ Root files (package.json, .gitignore, Makefile, etc.)
93+
# may be missing — cone mode generates patterns that exclude root
94+
# unless root files are explicitly requested
95+
96+
# Downstream failure:
97+
- run: npm install # Error: ENOENT: no such file or directory, open 'package.json'
98+
- language: yaml
99+
label: 'Fixed: add /* to explicitly include all root-level files'
100+
code: |
101+
- uses: actions/checkout@v4
102+
with:
103+
sparse-checkout: |
104+
/*
105+
src/
106+
packages/
107+
sparse-checkout-cone-mode: true
108+
# ✅ /* matches all files in the root directory (non-recursive)
109+
# src/ and packages/ are still the only subdirectories checked out
110+
111+
- run: npm install # Now works — package.json found at root
112+
- language: yaml
113+
label: 'Alternative: non-cone mode with explicit file list'
114+
code: |
115+
- uses: actions/checkout@v4
116+
with:
117+
sparse-checkout: |
118+
package.json
119+
yarn.lock
120+
.gitignore
121+
.npmrc
122+
src/
123+
packages/
124+
sparse-checkout-cone-mode: false
125+
# Non-cone mode uses gitignore-style patterns — more explicit
126+
# but each root file must be listed individually
127+
prevention:
128+
- 'Always add `/*` to `sparse-checkout-patterns` when using `sparse-checkout-cone-mode: true` to ensure root files are included.'
129+
- 'After setting up sparse checkout, verify that expected root files (package.json, Makefile, etc.) exist before running build steps.'
130+
- 'If root file inclusion is uncertain, add a validation step: `ls package.json yarn.lock || echo "WARNING: root files missing"` before builds.'
131+
- 'When in doubt, prefer `sparse-checkout-cone-mode: false` (non-cone mode) with explicit file patterns — behavior is more predictable.'
132+
- 'Test sparse-checkout configurations on a local clone before deploying to CI: `git sparse-checkout init --cone && git sparse-checkout set src/`'
133+
docs:
134+
- url: 'https://github.com/actions/checkout/issues/2366'
135+
label: 'actions/checkout#2366 — Sparse checkout cone mode does not include root files by default (Jan 2026)'
136+
- url: 'https://github.com/actions/checkout#sparse-checkout'
137+
label: 'actions/checkout README — sparse-checkout and sparse-checkout-cone-mode inputs'
138+
- url: 'https://git-scm.com/docs/git-sparse-checkout#_cone_mode_handling'
139+
label: 'Git docs — sparse-checkout cone mode: root directory handling'

‎errors/runner-environment/self-hosted-runner-worker-process-hangs-stuck-active.yml‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
id: runner-environment-433
2+
title: 'Self-Hosted Runner Worker Process Hangs — Runner Stuck in Active State, All Job Pickup Blocked'
3+
category: runner-environment
4+
severity: error
5+
tags:
6+
- self-hosted
7+
- runner
8+
- hanging-process
9+
- stuck-active
10+
- timeout
11+
- worker-process
12+
patterns:
13+
- regex: 'Waiting for a runner to pick up this job'
14+
flags: 'i'
15+
- regex: 'Runner.Worker.*running|worker.*process.*attached'
16+
flags: 'i'
17+
- regex: 'A session for this runner already exists'
18+
flags: 'i'
19+
error_messages:
20+
- "Waiting for a runner to pick up this job..."
21+
- "A session for this runner already exists."
22+
- "Runner.Worker process remains attached despite job completion"
23+
root_cause: |
24+
When a job spawns a subprocess that hangs (e.g., a test runner like `vitest --coverage`,
25+
a server process that doesn't exit, or a background task that outlives the job), the
26+
runner's `Runner.Worker` process stays alive waiting for the subprocess to complete.
27+
28+
While `Runner.Worker` is alive, the self-hosted runner remains in **active/busy** state
29+
in GitHub's UI. It does not transition back to idle and cannot accept new job assignments.
30+
This blocks all queued jobs across every repository assigned to that runner host — not just
31+
the current repository.
32+
33+
The root cause is that the runner process tree does not forcibly kill lingering subprocesses
34+
when the workflow finishes. The runner waits indefinitely for the worker process to exit
35+
naturally.
36+
37+
**How it manifests:**
38+
- Runner appears as "Active" in GitHub UI but no job is visibly running
39+
- New queued jobs sit at "Waiting for a runner to pick up this job..." for hours
40+
- `ps aux` or Task Manager shows a lingering child process (test runner, coverage tool, etc.)
41+
- Restarting the runner service resolves the issue immediately
42+
- If the runner restarts, it gets `TaskAgentSessionConflictException: A session for this
43+
runner already exists` — GitHub still thinks the previous session is active
44+
45+
**Conditions that trigger this:**
46+
- Long-running test coverage tools (Jest, Vitest, Istanbul) that don't auto-exit
47+
- Server processes started in CI without explicit `kill` steps
48+
- Node.js/Python subprocesses that keep the event loop open
49+
- OOM kill of the worker process (leaves runner in broken state)
50+
51+
Without explicit `timeout-minutes` on the job, the runner will block indefinitely.
52+
GitHub's default 6-hour job timeout applies to hosted runners, but self-hosted runners
53+
may not reliably enforce it in all failure modes.
54+
55+
Source: actions/runner#4312 (Mar 2026, 4 reactions) — multiple repos blocked for 3+ hours
56+
until the runner was manually restarted.
57+
fix: |
58+
**Immediate fix:** Restart the runner service on the affected host:
59+
```
60+
sudo systemctl restart actions.runner.<org>.<runner-name>.service
61+
```
62+
63+
**Permanent fix 1: Add `timeout-minutes` to all jobs that use self-hosted runners.**
64+
This ensures the runner kills the job (and its process tree) if it exceeds the limit:
65+
66+
```yaml
67+
jobs:
68+
test:
69+
runs-on: self-hosted
70+
timeout-minutes: 30 # forces job termination after 30 minutes
71+
steps:
72+
- run: npx vitest run --coverage
73+
```
74+
75+
**Permanent fix 2: Ensure subprocesses exit cleanly.**
76+
Background processes started in `run:` steps that must be killed before the job ends
77+
need an explicit cleanup step:
78+
79+
```yaml
80+
steps:
81+
- name: Start server
82+
run: ./server &
83+
shell: bash
84+
85+
- name: Run tests
86+
run: npx jest
87+
88+
- name: Stop server
89+
if: always()
90+
run: pkill -f ./server || true
91+
```
92+
93+
**Permanent fix 3: Use `cancel-in-progress` + monitor for hung runners.**
94+
Add health monitoring to detect runners stuck in active state with no active job session
95+
and trigger automatic restarts.
96+
fix_code:
97+
- language: yaml
98+
label: 'Add timeout-minutes to prevent runner hanging indefinitely'
99+
code: |
100+
jobs:
101+
test:
102+
runs-on: self-hosted
103+
timeout-minutes: 30 # kills the job if it runs longer than 30 minutes
104+
steps:
105+
- uses: actions/checkout@v4
106+
- name: Run tests with coverage
107+
run: npx vitest run --coverage
108+
# Without timeout-minutes, vitest coverage can hang indefinitely,
109+
# keeping the runner in active state and blocking all other jobs.
110+
- language: yaml
111+
label: 'Kill background processes in cleanup step'
112+
code: |
113+
jobs:
114+
integration-test:
115+
runs-on: self-hosted
116+
timeout-minutes: 20
117+
steps:
118+
- uses: actions/checkout@v4
119+
- name: Start application server
120+
run: ./server.sh &
121+
shell: bash
122+
- name: Wait for server to be ready
123+
run: npx wait-on http://localhost:3000
124+
- name: Run integration tests
125+
run: npx jest --testPathPattern=integration
126+
- name: Stop application server
127+
if: always() # runs even if tests fail
128+
run: pkill -f server.sh || true
129+
shell: bash
130+
prevention:
131+
- 'Always set `timeout-minutes` on jobs using self-hosted runners — the default 6-hour timeout may not be reliably enforced when worker processes hang.'
132+
- 'Add `if: always()` cleanup steps to kill any background processes your job starts — test servers, mock services, coverage daemons, etc.'
133+
- 'Monitor self-hosted runners for jobs stuck in "active" state with no visible job log activity — automate restarts for runners idle for >N minutes.'
134+
- 'Use `pkill -P $$` in a `post` composite action step to kill all child processes of the current shell when the job ends.'
135+
- 'If a runner frequently gets stuck, check for zombie subprocesses left by test coverage tools (Jest --coverage, Vitest, Istanbul) that keep the Node.js process alive.'
136+
docs:
137+
- url: 'https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes'
138+
label: 'GitHub Docs — jobs.<job_id>.timeout-minutes'
139+
- url: 'https://github.com/actions/runner/issues/4312'
140+
label: 'actions/runner#4312 — Self-hosted runner stuck in active state, blocking queued jobs (Mar 2026)'
141+
- url: 'https://github.com/actions/actions-runner-controller/pull/4399'
142+
label: 'actions-runner-controller#4399 — Attempted fix for stale EphemeralRunner registrations (Apr 2026)'

0 commit comments

Comments
 (0)