Several CIs timeout since 11/01

[wangzhen127]

Failing ones:

• https://testgrid.k8s.io/sig-node-node-problem-detector#ci-npd-e2e-kubernetes-gce-gci
• https://testgrid.k8s.io/sig-node-node-problem-detector#ci-npd-e2e-kubernetes-gce-gci-custom-flags
• https://testgrid.k8s.io/sig-node-node-problem-detector#ci-npd-e2e-kubernetes-gce-ubuntu
• https://testgrid.k8s.io/sig-node-node-problem-detector#ci-npd-e2e-kubernetes-gce-ubuntu-custom-flags

@hakman @SergeyKanzhelev Do you know if anything could cause this?

[ajaysundark]

The test container seem to be OOMKilled.

"name": "test",
					"state": {
						"terminated": {
							"exitCode": 127,
							"reason": "OOMKilled",
							"message": ", line 57, in check_env\n    subprocess.check_call(cmd, env=env)\n  File \"/usr/lib/python3.11/subprocess.py\", line 413, in check_call\n    raise CalledProcessError(retcode, cmd)\nsubprocess.CalledProcessError: Command '('kubetest', '--dump=/logs/artifacts', '--gcp-service-account=/etc/service-account/service-account.json', '--up', '--down', '--test', '--provider=gce', '--cluster=e2e-f0faaac475-7abb6', '--gcp-network=e2e-f0faaac475-7abb6', '--extract=ci/latest', '--gcp-node-image=gci', '--gcp-zone=us-central1-b', '--ginkgo-parallel=30', '--test_args=--ginkgo.focus=NodeProblemDetector', '--timeout=60m')' returned non-zero exit status 1.\n+ EXIT_VALUE=1\n+ set +o xtrace\n{\"component\":\"entrypoint\",\"file\":\"sigs.k8s.io/prow/pkg/entrypoint/run.go:169\",\"func\":\"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess\",\"level\":\"error\",\"msg\":\"Process did not finish before 2h0m0s timeout\",\"severity\":\"error\",\"time\":\"2025-11-23T17:21:21Z\"}\n{\"component\":\"entrypoint\",\"error\":\"os: process already finished\",\"file\":\"sigs.k8s.io/prow/pkg/entrypoint/run.go:255\",\"func\":\"sigs.k8s.io/prow/pkg/entrypoint.gracefullyTerminate\",\"level\":\"error\",\"msg\":\"Could not interrupt process after timeout\",\"severity\":\"error\",\"time\":\"2025-11-23T17:21:21Z\"}\n{\"component\":\"entrypoint\",\"file\":\"sigs.k8s.io/prow/pkg/entrypoint/run.go:267\",\"func\":\"sigs.k8s.io/prow/pkg/entrypoint.gracefullyTerminate\",\"level\":\"error\",\"msg\":\"Process did not exit before 15m0s grace period\",\"severity\":\"error\",\"time\":\"2025-11-23T17:36:21Z\"}\n{\"component\":\"entrypoint\",\"error\":\"os: process already finished\",\"file\":\"sigs.k8s.io/prow/pkg/entrypoint/run.go:269\",\"func\":\"sigs.k8s.io/prow/pkg/entrypoint.gracefullyTerminate\",\"level\":\"error\",\"msg\":\"Could not kill process after grace period\",\"severity\":\"error\",\"time\":\"2025-11-23T17:36:21Z\"}\n{\"component\":\"entrypoint\",\"error\":\"process timed out\",\"file\":\"sigs.k8s.io/prow/pkg/entrypoint/run.go:84\",\"func\":\"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun\",\"level\":\"error\",\"msg\":\"Error executing test process\",\"severity\":\"error\",\"time\":\"2025-11-23T17:36:21Z\"}\n",
							"startedAt": "2025-11-23T15:21:21Z",
							"finishedAt": "2025-11-23T17:36:21Z",
							"containerID": "containerd://a7c5033b5731df63b2b9f9398f9d9fe768be227f21812187ba8c247b81a1fdc4"
						}
					},
					"lastState": {},
					"ready": false,
					"restartCount": 0,
					"image": "gcr.io/k8s-staging-test-infra/kubekins-e2e:v20251029-79c2132152-master",
					"imageID": "gcr.io/k8s-staging-test-infra/kubekins-e2e@sha256:7aeb7efbace9154e9a7005f861bd6f1a4b92c1e4588ee5e955b6752d5725b5ed",
					"containerID": "containerd://a7c5033b5731df63b2b9f9398f9d9fe768be227f21812187ba8c247b81a1fdc4",
					"started": false,
					"allocatedResources": {
						"cpu": "4",
						"memory": "4Gi"
					},
					"resources": {
						"limits": {
							"cpu": "4",
							"memory": "4Gi"
						},
						"requests": {
							"cpu": "4",
							"memory": "4Gi"
						}
					},

[hakman]

Thanks for the memory tip @ajaysundark, should be fixed now via kubernetes/test-infra#36077.
/close

[k8s-ci-robot]

@hakman: Closing this issue.

In response to this:

Thanks for the memory tip @ajaysundark, should be fixed now via kubernetes/test-infra#36077.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.