Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: 'Handling pod eviction'
description: How to handle pod eviction in Kubernetes deployments.
sidebar_position: 4400
---

When running Dagster in Kubernetes environments, pod evictions can occur due to resource constraints or node scaling operations. To prevent runs from hanging indefinitely, you can implement the following solutions:

```
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
```

Configure job-specific timeouts in your Dagster configuration to ensure runs don't hang indefinitely:

```python
@job(
config={
"execution": {
"config": {
"max_concurrent": 1,
"timeout_seconds": 7200 # 2 hours
}
}
}
)
```

Pod evictions can occur due to:

- Node running out of memory
- Node scale-down operations
- Resource limit constraints

To mitigate these issues, consider requesting more resources during pod creation and implementing appropriate timeout configurations.

## Next steps

- Learn more about [run monitoring and timeout configurations](/deployment/execution/run-monitoring)