VPA InPlaceOrRecreate doc update with InPlacePodResize 1.34

[maxcao13]

I think some parts of our existing docs are not correct anymore with 1.34.

https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/features.md#in-place-updates-inplaceorrecreate

In particular, the part where it mentions that we will fall back to pod recreation if we do limit downscaling on a container with NotRequired (previously renamed to PreferNoRestart but got reverted) resize policy. I think previously this was in the KEP docs:

In the initial beta release of in-place resize, we will disallow `PreferNoRestart` memory limit decreases, enforced through API validation. The intent is for this restriction to be relaxed in the future, but the design of how limit decreases will be approached is still undecided.

I don't exactly remember how we handled this, but if the API validation triggered an API error, then the VPA probably returned that error and then indeed fell back to Recreate.

But now in 1.34, memory limit decrease is best-effort, but can trigger an OOMkill on the container. So there are now 3 scenarios with in place pod resize:

1. It is possible that nothing goes wrong with a memory limit decrease, and that the kubelet is okay with the change, and everything is works as expected. This is because the kubelet compares the container's current memory usage number to the the proposed memory limit decrease number. If the usage is lower, than we're good, and the resize can go through, and the VPA won't fall back to anything.
2. In the case where the kubelet sees that memory usage is above the decreased limit number, then the kubelet will simply reject the resize, and try again later. In this case, the VPA will not fallback to recreate, but if the resize continues to stay in progress (i.e., if the pod continues to have the condition PodResizeInProgress for over 1 hour, then the VPA will finally mark this resize as failed and fallback to Recreate mode).
3. The last case is where a race condition may happen and that the kubelet says we're good to resize, but in the time in between the kubelet signaling all is good and the time the resize actually goes through, it's possible the memory could've changed such that an OOM kill will happen.

This last case, I'm not entirely sure about because I'm not sure if the OOM kill restarts the container with the correct resources or not. If so, then we're okay.

If everything I'm assuming is correct, we should remove these references to falling back to pod recreation on memory limit decrease, but maybe I'm missing something.

[maxcao13]

/area vertical-pod-autoscaler

[maxcao13]

/kind documentation

[maxcao13]

This might be a good time to introduce configurable InPlacePodResize + InPlacePodDeferred timeouts, instead of the defaults. Because container memory decrease resizes with NotRequired policy may be stuck in limbo for an hour, instead of getting just getting rejected like before.

[omerap12]

Yeah, I was just thinking we should update our docs as well!

If everything I'm assuming is correct, we should remove these references to falling back to pod recreation on memory limit decrease, but maybe I'm missing something.

That’s how I read the section too - and please correct me if I'm wrong. Since memory limit changes are best effort, the KEP states: “If usage is greater than the desired limit, the resize will be skipped for that container. The PodResizeInProgress condition will remain, with an Error reason, and a message reporting the current usage & desired limit.” We already have our own timeout mechanism for this (InPlacePodResize).

Additionally, the KEP notes: “Setting the memory limit below current memory usage can cause problems… With cgroups v1 the change will be rejected, whereas with cgroups v2 it will trigger an OOM kill.” AFAIK, an OOM kill restarts the container, but the updated resources still wouldn't be applied in that case.

This might be a good opportunity to introduce configurable timeouts for InPlacePodResize and InPlacePodDeferred, instead of relying on the defaults.

Are we considering configuration at the VPA resource level, or a global configuration?

/triage accepted

[adrianmoisey]

This last case, I'm not entirely sure about because I'm not sure if the OOM kill restarts the container with the correct resources or not. If so, then we're okay.

That's an interesting question.
From a VPA perspective, it may receive the OOM and then respond by increasing the memory recommendation, so the VPA could help recover from that scenario.
I don't know what kubelet would do.

[maxcao13]

Apparently if a container gets OOMkilled, it will get restarted with the resized resources: https://kubernetes.slack.com/archives/C06FSK01BGU/p1762993153963959 so I believe VPA does not have to do anything special here. We probably just have to document that OOMkills may happen.

We probably should document both cgroups v1 and v2 behaviours, and/or point to InPlacePodResizing docs?

[omerap12]

Apparently if a container gets OOMkilled, it will get restarted with the resized resources: https://kubernetes.slack.com/archives/C06FSK01BGU/p1762993153963959 so I believe VPA does not have to do anything special here. We probably just have to document that OOMkills may happen.

We probably should document both cgroups v1 and v2 behaviours, and/or point to InPlacePodResizing docs?

Cool. thanks Max!
AFAIK cgroups v1 is being deprecated (kubernetes/enhancements#5573) so I think maybe just pointing to the InPlacePodResizing will be enough.