e2e:serial: pull daemonset object in eventually to ensure updates

Sometimes it takes longer time to have the daemonset updates its pods
amounts. The problem with the old code is that in an Eventually it waits
for the daemonset pods number to get updated to the expected number
while using an old daemonset object. To fix this, just pull the
daemonset object in every iteration to ensure the code in the eventaully
is using the most updated data.

Signed-off-by: Shereen Haj <[email protected]>

Author: shajmakh

test/e2e/serial/tests/tolerations.go

@@ -469,10 +469,11 @@ var _ = Describe("[serial][disruptive][rtetols] numaresources RTE tolerations su
 
 				// NoExecute promises the pod will be evicted "immediately" but the system will still need nonzero time to notice
 				// and the pod will take nonzero time to terminate, so we need a Eventually block.
+				By(fmt.Sprintf("ensuring the RTE DS is running with less pods because taints (expected pods=%v)", len(workers)-1))
 				Eventually(func(g Gomega) {
+					Expect(fxt.Client.Get(ctx, dsKey.AsKey(), updatedDs)).To(Succeed())
 					pods, err = podlist.With(fxt.Client).ByDaemonset(ctx, *updatedDs)
 					Expect(err).ToNot(HaveOccurred(), "failed to get the daemonset pods %s: %v", dsKey.String(), err)
-					By(fmt.Sprintf("ensuring the RTE DS is running with less pods because taints (expected pods=%v)", len(workers)-1))
 					g.Expect(int(updatedDs.Status.NumberReady)).To(Equal(len(workers)-1), "updated DS ready=%v original worker nodes=%v", updatedDs.Status.NumberReady, len(workers)-1)
 					g.Expect(int(updatedDs.Status.NumberReady)).To(Equal(len(pods)), "updated DS ready=%v expected pods", updatedDs.Status.NumberReady, len(pods))
 				}).WithTimeout(5 * time.Minute).WithPolling(10 * time.Second).Should(Succeed())