Skip to content

Conversation

@pchmieli
Copy link
Contributor

@pchmieli pchmieli commented Nov 4, 2025

Q A
Bug fix? [x]
New feature? no
API breaks? no
Deprecations? no
Related tickets fixes #182
License Apache 2.0

What's in this PR?

When scaling the cluster out, added check for joining nodes before finishing action (ScaleOut=Done)

Why?

To avoid setting cluster Phase=Running precociously.
Details in issue #182

Checklist

  • Implementation tested

reduced number of unnecessary sleep during tests
@przysiadZeSztanga
Copy link

@cscetbon @AKamyshnikova could you please review?

@cscetbon
Copy link
Owner

/kuttl-tests

when scaling the cluster out, check joining nodes before finishing action

Fixes cscetbon#182
sort pods by name ascending when choosing last pod from the slice
@pchmieli
Copy link
Contributor Author

/kuttl-tests

@cscetbon Do you mean I should execute those tests or write new scenario? I see there is a scenario for scale-up already

@pchmieli
Copy link
Contributor Author

pchmieli commented Dec 2, 2025

/kuttl-tests

Hi @cscetbon, I ran scaling kuttl test on my branch and it passed, see details below.
Is it enough? I think extending this test to check details such as order of actions (all nodes are joint and only then action ScaleUp is finished) is quite cumbersome in kuttl. By the way, it is thoroughly tested via unit tests.

$ kuttl test --config ./test/kuttl/kuttl-test.yaml ./test/kuttl/ --test scaling

test logs
$ kuttl test --config ./test/kuttl/kuttl-test.yaml ./test/kuttl/ --test scaling
2025/12/02 13:05:19 kutt-test config testdirs is overridden with args: [ ./test/kuttl/ ]
=== RUN   kuttl
    harness.go:459: starting setup
    harness.go:257: running tests using configured kubeconfig.
    harness.go:280: Successful connection to cluster at: https://127.0.0.1:38055
    harness.go:365: running tests
    harness.go:77: going to run test suite with timeout of 1000 seconds for each step
    harness.go:377: testsuite: ./test/kuttl/ has 6 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/scaling
=== PAUSE kuttl/harness/scaling
=== CONT  kuttl/harness/scaling
    logger.go:42: 13:05:19 | scaling | Creating namespace "default"
    logger.go:42: 13:05:19 | scaling | Namespace "default" already exists
    logger.go:42: 13:05:19 | scaling/0-createCluster | starting test step 0-createCluster
    logger.go:42: 13:05:19 | scaling/0-createCluster | CassandraCluster:default/cassandra-e2e created
    logger.go:42: 13:07:31 | scaling/0-createCluster | test step completed 0-createCluster
    logger.go:42: 13:07:31 | scaling/1-scaleUp | starting test step 1-scaleUp
    logger.go:42: 13:07:31 | scaling/1-scaleUp | CassandraCluster:default/cassandra-e2e updated
    logger.go:42: 13:08:58 | scaling/1-scaleUp | test step completed 1-scaleUp
    logger.go:42: 13:08:58 | scaling/2-scaleDown | starting test step 2-scaleDown
    logger.go:42: 13:08:58 | scaling/2-scaleDown | CassandraCluster:default/cassandra-e2e updated
    logger.go:42: 13:12:31 | scaling/2-scaleDown | test step completed 2-scaleDown
    logger.go:42: 13:12:31 | scaling | scaling events from ns default:
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:22:20 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0           FailedScheduling        0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:22:20 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Claim data-cassandra-e2e-dc1-rack1-0 Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 success
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:22:20 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:22:20 +0100 CET    Normal  PodDisruptionBudget.policy cassandra-e2e                NoPods  No matching pods found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:22:20 +0100 CET    Warning PersistentVolumeClaim data-cassandra-e2e-dc1-rack1-0            ProvisioningFailed      storageclass.storage.k8s.io "local-path" not found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:25:15 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0           FailedScheduling        0/1 nodes are available: persistentvolumeclaim "data-cassandra-e2e-dc1-rack1-0" is being deleted. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:25:15 +0100 CET    Warning PersistentVolumeClaim data-cassandra-e2e-dc1-rack1-0            ProvisioningFailed      storageclass.storage.k8s.io "local-path" not found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:24 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-0 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:24 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:24 +0100 CET    Normal  PodDisruptionBudget.policy cassandra-e2e                NoPods  No matching pods found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:34 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:3.11.19" in 9.391367791s (9.391385432s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:34 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:34 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:35 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 11.502104908s (11.502113583s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:26:56 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:22 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 26.440985624s (26.44100588s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:22 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:22 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:24 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.471944647s (1.471952319s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:27:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Pulling Pulling image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:18 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Pulled  Successfully pulled image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" in 52.535772261s (52.535778935s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:18 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:18 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:36 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-1 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:36 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Pod cassandra-e2e-dc1-rack1-1 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:37 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:38 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:3.11.19" in 1.470181266s (1.470186457s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:38 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:38 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:39 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:40 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.517828267s (1.517834705s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:41 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:41 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:50 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 832.663635ms (832.670504ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:50 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:50 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:51 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:52 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.434725007s (1.434741338s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:52 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:28:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:02 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Killing Stopping container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:02 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Killing Stopping container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:02 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulDelete        delete Pod cassandra-e2e-dc1-rack1-1 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:05 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-1 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:05 +0100 CET    Warning StatefulSet.apps cassandra-e2e-dc1-rack1                RecreatingFailedPod     StatefulSet default/cassandra-e2e-dc1-rack1 is recreating failed Pod cassandra-e2e-dc1-rack1-1
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:05 +0100 CET    Warning StatefulSet.apps cassandra-e2e-dc1-rack1                FailedDelete    delete Pod cassandra-e2e-dc1-rack1-1 in StatefulSet cassandra-e2e-dc1-rack1 failed error: pods "cassandra-e2e-dc1-rack1-1" not found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:06 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:07 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:3.11.19" in 1.525162359s (1.525193328s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:07 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:07 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:08 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:10 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.377982977s (1.377989142s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:10 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:10 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:19 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:20 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 886.686456ms (886.712004ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:20 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:20 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:21 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.41305825s (1.413064525s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:30:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:26 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Killing Stopping container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:26 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Killing Stopping container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:26 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulDelete        delete Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:29 +0100 CET    Warning StatefulSet.apps cassandra-e2e-dc1-rack1                RecreatingFailedPod     StatefulSet default/cassandra-e2e-dc1-rack1 is recreating failed Pod cassandra-e2e-dc1-rack1-0
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:30 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-0 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:30 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:32 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:3.11.19" in 1.607143688s (1.607148874s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:32 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:32 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:32 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:34 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.576266875s (1.576274445s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:34 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:34 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:44 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:45 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 899.418434ms (899.425167ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:45 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:48 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.78298171s (1.782989038s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:48 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:48 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:48 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:48 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:31:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Killing Stopping container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Killing Stopping container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:56 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.11:4567/status": dial tcp 10.244.0.11:4567: connect: connection refused
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:56 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Unhealthy       Readiness probe failed:
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:57 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-1 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:57 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:59 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:3.11.19" in 1.404742655s (1.404757376s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:59 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:32:59 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:00 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:01 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.414967545s (1.414973445s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:01 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:01 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:10 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 970.875063ms (970.905539ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:12 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:13 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.470431456s (1.470436922s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:13 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:13 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:13 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:13 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:33:13 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:17 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Killing Stopping container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:17 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Killing Stopping container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:20 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.12:4567/status": dial tcp 10.244.0.12:4567: connect: connection refused
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:20 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Unhealthy       Readiness probe failed:
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:21 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-0 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:21 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:3.11.19" in 1.379174048s (1.379179084s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:23 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:24 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.414446918s (1.414452445s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:25 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:35 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:36 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 863.968992ms (863.975819ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:36 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:36 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:42 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulling Pulling image "cassandra:3.11.19"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:43 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.437620944s (1.437628879s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:43 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:44 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:44 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:44 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:34:44 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:35:56 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:3.11.19" in 1.467131623s (1.467138252s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:36:17 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                BackOff Back-off restarting failed container cassandra in pod cassandra-e2e-dc1-rack1-0_default(a2d92a86-7c10-4040-846d-41815722b443)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:37:54 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Killing Stopping container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:37:54 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Killing Stopping container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:37:57 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.13:4567/status": dial tcp 10.244.0.13:4567: connect: connection refused
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:37:57 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Unhealthy       Readiness probe failed:
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:48:31 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0           FailedScheduling        0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:48:31 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Claim gc-logs-cassandra-e2e-dc1-rack1-0 Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 success
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:48:31 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:48:31 +0100 CET    Normal  PodDisruptionBudget.policy cassandra-e2e                NoPods  No matching pods found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 12:48:31 +0100 CET    Warning PersistentVolumeClaim gc-logs-cassandra-e2e-dc1-rack1-0         ProvisioningFailed      storageclass.storage.k8s.io "local-path" not found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:11 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0           FailedScheduling        0/1 nodes are available: persistentvolumeclaim "data-cassandra-e2e-dc1-rack1-0" is being deleted. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:11 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Claim data-cassandra-e2e-dc1-rack1-0 Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 success
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:11 +0100 CET    Warning PersistentVolumeClaim data-cassandra-e2e-dc1-rack1-0            ProvisioningFailed      storageclass.storage.k8s.io "local-path" not found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:11 +0100 CET    Warning PersistentVolumeClaim gc-logs-cassandra-e2e-dc1-rack1-0         ProvisioningFailed      storageclass.storage.k8s.io "local-path" not found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:20 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-0 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:20 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:4.0.2"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:20 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Pod cassandra-e2e-dc1-rack1-0 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:20 +0100 CET    Normal  PodDisruptionBudget.policy cassandra-e2e                NoPods  No matching pods found
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:4.0.2" in 33.236576984s (33.236583351s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:53 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:54 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:56 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.533061367s (1.533078467s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:56 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:05:56 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:07 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:08 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 891.352054ms (891.373738ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:08 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:08 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:09 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulling Pulling image "cassandra:4.0.2"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:4.0.2" in 1.488255098s (1.488261604s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:11 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:06:30 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.15:4567/status": dial tcp 10.244.0.15:4567: connect: connection refused
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:25 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-0.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.15:4567/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:31 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1           Scheduled       Successfully assigned default/cassandra-e2e-dc1-rack1-1 to kind-control-plane
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:31 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulling Pulling image "cassandra:4.0.2"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:31 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulCreate        create Pod cassandra-e2e-dc1-rack1-1 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:33 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Pulled  Successfully pulled image "cassandra:4.0.2" in 1.424016135s (1.424022331s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:33 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Created Created container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:33 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{base-config-builder}          Started Started container base-config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:33 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulling Pulling image "datastax/cass-config-builder:1.0.4"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:35 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Pulled  Successfully pulled image "datastax/cass-config-builder:1.0.4" in 1.648066975s (1.64807264s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:35 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Created Created container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:35 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{config-builder}               Started Started container config-builder
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:45 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulling Pulling image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Pulled  Successfully pulled image "ghcr.io/cscetbon/casskop-bootstrap:0.1.21" in 887.929313ms (887.955124ms including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Created Created container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:46 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.initContainers{bootstrap}            Started Started container bootstrap
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:47 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulling Pulling image "cassandra:4.0.2"
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Pulled  Successfully pulled image "cassandra:4.0.2" in 1.599940434s (1.599946058s including waiting)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Created Created container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Started Started container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Pulled  Container image "ghcr.io/cscetbon/instaclustr-icarus:1.1.3" already present on machine
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Created Created container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:07:49 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Started Started container backrest-sidecar
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:08:01 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.16:4567/status": dial tcp 10.244.0.16:4567: connect: connection refused
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:08:46 +0100 CET    Warning Pod cassandra-e2e-dc1-rack1-1.spec.containers{backrest-sidecar}         Unhealthy       Readiness probe failed: Get "http://10.244.0.16:4567/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:12:09 +0100 CET    Normal  Pod cassandra-e2e-dc1-rack1-1.spec.containers{cassandra}                Killing Stopping container cassandra
    logger.go:42: 13:12:31 | scaling | 2025-12-02 13:12:09 +0100 CET    Normal  StatefulSet.apps cassandra-e2e-dc1-rack1                SuccessfulDelete        delete Pod cassandra-e2e-dc1-rack1-1 in StatefulSet cassandra-e2e-dc1-rack1 successful
    logger.go:42: 13:12:31 | scaling | Skipping deletion of pre-existing user supplied namespace default
=== NAME  kuttl
    harness.go:403: run tests finished
    harness.go:510: cleaning up
    harness.go:567: removing temp folder: ""
--- PASS: kuttl (431.59s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/scaling (431.56s)
PASS

@pchmieli
Copy link
Contributor Author

pchmieli commented Dec 4, 2025

Hi @cscetbon, thanks for approval! Can you merge it or we wait for other reviewers?

@cscetbon cscetbon merged commit 0e5a829 into cscetbon:master Dec 7, 2025
7 checks passed
@cscetbon
Copy link
Owner

cscetbon commented Dec 7, 2025

@pchmieli until we make a release you won't be able to use this unless you use master or issue182 as the image tag. When do you need this out ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cassandra scale-out workflow should wait for last replica being scaled-out

3 participants