Skip to content

Conversation

@UJESH2K
Copy link
Contributor

@UJESH2K UJESH2K commented Nov 12, 2025

Fixes: #5208

Issue: Issues With Logging When Running K6 Load Test on Litmus


🧩 What happened

When running K6 load tests using LitmusChaos, users observed the following message in ChaosCenter logs:

Even though the actual K6 test results existed, they were hidden because the backend only fetched logs from the main or runner pods.
However, in K6 experiments, the real results are stored in the helper pod (*-helper).


⚙️ Root Cause

K6 experiments follow a 4-level Kubernetes pod hierarchy:

  1. Argo Workflow Pod: experiment-name-timestamp-xxxxx
  2. Chaos Runner Pod: experiment-runner-xxxxx
  3. Job Pod: experiment-job-xxxxx
  4. Helper Pod: experiment-helper-xxxxx ← contains actual K6 load test logs

The backend code at
chaoscenter/subscriber/pkg/k8s/log.go
did not include fallback logic to check for the helper pod, leading to empty or failed log retrieval.


🧠 What this PR does

✅ Adds fallback logic to retrieve logs from the helper container when main logs are unavailable.
✅ Improves error handling to clearly report missing log sources.
✅ Ensures K6 experiment logs are now visible in ChaosCenter.
✅ Keeps existing functionality intact for other experiment types.

Updated backend logic snippet:

mainLog, err := k8s.GetLogs(podLog.PodName, podLog.PodNamespace, "main")
if err != nil {
    logrus.Warnf("main container log not found, retrying helper container for pod %v", podLog.PodName)
    helperLog, helperErr := k8s.GetLogs(podLog.PodName, podLog.PodNamespace, "")
    if helperErr != nil {
        logrus.Errorf("failed to get helper container logs for pod %v: %v", podLog.PodName, helperErr)
        logDetails.MainPod = "logs not found"
    } else {
        helperLog = strconv.Quote(strings.Replace(helperLog, `"`, `'`, -1))
        logDetails.MainPod = helperLog[1 : len(helperLog)-1]
    }
} else {
    mainLog = strconv.Quote(strings.Replace(mainLog, `"`, `'`, -1))
    logDetails.MainPod = mainLog[1 : len(mainLog)-1]
}

@UJESH2K UJESH2K force-pushed the fix/k6-helper-pod-logs-v2 branch from d0e0726 to 570d94b Compare November 12, 2025 06:06
@UJESH2K
Copy link
Contributor Author

UJESH2K commented Nov 12, 2025

@PriteshKiri I've resolved the error part from the original pr here.

@UJESH2K UJESH2K force-pushed the fix/k6-helper-pod-logs-v2 branch from 570d94b to 345438a Compare November 15, 2025 18:29
@UJESH2K UJESH2K force-pushed the fix/k6-helper-pod-logs-v2 branch from 345438a to f4d61cc Compare November 18, 2025 05:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issues With Logging When Running K6 Load Test on Litmus

2 participants