[experiments] Daily Experiment Report — 2026-05-03

[github-actions[bot]]

🧪 Daily Experiment Report — 2026-05-03

2 experiments analysed across 2 workflows; 0 reached significance (p < 0.05). Both experiments are in early stages with insufficient data — EXTEND recommended for all.

---

caveman · smoke-copilot

Variants: yes vs no · Window: last 30 runs · Analysed: 16 runs with experiment steps
min_samples: 20 per variant · Owner: (not specified)

Hypothesis: (not specified)

⚠️ Note: All 16 experiment runs were assigned to the yes variant. This is because each smoke-copilot run targets a different PR branch, and GitHub Actions cache is branch-scoped — the experiment state (cumulative variant counts) cannot be shared across branches, causing every run to start fresh with counts {yes:0, no:0} and always select the first-listed variant (yes).

Experiment : caveman
Workflow   : smoke-copilot.lock.yml
Hypothesis : (not specified)
Owner      : (not specified)
Window     : last 30 runs  |  Analysed: 16 runs with experiments (14 non-skipped)
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
| yes (control)    |  16  |  81.2%   |    626.8       | [552.2 , 701.4]    |  (ref)    | 16/20 (80%)   |
| no               |   0  |   N/A    |     N/A        |       N/A          |    N/A    |  0/20  (0%)   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
Significance: * p<0.05   ** p<0.01   *** p<0.001
p-value is two-tailed, compared against the control (first) variant.

Guardrails: (none declared)

Recommendation: EXTEND
Rationale     : The `no` variant has 0 samples (needs 20); the experiment state
                is not persisting across branches — investigate cache key scoping.

Recommendation: EXTEND — The no variant has received 0 runs due to branch-scoped cache isolation. The experiment state must be refactored to use a persistent store (e.g., a repository variable or a dedicated cache key on the default branch) to achieve proper round-robin assignment across runs on different branches.

---

prompt_style · daily-community-attribution

Variants: concise vs verbose · Window: last 30 runs · Analysed: 2 runs with experiment steps
min_samples: 20 per variant · Owner: (not specified)

Hypothesis: (not specified)

Experiment : prompt_style
Workflow   : daily-community-attribution.lock.yml
Hypothesis : (not specified)
Owner      : (not specified)
Window     : last 30 runs  |  Analysed: 2 runs with experiments
min_samples: 20 per variant

+------------------+------+----------+----------------+--------------------+-----------+---------------+
| Variant          |  n   | Succ %   | Mean dur (s)   | 95% CI (s)         |  p-value  | min_samples   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
| concise (ctrl)   |   2  | 100.0%   |    580.5       |       N/A*         |  (ref)    |  2/20 (10%)   |
| verbose          |   0  |   N/A    |     N/A        |       N/A          |    N/A    |  0/20  (0%)   |
+------------------+------+----------+----------------+--------------------+-----------+---------------+
* CI not shown for n=2 due to extreme width (t-critical=12.706 at df=1)
Significance: * p<0.05   ** p<0.01   *** p<0.001
p-value is two-tailed, compared against the control (first) variant.

Guardrails: (none declared)

Recommendation: EXTEND
Rationale     : Both variants need 20 samples minimum; concise has 2, verbose has 0.

Recommendation: EXTEND — The experiment was added recently (May 2, 2026). Both runs went to concise because the cache was not found (fresh start). The verbose variant has 0 samples. Continue gathering data — approximately 40+ more runs needed (20 per variant).

---

📊 Summary

Experiment	Workflow	Control	Best variant	p-value	Guardrails	Recommendation
caveman	smoke-copilot	yes (n=16, 81.2%)	no (n=0)	N/A	(none)	EXTEND
prompt_style	daily-community-attribution	concise (n=2, 100%)	verbose (n=0)	N/A	(none)	EXTEND

Analysis window: last 30 runs per workflow · Significance threshold: p < 0.05 (two-tailed)
Run: 25274236598

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

• productionresultssa19.blob.core.windows.net
• productionresultssa8.blob.core.windows.net
• resultsreceiver.actions.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "productionresultssa19.blob.core.windows.net"
    - "productionresultssa8.blob.core.windows.net"
    - "resultsreceiver.actions.githubusercontent.com"

See Network Configuration for more information.

Generated by daily-experiment-report · ● 6.3M · ◷

• expires on May 6, 2026, 8:41 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by daily-experiment-report.

A newer discussion is available at Discussion #30107.