Performance Design Discussion

[kusumachalasani]

Validation & Experiment Runs:
- RECCE: Using the default configuration, calculate the no.of warmups required by doing a long run. Look into each cycle data to check till how many warmup cycles are needed. Measure variance/CI range is accepted.
- Baseline
- Experiment Run

Few design points from the discussion:

1. Use the variance and CI data to check if the trial has to be re-run. Confirm on whether the warm-up cycles are enough for the combination of tunables used.
2. Variance range should also be part of warmup / measurement cycles parameters as variance would differ based on shared / dedicated n/w.
3. First Validation experiment to get the warmup cycles , variance range , CI.
4. Conduct an experiment to understand the variance of
   1. average values in 3 iterations. ( has only 3 data points)
   2. individual data points from measurement cycles in iterations. (has 9 data points)
   3. per iteration variance - to confirm is the warmup enough
5. Validation experiment to decide whether to continue or not the experiment for provided slo. ( if the variance/CI is high)
6. Switch turn ON/OFF of a validation in an experiment ( if a particular trial needs re-run depending on warmup cycles / variance /CI )
   • Recommended when we already have some data.