Implement Test Corrections

[kslote1]

Multiple Testing Corrections

The Multiple Testing Problem

In causal discovery, we typically test m hypotheses simultaneously:

$$ H_{0,k}: I(X_{j_k}^{(t-\tau_k)}; X_i^{(t)} \mid \mathbf{Z}_i^{(t)}) = 0, \quad k = 1, \ldots, m $$

Family-Wise Error Rate (FWER):

$$ \text{FWER} = P(\text{at least one false rejection}) = P\left(\bigcup_{k \in \mathcal{H}_0} {p_k \leq \alpha}\right) $$

False Discovery Rate (FDR):

$$ \text{FDR} = \mathbb{E}\left[\frac{V}{\max(R, 1)}\right] $$

where (V) is the number of false rejections and (R) is the total number of rejections.

---

Bonferroni Correction

Method: Reject (H_{0,k}) if (p_k \leq \alpha / m)

Properties:

• Controls FWER exactly: (\text{FWER} \leq \alpha)
• Conservative (low power) when (m) is large
• Appropriate when few true relationships exist

Application in oCSE:
Use when testing a small number of pre-specified relationships or
when strong FWER control is required.

---

False Discovery Rate Control

Benjamini–Hochberg Procedure:

1. Order p-values: (p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)})
2. Find largest (k) such that (p_{(k)} \leq \frac{k}{m}\alpha)
3. Reject (H_{0,(1)}, \ldots, H_{0,(k)})

Adaptive FDR:

Estimate the proportion of true nulls (\pi_0):

$$ \hat{\pi}_0 = \frac{\text{num}{p_i > \lambda}}{m(1 - \lambda)} $$

Then use threshold:

$$ p_{(k)} \leq \frac{k}{m \hat{\pi}_0}\alpha $$

By-Stage Methods:
Control FDR at each stage of forward/backward selection.