Chapter 3: Hypothesis Testing

Overview

Hypothesis testing helps us determine whether an observed effect is statistically significant or could have occurred by chance.

Null and Alternative Hypotheses

• Null Hypothesis \( H_0 \): No effect (e.g., \( \mu_D = \mu_C \))
• Alternative Hypothesis \( H_A \): Real effect (e.g., \( \mu_D > \mu_C \))

Test Statistic

The test statistic summarizes the observed data, often as a difference in sample means:

\[ \Delta\bar{X} = \bar{X}_D - \bar{X}_C \]

General notation:

\[ t = T(x_1, \dots, x_m) \]

Example: Mice Maze Experiment

6 mice: 3 control [18, 21, 22], 3 drug [30, 25, 20]

\[ \bar{X}_C = 20.33, \quad \bar{X}_D = 25, \quad \Delta \bar{X} = 4.67 \]

We ask whether this difference is due to the drug or due to chance.

Exact Null Distribution

There are \( \binom{6}{3} = 20 \) permutations. Calculate \( \Delta \bar{X} \) for each permutation. From table:

• 3/20 permutations have \( \Delta \bar{X} \geq 4.67 \)
• P-value: \( p = 0.15 \)

---

Approximate Null Distribution via Resampling

Use when \( n \) is large and exact enumeration is impractical.

1. Randomly split data into two groups
2. Compute \( t_i = \bar{x}_M^{(i)} - \bar{x}_F^{(i)} \)
3. Repeat for \( N \) trials to form \( \hat{T} = \{t_1, \dots, t_N\} \)
4. Estimate p-value: \( \tilde{p} = \frac{\text{# exceedences} + 1}{N + 1} \)

See derivations and formulae: ch3_permute-resample.html

Equivalence of Test Statistics

We can express \( \Delta\bar{X} \) in terms of \( \bar{x}_B \) and \( \bar{x}_{\text{null}} \):

\[ \Delta\bar{X} = \frac{n}{n_A}(\bar{x}_B - \bar{x}_{\text{null}}) \Rightarrow p = P(\bar{X}_B \geq \bar{x}_B) \]

Transformation Invariance (Theorem 3.1)

If \( T_1 = f(T_2) \) where \( f \) is strictly increasing, then p-values are identical.

Example: \( T_1 = \Delta\bar{X}, \quad T_2 = \bar{x}_B \)

Accuracy of Resampling

\[ \text{Var}(\hat{p}) \approx \frac{\hat{p}(1 - \hat{p})}{N} \Rightarrow \hat{p} \approx 0.0019 \pm 0.0004 \]

Two-Sided Tests

For two-sided alternative:

• \( H_+: \mu_D > \mu_C \Rightarrow p = 3/20 \)
• \( H_-: \mu_D < \mu_C \Rightarrow p = 18/20 \)

Two-sided p-value: \( 2 \times \min(p_+, p_-) = 0.3 \)