Hypothesis Testing

Is this result real, or just noise?

Hypothesis testing is the formal framework for deciding whether an observed effect is unlikely to be due to chance. It's the backbone of experiments, A/B tests, and scientific claims.

Null and alternative hypotheses

• Null hypothesis (H₀) — "no effect", "no difference", the default assumption. We assume it's true.
• Alternative hypothesis (H₁) — the claim we're testing for: there IS an effect.

H₀:  The new design does not change conversion.
H₁:  The new design changes conversion.

The p-value

The p-value is the probability of seeing a result at least as extreme as the one you observed, assuming the null hypothesis is true. A small p-value means your data would be surprising if nothing were going on.

A p-value does NOT tell you the probability that the hypothesis is true. It assumes the null is true and measures how weird your data is under that assumption.

Significance level (alpha)

You set a threshold α before the test. If p < α, you reject H₀. The conventional choice is α = 0.05 — a 5% chance of wrongly rejecting a true null.

Type I and Type II errors

• Type I (false positive) — rejecting H₀ when it's actually true. Probability = α.
• Type II (false negative) — failing to reject H₀ when H₁ is actually true. Probability = β.

Confidence intervals

A 95% confidence interval gives a range that, over many repeated samples, would contain the true value 95% of the time. Wider intervals mean more uncertainty; narrowing them requires more data.