P-values and probability

The probability of obtaining a p-value of 0.1 or lower is exactly 0.1 (when H0 holds).

p-value distns

Similarly, the probability of obtaining a p-value of 0.01 or lower is exactly 0.01, etc. (when H0 holds).

P-values are most likely to be near 0 if the alternative hypothesis holds

Example: Test for zero mean

Take about 50 samples. The cumulative proportions on the top right are close to a straight line between (0,0) and (1,1), so about 50% of p-values are less than 0.5, 20% are less than 0.2, etc. when H0 is true.

Use the slider to change the true population mean to 1.5 and repeat. Now more than 50% of p-values are less than 0.5, more than 20% are less than 0.2, etc.

Interpretation of low p-values

If the p-value for a test is 0.0023, we note that such a low p-value is unlikely if H0 is true -- it would only occur with probability 0.0023. We therefore conclude that the data give strong evidence that H0 does not hold.

A p-value of 0.0023 could arise when either H0 or HA holds. However it is unlikely when H0 is true and more likely when HA is true.

General guide

p-value Interpretation
over 0.1 no evidence that the null hypothesis does not hold
between 0.05 and 0.1 very weak evidence that the null hypothesis does not hold
between 0.01 and 0.05 moderately strong evidence that the null hypothesis does not hold
under 0.01 strong evidence that the null hypothesis does not hold