O1StatisticsCore

The Language of Hypothesis Testing

Q: What does a 5% significance level actually mean?

That if H_0 were true, there is a 5% chance of obtaining a result extreme enough to reject it. It is the false-alarm rate you have chosen to tolerate.

A hypothesis test answers one question: is this result surprising enough to abandon what we assumed? The mathematics differs between tests, but the logic and the wording never do — and in this topic the wording carries as many marks as the calculation.

35 min Hypothesis Testing

The big picture

The structure is a proof by contradiction with probability in place of certainty. You assume the thing you doubt, work out how likely the observed data would be under that assumption, and reject it only if the data would be unreasonably unlikely. That is why a test can never $\textit{prove}$ the null hypothesis — failing to find evidence against something is not evidence for it. Getting that asymmetry into your conclusions is what separates a full-mark answer from a half-mark one, and it is the same reasoning that underpins every statistical claim you will meet afterwards.

What you'll be able to do

State null and alternative hypotheses correctly using population parameters
Distinguish one-tailed from two-tailed tests and choose between them
Explain significance level, critical region and critical value
Write conclusions that are appropriately tentative and in context

The two hypotheses

The $\textbf{null hypothesis}$ $H_{0}$ is the assumption being tested — usually "nothing has changed" or "the claim is correct". It always states a specific value for a population parameter, because you need a definite number to calculate with.

The $\textbf{alternative hypothesis}$ $H_{1}$ is what you suspect instead. It is the claim the test is looking for evidence of.

Both must be about a $\textbf{population parameter}$ — $p$ for a proportion, $\mu$ for a mean — never about the sample. The sample value is the evidence; the hypotheses are about the population it came from.

H_{0}: p = p_{0}

A specific value, so probabilities can be computed.

H_{1}: p \neq p_{0} \;\text{or}\; p > p_{0} \;\text{or}\; p < p_{0}

Two-tailed or one-tailed.

Example 1A manufacturer claims 30% of its sweets are red. A customer suspects there are fewer. State the hypotheses.

1The claim under test is the null:

p = 0.3

2"Fewer" points in one direction only.

Answer

H_{0}: p = 0.3

;

H_{1}: p < 0.3

Tip — Write $H_{0}: p = 0.3$ , not $H_{0}: \hat{p} = 0.3$ and not "the sweets are 30% red". Hypotheses are statements about population parameters in symbols.

One-tailed or two-tailed

A $\textbf{one-tailed}$ test looks for a change in a specified direction — an increase or a decrease. Use it when the question indicates direction: "has it increased?", "does he suspect fewer?".

A $\textbf{two-tailed}$ test looks for any change. Use it when the wording is neutral: "has the proportion changed?", "is the claim wrong?".

The choice matters because it determines how the significance level is used. In a two-tailed test at $5\%$ the probability is split between the two ends, giving $2.5\%$ in each — so a result must be more extreme to be significant than in a one-tailed test at the same level.

Example 1For each, decide the tail: (a) "has the mean changed?" (b) "has the pass rate improved?"

1(a) "Changed" gives no direction, so any difference counts.

2(b) "Improved" specifies an increase.

Answer(a) Two-tailed,

H_{1}: \mu \neq \mu_{0}

. (b) One-tailed,

H_{1}: p > p_{0}

Why this works

Decide the tail from the wording $\textit{before}$ seeing the data. Choosing a one-tailed test after noticing which way the sample went doubles your effective significance level, which is a real methodological error and not just an exam convention.

Significance level and critical region

The $\textbf{significance level}$ $\alpha$ is the probability of rejecting $H_{0}$ when it is actually true — the risk of a false alarm that you are willing to accept. Common values are $5\%$ , $1\%$ and $10\%$ .

The $\textbf{critical region}$ is the set of outcomes that would lead to rejecting $H_{0}$ . Its boundary is the $\textbf{critical value}$ . Because the distribution is discrete for a binomial test, the actual probability of the critical region is usually a little under $\alpha$ rather than exactly equal to it.

That actual probability is the $\textbf{actual significance level}$ , and questions ask for it specifically. It is the probability of landing in the critical region assuming $H_{0}$ is true.

\alpha = P(\text{reject } H_{0} \mid H_{0} \text{ true})

The false-alarm rate.

\text{two-tailed at } 5\% \Rightarrow 2.5\% \text{ each tail}

The level is shared between the tails.

Example 1For

X \sim B(20, 0.4)

testing

H_{1}: p < 0.4

5\%

, the critical region is found to be

X \leq 3

with

P(X \leq 3) = 0.0160

. State the actual significance level.

1The critical region is

X \leq 3

2Its probability under

H_{0}

0.0160

AnswerThe actual significance level is

1.60\%

, which is below the nominal

5\%

Tip — The actual significance level is the probability of the critical region, not the significance level you were given. Quoting $5\%$ when asked for it is a standard lost mark.

Writing the conclusion

A full conclusion has two parts. First the statistical decision — reject $H_{0}$ or do not reject it. Then the interpretation in the context of the question, phrased tentatively.

The tentativeness is not politeness. A test can be wrong: rejecting a true $H_{0}$ happens with probability $\alpha$ by design. So write "there is sufficient evidence at the $5\%$ level to suggest the proportion has decreased", not "the proportion has decreased".

Crucially, never write "accept $H_{0}$ ". Failing to find evidence against a hypothesis is not evidence for it — the sample may simply have been too small to detect a real effect. The correct phrasing is "do not reject $H_{0}$ " or "there is insufficient evidence to suggest a change".

Example 1A test of

H_{0}: p = 0.3

against

H_{1}: p < 0.3

5\%

gives a probability of

0.021

for the observed result. Write the conclusion.

1Compare:

0.021 < 0.05

, so the result falls in the critical region.

2Reject

H_{0}

3Interpret in context, tentatively.

AnswerSince

0.021 < 0.05

, reject

H_{0}

. There is sufficient evidence at the

5\%

level to suggest the proportion of red sweets is less than

0.3

Why this works

The comparison line — " $0.021 < 0.05$ " — is worth writing explicitly. It shows the marker exactly why the decision was made, and it is frequently a mark on its own.

Think like an examiner

State hypotheses in symbols about a population parameter, and define that parameter in words.
Decide the tail from the wording, and halve the significance level for a two-tailed test.
Show the comparison of probability against significance level explicitly.
Never write "accept $H_{0}$ "; write "do not reject $H_{0}$ " and phrase conclusions tentatively.

Common misconceptions

Writing "accept

H_{0}

Say "do not reject

H_{0}

". Absence of evidence is not evidence of absence.

Stating hypotheses about the sample.

They concern the population parameter. The sample provides the evidence.

Using the full

\alpha

in each tail of a two-tailed test.

Split it:

5\%

two-tailed means

2.5\%

in each tail.

Concluding "the proportion has decreased" outright.

Say there is sufficient evidence to suggest it, at the stated level. The test can be wrong.

Test structure

H_{0}: p = p_{0}

Specific value.

H_{1}: p \neq, >, < p_{0}

Two- or one-tailed.

\alpha = P(\text{reject } H_{0}\mid H_{0})

Significance level.

\text{actual } \alpha = P(\text{critical region})

Usually below the nominal level.

Stretch yourself

A researcher tests whether a coin is biased. They toss it 20 times, obtain 14 heads, and having seen the result decide to test $H_{1}: p > 0.5$ at the $5\%$ level. Identify the methodological error, and state what they should have done.

Hint — Consider when the direction of the alternative hypothesis was chosen.

Reveal solution

SolutionThe error is choosing a one-tailed test

\textit{after}

seeing which way the data fell. The question asked whether the coin is biased, which is directionless and calls for a two-tailed test with

H_{1}: p \neq 0.5

. By waiting to see 14 heads before deciding to test only for an excess, the researcher has effectively given themselves two chances at a one-tailed test — one in each direction — so the true probability of a false rejection is about

10\%

, not the

5\%

claimed. The correct procedure is to state hypotheses and the significance level before collecting or examining the data, using

H_{0}: p = 0.5

against

H_{1}: p \neq 0.5

with

2.5\%

in each tail.

\blacksquare

Questions students ask

Why can I not accept the null hypothesis?

Because failing to find evidence against something does not establish it. A small sample may simply lack the power to detect a real effect, so the honest statement is that there is insufficient evidence to reject

H_{0}

How do I know whether a test is one- or two-tailed?

From the wording. Directional language — increased, decreased, improved, fewer — indicates one tail. Neutral language — changed, differs, is the claim wrong — indicates two.

What does a 5% significance level actually mean?

That if

H_{0}

were true, there is a 5% chance of obtaining a result extreme enough to reject it. It is the false-alarm rate you have chosen to tolerate.

Why is the actual significance level usually less than the nominal one?

Because a discrete distribution cannot always produce a critical region of exactly 5%. You take the largest region not exceeding the level, so the actual probability falls a little short.

Key takeaways

$H_{0}$ states a specific parameter value; $H_{1}$ states what you suspect instead.
Hypotheses concern population parameters, never the sample.
Directional wording gives a one-tailed test; neutral wording gives two, with $\alpha$ split.
The critical region is the set of outcomes leading to rejection; its true probability is the actual significance level.
Conclusions are tentative and in context, and $H_{0}$ is never "accepted".