Why a test can be wrong
A significance test makes a decision about a population from one random sample. Samples vary. Even when you do everything correctly, chance can give a sample that points the wrong way. So every decision carries a risk of error.
There are two possible decisions (reject H₀, or fail to reject H₀) and two possible truths (H₀ is true, or Hₐ is true). That gives four outcomes: two correct and two errors.
| H₀ is actually true | Hₐ is actually true | |
|---|---|---|
| Reject H₀ (convincing evidence for Hₐ) | Type I error (probability α) | Correct decision (probability = power) |
| Fail to reject H₀ (no convincing evidence for Hₐ) | Correct decision (probability 1 − α) | Type II error (probability = 1 − power) |
Read the table by columns. Each column is one possible truth, and the two probabilities in a column add to 1.
Type I and Type II errors
- A Type I error happens when the test finds convincing evidence for Hₐ (the p-value is small, so you reject H₀), but in fact H₀ is true. You “see an effect that is not there”.
- A Type II error happens when the test does not find convincing evidence for Hₐ (the p-value is not small, so you fail to reject H₀), but in fact Hₐ is true. You “miss an effect that is there”.
To describe an error in context, say what the test concluded and what is actually true, using the parameter and the population. For a test of H₀: p = 0.30 against Hₐ: p > 0.30, where p is the proportion of a town’s adults who use its swimming pool:
- Type I: the test finds convincing evidence that more than 30% of the town’s adults use the pool, when really the proportion is 30%.
- Type II: the test does not find convincing evidence that more than 30% use the pool, when really more than 30% do.
Two facts follow from the table:
- After you reject H₀, the only error you could have made is a Type I error.
- After you fail to reject H₀, the only error you could have made is a Type II error.
You never know for sure whether you made an error, because you never know the true value of p.
The probability of each error
Type I error. The significance level α is the probability of a Type I error. You choose it before you collect data, usually a small value such as 0.01, 0.05 or 0.10. If α = 0.05 and H₀ is true, the test will wrongly reject H₀ in about 5% of all possible samples.
Power. The power of a test is the probability that it correctly rejects H₀ when H₀ is false. Power depends on the true value of the parameter: a test has more power to detect a large departure from p₀ than a small one. So a power value is always “against” a particular alternative value, such as “power = 0.72 if p is really 0.68”.
Type II error. The probability of a Type II error is often written β:
P(Type II error) = β = 1 − power
A well-planned study aims for a small β and a large power, for example β ≤ 0.20, so power ≥ 0.80.
In this course you are given power values (from software or a table). You are not expected to calculate power from scratch. Figure 1 shows where the numbers come from, so you can see why the factors in the next section matter.
What affects the power
With everything else held the same, the power increases (and the probability of a Type II error decreases) when:
- The sample size n increases. Larger samples give less variable sample proportions, so a real difference is easier to detect.
- The standard error decreases. For a proportion, SE = √[p₀(1 − p₀) / n], so in practice this comes from a larger n. Narrower sampling distributions overlap less.
- The true parameter value is further from p₀. A big difference is easier to spot than a small one.
- The significance level α increases. A larger α makes it easier to reject H₀. The cost is a higher chance of a Type I error.
Point 4 is a trade-off. With a fixed sample size, lowering α to protect against a Type I error raises the chance of a Type II error, and the other way round. Increasing the sample size is the way to lower the Type II error probability without raising α.
Consequences decide the design
The two errors usually have different consequences. Think about them before you collect the data, because they guide two design choices:
- If a Type I error is more serious, choose a small α (such as 0.01). Because α is the Type I error probability, the consequences of a Type I error drive the choice of α.
- If a Type II error is more serious, choose a larger α (such as 0.10) and, above all, a larger sample. Because the sample size affects the Type II error probability, the consequences of a Type II error drive the choice of n.
Worked example 1: describing errors and their consequences
Context (fictional). The air-quality office of the fictional city of Corlan will limit traffic in the city centre if more than 10% of summer days have ozone above a health threshold. It selects a random sample of summer days from recent years and tests H₀: p = 0.10 against Hₐ: p > 0.10, where p is the true proportion of summer days with ozone above the threshold.
Question. (a) Describe a Type I error and a Type II error in context. (b) Give a consequence of each. (c) Which error is more serious? Use your answer to suggest α = 0.01 or α = 0.10.
(a)
- Type I error: the office finds convincing evidence that more than 10% of summer days have ozone above the threshold, when really the proportion is 10%.
- Type II error: the office does not find convincing evidence that more than 10% of summer days are above the threshold, when really the proportion is greater than 10%.
(b)
- Type I consequence: the city limits traffic when it is not needed. Businesses and commuters face disruption and cost for no health benefit.
- Type II consequence: the city does not limit traffic when it should. Residents, especially people with breathing problems, are exposed to more high-ozone days.
(c) A Type II error puts people’s health at risk, while a Type I error mainly costs money and convenience. Most people would judge the Type II error more serious. To reduce its probability, the office should use the larger significance level, α = 0.10, and collect as many days of data as it can.
Check. Each error statement says what the test concluded and what is actually true, and both refer to the parameter (the proportion of summer days) and the population (Corlan’s summer days). Another answer can earn credit if it is well argued, for example that unnecessary traffic limits would damage public trust. What matters is linking the more serious error to the choice of α.
Worked example 2: probabilities of errors and power
Context (fictional). Pinecrest Water says that 75% of its customers are satisfied with its service. A consumer watchdog suspects the true proportion is lower. It plans to survey a random sample of 200 customers and test H₀: p = 0.75 against Hₐ: p < 0.75 at α = 0.05. Its statistician uses software to find the power of the test if the true proportion is 0.68. She also finds the power after changing one feature of the plan at a time.
| Plan (one change at a time) | Power | P(Type II error) |
|---|---|---|
| Original: n = 200, α = 0.05, true p = 0.68 | 0.72 | 0.28 |
| n = 400 | 0.93 | 0.07 |
| α = 0.10 | 0.82 | 0.18 |
| α = 0.01 | 0.49 | 0.51 |
| True p = 0.65 | 0.93 | 0.07 |
Question. (a) For the original plan, what is the probability of a Type I error? (b) Find the probability of a Type II error if p = 0.68, and interpret the power. (c) Use the table to explain how each change affects the power. (d) The watchdog wants power of at least 0.80 against p = 0.68 without raising the chance of a Type I error. What should it do?
(a) P(Type I error) = α = 0.05. If 75% of customers really are satisfied, there is a 5% chance the watchdog wrongly finds convincing evidence that fewer than 75% are satisfied.
(b) P(Type II error) = 1 − 0.72 = 0.28. Interpretation of power: if the true proportion of satisfied Pinecrest customers is 0.68, there is a 0.72 probability that the test will find convincing evidence that the proportion is less than 0.75.
(c)
- Doubling the sample size raises the power from 0.72 to 0.93, because p̂ varies less, so the standard error is smaller.
- Raising α to 0.10 raises the power to 0.82, but the Type I error probability doubles to 0.10.
- Lowering α to 0.01 cuts the power to 0.49, so a Type II error becomes more likely than not.
- If the true proportion is 0.65 (further from 0.75), the power is 0.93. Bigger departures from p₀ are easier to detect.
(d) Increase the sample size. With n = 400 the power is 0.93 against p = 0.68, well above 0.80, and α stays at 0.05. Raising α would also give power above 0.80, but it increases the Type I error probability, which the watchdog wants to avoid.
Check. In every row, power + P(Type II error) = 1. Changing α changes the power in the same direction: larger α, larger power.
Common misconceptions
- “α is the probability that H₀ is true.” α is the probability of rejecting H₀ if H₀ is true.
- “Power is the probability that Hₐ is true.” Power is the probability of rejecting H₀ if a particular alternative value is true.
- “P(Type II error) = 1 − α.” 1 − α is the probability of a correct decision when H₀ is true. P(Type II) = 1 − power.
- Describing an error without context. “Rejecting H₀ when it is true” earns little. Say what is concluded and what is true, using the parameter and population.
- Describing the decision only. “The test concludes the proportion is greater than 0.10” is not an error until you add “when really it is 0.10”.
- “A smaller α is always better.” It lowers the Type I risk but raises the Type II risk unless n increases.
- “After failing to reject H₀ I might have made a Type I error.” A Type I error needs a rejection. Only a Type II error is possible here.
- “Larger samples reduce α.” α is chosen by the researcher. A larger n increases the power; it does not change α.
Where this leads
Next you will extend inference to two populations, starting with Topic 3.9, the sampling distribution for a difference between sample proportions. Type I and Type II errors and power apply to every significance test in the course, including chi-square tests and tests about means. To revise the decision step, look back at Topic 3.7. Try the practice questions now, then use the revision notes and the checklist to consolidate.
Quick questions
Can a test make both errors at once?
No. If you reject H₀, the only possible error is a Type I error. If you fail to reject H₀, the only possible error is a Type II error. In a real study you never know for certain whether an error was made, because you do not know the true parameter.
Why not just set α very small to avoid Type I errors?
With the sample size fixed, a smaller α makes it harder to reject H₀, so the power falls and the probability of a Type II error rises. Reducing both error probabilities at once needs more data.