The question a p-value answers
In Topic 3.5 you set up a test: a parameter p, a null hypothesis H₀: p = p₀ and an alternative Hₐ. Now you have a sample result, such as p̂ = 0.26. That result is almost never exactly p₀. The question is whether the difference is too big to blame on chance.
A p-value answers this question:
If H₀ were true, how likely is it that random sampling alone would give a result as extreme as ours, or more extreme, in the direction that Hₐ suggests?
If that probability is very small, our result would be surprising under H₀, and the data give evidence for Hₐ. If it is not small, our result is the kind of thing that happens by chance when H₀ is true.
The null distribution
To find that probability, you need to know how the test statistic behaves when H₀ is true. Its distribution under that assumption is called the null distribution. There are two ways to get it.
- Simulation. Use technology to take many random samples of the same size from a population where p really equals p₀, and record the statistic for each one. You met this kind of simulation in Topic 2.3.
- A probability model. When the conditions from Topic 3.5 are met, the standardised test statistic z has approximately a standard normal distribution when H₀ is true. You will calculate z in Topic 3.7. In this topic the value of z is given to you.
Either way, the p-value is an area or a proportion in the null distribution, in the direction of Hₐ.
Which tail? The rule for each alternative
“More extreme” always means “further in the direction of Hₐ”.
| Alternative | p-value from a normal model (observed z) | p-value from a simulation (observed statistic) |
|---|---|---|
| Hₐ: p > p₀ | P(Z ≥ z), the area at or above z | proportion of simulated values at or above the observed value |
| Hₐ: p < p₀ | P(Z ≤ z), the area at or below z | proportion of simulated values at or below the observed value |
| Hₐ: p ≠ p₀ | P(Z ≤ −|z|) + P(Z ≥ |z|), both tails | proportion of simulated values at least as far from p₀ as the observed value, on either side |
For a two-sided test with a normal model, the two tails are equal, so the p-value is 2 × P(Z ≥ |z|).
For a two-sided test with a simulated p̂, measure the distance d = |p̂ − p₀|. Then count simulated values at or below p₀ − d and at or above p₀ + d. A simulated distribution is rarely perfectly symmetric, so count both tails rather than doubling one.
Interpreting a p-value in context
A good interpretation has three parts:
- The assumption: “Assuming the true proportion of [population] who [response] is [p₀]…”
- The probability: “…there is a [p-value] probability…”
- The event: “…of getting a sample proportion of [observed p̂] or [higher / lower / further from p₀] in a random sample of [n] [individuals], by chance alone.”
Leaving out part 1 is the most common error. Without it, the statement describes something else, and it is not the p-value.
What small and not-small p-values tell you
- A small p-value means the observed result would be unusual if H₀ were true. So it gives evidence for Hₐ. The smaller the p-value, the more convincing the evidence.
- A p-value that is not small means the result would not be unusual if H₀ were true. So the data do not give convincing evidence for Hₐ. This is not evidence that H₀ is true. The true proportion might still differ from p₀, and the sample was simply not able to show it.
A useful comparison is a court case. “Not guilty” means the evidence was not convincing; it does not prove the defendant innocent. A large p-value works the same way.
How small counts as “small” is set by a significance level, α, chosen before collecting data. Comparing the p-value with α to make a formal decision is the job of Topic 3.7.
What a p-value is not
- It is not the probability that H₀ is true.
- It is not the probability that Hₐ is true, and 1 − p-value is not that either.
- It is not the probability of making a mistake.
- It is not the size of the difference. A tiny difference from p₀ can have a small p-value if the sample is very large.
Worked example 1: a p-value from a simulation
Question. The fictional cinema chain Starlite says that 20% of its customers buy a large popcorn. A new manager suspects that the proportion is higher. In a random sample of 50 customers, 13 bought a large popcorn, so p̂ = 13/50 = 0.26. The hypotheses are H₀: p = 0.20 and Hₐ: p > 0.20, where p is the true proportion of all Starlite customers who buy a large popcorn.
The manager uses software to simulate 200 random samples of 50 customers from a population where p = 0.20, and records p̂ for each. Figure 1 shows the results.
(a) Use the simulation to estimate the p-value. (b) Interpret the p-value in context. (c) Do the data give convincing evidence that more than 20% of Starlite customers buy a large popcorn?
(a) Hₐ is “greater than”, so count the simulated values at or above 0.26:
13 + 12 + 7 + 3 + 1 + 1 + 1 = 38. The estimated p-value is 38 ÷ 200 = 0.19.
(b) Assuming the true proportion of Starlite customers who buy a large popcorn is 0.20, there is about a 0.19 probability of getting a sample proportion of 0.26 or higher in a random sample of 50 customers, by chance alone.
(c) No. A result like this would happen in about 19% of samples even if the true proportion were 0.20. That is not unusual, so the data do not give convincing evidence that the proportion is greater than 0.20. This does not show that the proportion is exactly 0.20; it shows only that this sample cannot rule it out.
Check. The simulated distribution is centred close to 0.20 (the mean of the 200 values is 0.2004), as it should be when H₀ is true. With a probability model, the exact chance of 13 or more buyers in 50 when p = 0.20 is about 0.186, close to the simulated 0.19.
Worked example 2: a p-value from the standard normal distribution
Question. Records at the fictional bakery chain Hearth & Crumb show that 30% of online orders include a gift note. After a website redesign, the manager wants to know whether this proportion has changed. In a random sample of 200 recent online orders, 45 include a gift note (p̂ = 0.225). The hypotheses are H₀: p = 0.30 and Hₐ: p ≠ 0.30. The conditions are met (np₀ = 60 and n(1 − p₀) = 140 are both at least 10, and the chain received far more than 2,000 recent online orders, so 200 ≤ 10% of them). The test statistic is z = −2.31.
(a) Find the p-value. (b) Interpret it in context. (c) What would the p-value be if the manager had suspected a decrease, Hₐ: p < 0.30? And for Hₐ: p > 0.30?
(a) Hₐ is two-sided, so add both tails beyond ±2.31.
- Lower tail: P(Z ≤ −2.31) = normalcdf(−99, −2.31, 0, 1) ≈ 0.0104.
- Upper tail: P(Z ≥ 2.31) ≈ 0.0104 (the curve is symmetric).
- p-value = 0.0104 + 0.0104 ≈ 0.021.
(b) Assuming the true proportion of Hearth & Crumb online orders that include a gift note is still 0.30, there is about a 0.021 probability of getting a sample proportion at least as far from 0.30 as 0.225 (in either direction) in a random sample of 200 orders, by chance alone.
This is a small probability. A result this far from 0.30 would be unusual if the proportion had not changed, so the data give evidence that the proportion of orders with a gift note has changed.
(c) For Hₐ: p < 0.30, use only the lower tail: p-value = P(Z ≤ −2.31) ≈ 0.0104. For Hₐ: p > 0.30, use the area at or above −2.31: p-value = P(Z ≥ −2.31) ≈ 0.9896. The sample proportion is below 0.30, so it gives no evidence at all that the proportion has increased.
Check. Whenever the observed statistic is on the opposite side of p₀ from Hₐ, the p-value is greater than 0.5. If you get a p-value near 1, check the direction of the tail before reporting it.
Common misconceptions
- “The p-value is the probability that H₀ is true.” The p-value assumes H₀ is true. It measures how surprising the data are under that assumption.
- Leaving out the assumption in an interpretation. Always start with “Assuming the true proportion … is p₀”.
- “A large p-value proves H₀.” It means only that the data are not convincing evidence for Hₐ.
- Using the wrong tail. For Hₐ: p < p₀, the p-value is the area at or below the observed z, not above it.
- Forgetting to add both tails for a two-sided alternative.
- Counting the wrong simulated values. Count values at least as extreme as the observed one, including the observed value itself.
- Thinking a small p-value means a big difference. It means the difference is hard to explain by chance; the size of the difference is a separate question.
- “1 − p-value is the probability that Hₐ is true.” No such probability comes from a test.
Where this leads
In Topic 3.7, Carrying Out a Test for a Population Proportion, you will calculate z yourself, compare the p-value with a significance level α, and write a full conclusion in context. Before that, try the practice questions, then use the revision notes and the checklist. To revise how a test is set up, see the Topic 3.5 study guide.
Quick questions
Is the p-value the probability that the null hypothesis is true?
No. The p-value is calculated by assuming H₀ is true. It is the probability of data like ours (or more extreme) if H₀ were true, not the probability that H₀ is true given our data.
How small does a p-value have to be?
That depends on the significance level, α, which is chosen before the test. Comparing the p-value with α to make a decision is the next step, in Topic 3.7. In this topic, focus on what the p-value means.