Experimental Design: Study Guide (Statistics 1.13)

Why run an experiment?

In an observational study you record what happens without changing anything. In an experiment you impose treatments and then measure the result. If people who drink more water also sleep better, some other habit may explain both. If chance decides who drinks more water, and that group sleeps better, the water becomes the most believable explanation.

A quick recap of the vocabulary from Topic 1.10:

• Experimental units: the individuals that receive the treatments. When they are people, they are called subjects or participants.
• Explanatory variable (factor): the variable whose levels the researcher imposes.
• Treatments: the levels of the factor, or the combinations of levels when there is more than one factor.
• Response variable: the outcome measured on each unit after the treatment.

The four principles of a well-designed experiment

Every well-designed experiment includes all four of these. Learn the names: questions often ask you to spot which one is missing.

Principle	What it means	Why it matters
Comparison	At least two treatment groups, one of which may be a control group	Without a comparison you cannot tell whether a result is unusual
Random assignment	Treatments are given to units by a chance process	Makes the groups similar on everything except the treatment
Replication	More than one experimental unit gets each treatment	Lets you see the natural variation between units, so one odd unit cannot decide the result
Direct control	Keep known extraneous variables the same for every unit	Removes those sources of variation from the response

Direct control and a control group are different ideas. Direct control means fixing conditions, such as the same room, the same time of day or the same amount of water. A control group is one of the treatment groups.

Control groups, placebos and the placebo effect

A control group exists for comparison. It might receive no treatment, the current standard treatment, or a placebo: something that looks like the real treatment but has no active ingredient, such as a sugar pill.

People sometimes respond just because they believe they are being treated. The placebo effect measures this:

placebo effect = (mean response to the placebo) − (mean response to no treatment)

To estimate it, the experiment needs both a placebo group and a no-treatment group. Worked example 1 shows the calculation.

Blinding

If participants know their treatment, their expectations can change the response. If the researchers who measure the response know, they may judge it differently without meaning to. Blinding (also called masking) prevents this.

• Single-blind: one side does not know which treatment each participant receives. Either the participants do not know but the researchers who deal with them do, or the other way round.
• Double-blind: neither the participants nor the researchers who interact with them know which treatment each participant receives.

Blinding is not always possible: people know whether they did an exercise programme. You can often still blind the person who measures the response.

Extraneous variables, confounding and random assignment

An extraneous variable is any variable, other than the explanatory variable you are studying, that you know or suspect has an effect on the response. In a fertiliser study, the amount of sunlight each plant gets is extraneous.

A confounding variable is related to the explanatory variable in a way that makes it hard to tell which of the two is changing the response. Suppose a teacher lets students choose a new revision method. Keen students may choose it more often, and keen students may also score higher anyway. Keenness is then confounded with the method.

Random assignment reduces this problem. Because chance decides who gets which treatment, each extraneous variable (keenness, age, sleep, and even variables nobody thought of) should end up with roughly the same distribution in every treatment group. The groups are then similar except for the treatment, so a large difference in the response points to the treatments as the cause.

Random assignment does not make the groups identical. Chance differences remain; with more units per group (replication), they tend to be smaller.

How to describe random assignment. Give enough detail that someone else could do it exactly:

1. Label the units 1 to N.
2. Use a random number generator to choose labels between 1 and N, ignoring repeats.
3. The first chosen labels form treatment group 1, the next form group 2, and so on, until every unit has a treatment.

Slips of paper also work: write each treatment name on the right number of identical slips, mix them in a container, and let each unit draw one without replacement.

Three experimental designs

Completely randomized design. Every unit is assigned to a treatment completely at random. The groups are often equal in size, but they do not have to be.

Randomized block design. First group the units into blocks of units that are similar on a blocking variable, a variable you expect to affect the response. Then randomly assign the treatments separately inside each block, so that every treatment appears in every block.

Blocking separates the variation caused by the blocking variable from the rest of the variation. Inside a block, the units are alike on that variable, so differences between treatments are easier to see. This gives more precise comparisons. Blocking does not replace random assignment; it happens inside each block.

Matched pairs design. This is a randomized block design with exactly two treatments. It has two forms:

• Units are put into pairs that are similar on one or more extraneous variables, and a chance process decides which member of each pair gets which treatment.
• Each unit receives both treatments, and the order is decided at random.

Choosing a design

No design is always best. Choose by thinking about the goal of the study, the units available and the variables involved.

• If the units are fairly similar, or you know of no variable that strongly affects the response, a completely randomized design is simple and fine.
• If one variable clearly affects the response (soil type, age, prior ability, sunlight), block on it so its variation does not hide the treatment effect.
• If the units come naturally in pairs (two arms, twins, the same person twice), or you can match them closely, a matched pairs design gives very precise comparisons. Giving each unit both treatments is only sensible if the first treatment does not change how the unit responds to the second.

What can you conclude?

Two separate questions decide the scope of a conclusion.

	Units randomly selected from a population	Units not randomly selected (for example, volunteers)
Treatments randomly assigned	Cause and effect, for the whole population	Cause and effect, for units similar to those in the study
Treatments not randomly assigned	Association only, for the whole population	Association only, for units similar to those in the study

Random assignment reduces confounding, so it allows a cause-and-effect conclusion. It is often unethical or impractical to pick experimental units at random from a population, so many experiments use volunteers. That does not spoil the causal claim. It limits the conclusion to a population of units similar to the volunteers.

Worked example 1: a completely randomized design with a placebo

Question. A fictional company makes a herbal capsule called Restwell and claims it increases sleep. Ninety adult volunteers agree to take part. Each will be in one of three groups for two weeks: Restwell capsules, identical-looking placebo capsules, or no capsule. The response is the change in average nightly sleep, in minutes, compared with the week before the study.

(a) Identify the experimental units, the explanatory variable, the treatments and the response variable. (b) Describe how to randomly assign the treatments. (c) The mean changes were Restwell +28 minutes, placebo +19 minutes and no capsule +6 minutes. Estimate the placebo effect and the effect of Restwell beyond the placebo. (d) Can the study be double-blind? (e) What can the company conclude, and about whom?

(a) Units: the 90 adult volunteers. Explanatory variable: type of capsule taken. Treatments: Restwell, placebo, no capsule. Response: change in average nightly sleep (minutes).

(b) Label the volunteers 1 to 90. Use a random number generator to choose integers from 1 to 90, ignoring repeats. The first 30 different labels get Restwell, the next 30 get the placebo, and the remaining 30 get no capsule. This also gives replication: 30 people per treatment.

(c) Placebo effect = 19 − 6 = 13 minutes: people taking an inactive capsule gained, on average, 13 minutes more sleep than people taking nothing. Effect of Restwell beyond the placebo = 28 − 19 = 9 minutes. Comparing Restwell with no capsule (28 − 6 = 22 minutes) would overstate the capsule’s own effect, because 13 of those minutes are the placebo effect.

(d) Partly. The Restwell and placebo capsules look identical, so if the bottles are coded, neither those participants nor the researchers who meet them need to know which capsule is which: that comparison can be double-blind. The no-capsule group knows it is getting nothing, so that group cannot be blinded.

(e) The treatments were randomly assigned, so the difference in sleep is likely caused by the treatments (a later unit tests whether a difference this size could be due to chance alone). The volunteers were not a random sample, so the conclusion applies only to adults similar to these volunteers.

Worked example 2: why a matched pairs design is more precise

Question. A school compares two fictional typing apps, Keyflow (K) and Tapwise (T). Sixteen students take a typing test first. The two slowest are paired, then the next two, and so on, giving 8 pairs of similar speed. In each pair a coin flip decides who uses Keyflow; the other uses Tapwise. After four weeks each student’s speed (words per minute, wpm) is measured again.

Pair	1	2	3	4	5	6	7	8
Keyflow (wpm)	25	30	37	40	47	52	60	66
Tapwise (wpm)	22	28	33	39	44	50	55	64
Difference K − T	3	2	4	1	3	2	5	2

(a) Name the design and the variable used for matching. (b) Use the data to explain why matching helps.

(a) Matched pairs design (a randomized block design with two treatments, where each pair is a block). The matching variable is starting typing speed.

(b)

1. The speeds within each app vary a lot: Keyflow speeds range from 25 to 66 wpm (s ≈ 14.30 wpm) and Tapwise speeds from 22 to 64 wpm (s ≈ 14.18 wpm). Most of this is because the students started at different speeds.
2. The differences within pairs vary much less: from 1 to 5 wpm (s ≈ 1.28 wpm).
3. The mean difference is 22 ÷ 8 = 2.75 wpm, and Keyflow was faster in all 8 pairs.

Comparing within pairs removes the variation due to starting speed. A gap of 2.75 wpm is small next to the spread of 14 wpm between students, but large next to the spread of the differences. So the matched comparison is much more precise than comparing two unmatched groups.

Check. 44.625 − 41.875 = 2.75 wpm, the same as the mean difference. Giving each student both apps in random order would be poor here: practice with the first app would improve speed with the second.

Worked example 3: choosing and describing a block design

Question. A community garden tests three fertilisers, A, B and C, on 24 plots of tomatoes. Twelve plots are in full sun and twelve are shaded by a wall. The response is the mass of tomatoes from each plot. Recommend a design and describe it.

1. Spot the extraneous variable. Sunlight strongly affects tomato yield and is not what we are studying.
2. Choose the design. Use a randomized block design with sunlight as the blocking variable: block 1 is the 12 sunny plots and block 2 is the 12 shaded plots (Figure 1).
3. Assign within blocks. In block 1, label the plots 1 to 12, use a random number generator to pick 4 different labels for A and 4 more for B; the last 4 get C. Repeat separately in block 2. Each fertiliser is then used on 4 plots per block and 8 plots in all.
4. Directly control other variables: same tomato variety, same watering schedule, same harvest date.
5. Justify. In a completely randomized design, chance could put most of fertiliser A on sunny plots, so A would look better only because of the sun. Blocking tests each fertiliser equally in sun and shade and removes sunlight’s variation from the comparison.

Common misconceptions

• “Random assignment and random sampling do the same job.” Random assignment supports cause and effect. Random selection supports generalising to a population.
• “A control group gets nothing.” It may get a placebo or the current standard treatment.
• “Direct control means having a control group.” Direct control means keeping conditions the same for all units.
• “Replication means doing the whole experiment again.” Here it means more than one unit per treatment.
• “Double-blind means two groups are blinded.” It means participants and the researchers who interact with them are both unaware.
• “The placebo effect is treatment mean minus placebo mean.” The placebo effect compares placebo with no treatment.
• “Random assignment makes the groups identical.” It makes them similar on average; chance differences remain.
• “Experiments with volunteers cannot show cause and effect.” With random assignment they can. Volunteers limit who the conclusion applies to.
• “Blocking replaces randomization.” Treatments are still assigned at random, inside each block.

Where this leads

This topic completes Unit 1. Topic 1.12 looked at bias in samples; here random assignment protects an experiment. Chance processes return in Unit 2 (Probability, Random Variables, and Probability Distributions). “Could this difference be due to chance?” is answered by tests for two proportions in Unit 3 and for paired data and two means in Unit 4. Try the practice questions now, then use the revision notes and the checklist to consolidate. You can also return to the course roadmap.