• 12. Study design summary

Links to: Summary. Chatbot tutor. Questions. Glossary. R packages. Additional resources.

Chapter summary

Cueball pulls a lever. Lightning hits Cueball. ZAP. Cueball still stands, obviously battered. An arrow labelled "normal person" points to a panel of Cueball thinking. Cueball (thinking): "I guess I shouldn't do that." An arrow labelled "scientist" points to a panel of Cueball about to pull the level again. Cueball (thinking): I wonder if that happens every time.
Figure 1: An xkcd comic. Rollover text: How could you choose avoiding a little pain over understanding a magic lightning machine?. See the related explain xkcd for more info. CC BY-NC 2.5, Comic by Randall Munroe, xkcd.com.

The ultimate success or failure of a statistical analysis is often shaped more by how the study was designed and the data were generated, than by how the statistics were performed. So, you should think carefully about who, what, and how to sample, measure, and assign individuals to a treatment group. As compared to observational studies, in which known or unknown confounds can prevent a clear causal understanding, the random assignment of individuals to treatment groups in experimental studies provides us an opportunity to learn about causation. It is therefore critical that we carefully design experiments to avoid potential sources of bias including experimental artifacts, placebo effects, and regression toward mediocrity. And, if you’re going through the effort of doing scientific research, and designing a good study, you should make sure that you have the design and sample size needed to find interesting results, if they do indeed exist. Finally, while experiments are great, we sometimes cannot do them. Don’t fret – we can do our best to mitigate confounding, by applying best practices from experimental design to our observational studies!

Chatbot tutor

(Add your ChatGPT and Gemini chatbot links here, matching the pattern from earlier chapters – I don’t have real links to provide, so I’ve left this as a placeholder rather than invent one.)

Please interact with this custom chatbot (ChatGPT has stopped supporting this :/, Gemini version here). I suggest interacting with at least ten back-and-forths to ramp up, and then stopping when you feel like you got what you needed from it.

Practice Questions

Try these questions!

Q1) Random assignment describes the a key feature of experimental design in which the experimenter

That second option describes random selection (or random sampling). Random selection is about randomly choosing who gets into the study, from the broader population you care about.

By contrast random assignment describes randomly assigning treatments to individuals in the study.

Q2) IMO “effect size” is a bad name. Take for example, our RIL study, which estimated the “effect size” of petal area on proportion hybrid seed as \(r = .227\). Why is calling this an effect size misleading?

Because we often apply the term “effect size” to observational studies, which do not show “effects,” just “associations.”

Q3) What distinguishes a scientific model from a statistical model?
Q4) Two techniques in study design – blocking and matching – have a similar purpose. In “blocking,” we randomly assign treatments in our experiment within “blocks” (e.g. fields, schools, etc) so that the all treatments are well-represented in all blocks. In a “matching” experimental design we create pairs that are as similar as possible to each other, and then randomly apply the treatment to one member of the pair. Compared to other experimental approaches, what underlying problem are both of these techniques trying to solve?

If you answered “Both aim to remove confounds confounding variables are eliminated,” you’re close. Random assignment already protects experiments against systematic confounding by treatment. Blocking and matching go a step further, they try to balance known sources of variation across treatments.

Q5) TRUE or FALSE: Random assignment controls for covariates by modeling the effect of each one.

Random assignment does not model covariates. It works through study design. Treatment is assigned independently of other variables, so potential confounders should be balanced across treatment groups on average. Modeling covariates is a separate statistical adjustment. This is a great benefit of random assignment – we can deal with confounds without guessing what they may be.

Q6) Say we ran an experiment with appropriate controls and failed to reject the null hypothesis. Does this prove there’s no true causal relationship? For each reason below, is it a valid explanation for why a true effect could still fail to reach significance?

  • The experiment’s conditions were unnatural enough that they blocked the causal pathway from operating at all (e.g., testing a pollinator-attraction hypothesis with no pollinators present). .
  • The treatment’s intensity or dose differed enough from what’s experienced in nature that it didn’t trigger the same effect. .
  • The study might simply have been underpowered – too small a sample to reliably detect a real but modest effect. .
  • Failing to reject the null proves the null hypothesis is true. .

“Absence of evidence is not evidence of absence.” A null result can mean the effect isn’t there – or it can mean the effect is there, but the study wasn’t built to catch it: wrong context, wrong dose, or not enough power. None of the first three possibilities require you to abandon the causal hypothesis; they just mean this particular experiment couldn’t settle the question.

For the remaining questions, consider this setup: Glyphosate is the active ingredient in Roundup, a very common herbicide. In 2015, the International Agency for Research on Cancer classified it as “probably carcinogenic to humans.” By contrast, the U.S. EPA has maintained that it’s unlikely to cause cancer when used as directed. Since 2018, Bayer (which acquired Monsanto, Roundup’s manufacturer) has faced > 100k lawsuits from people who developed non-Hodgkin lymphoma after using the product, including school groundskeeper Dewayne Johnson, whose 2018 trial was the first to reach a jury, and Edwin Hardeman, who sprayed Roundup on his own (56-acre) property for 20 years. The company formerly known as Monsanto (Bayer) has paid > $10 billion in settlements and verdicts to date, while still denying that glyphosate causes cancer.

Q7) Agricultural workers exposed to glyphosate were often apply several other pesticides, so it can be hard to isolate glyphosate’s effect from the others. Why is this is a threat to internal validity

Co-exposure is a classic confound: workers weren’t randomly assigned their glyphosate exposure, they were farmers doing normal farm work – which came bundled with exposure to other chemicals too. A study can find a real, non-random association between glyphosate and cancer risk without being able to say glyphosate itself is the cause, because glyphosate use and other-pesticide use move together.

Q8) The strongest evidence connecting glyphosate to human cancer risk comes from studies of licensed agricultural applicators – people with heavy, often decades-long occupational exposure. But the lawsuits about glyphosate include plaintiffs like Hardeman, whose exposure was residential. Why does this gap raises a question of external validity?

This is a real, live tension, not a hypothetical one: the core evidence base is heaviest for people like licensed applicators, but the plaintiff population is much broader. Whether findings from one population extend to the other is exactly what external validity asks – and it’s a genuine point of dispute in how this evidence has been applied in court and by regulators.

Q9) Some of the animal evidence connecting glyphosate to cancer came from rodent studies using doses far above typical human exposure – a standard practice called testing at the “maximum tolerated dose.” Why does this raise a question of ecological validity?

Testing at high doses isn’t a design flaw on its own – regulators require it specifically because it’s often the only practical way to detect a rare effect in a study with a manageable number of animals. But it creates a real interpretive gap: a dose-response relationship near the maximum tolerated dose doesn’t automatically tell you whether the same relationship holds at everyday exposure levels, which is exactly what regulatory agencies have had to argue about when interpreting this evidence.

Q10) As of 2026, roughly 170,000 Roundup-related lawsuits have been filed. For each statement, is it a legitimate reason this count is an untrustworthy estimate of how many people glyphosate exposure has actually harmed?

  • It could plausibly overstate true harm. Filing a lawsuit doesn’t require proving glyphosate caused that specific person’s cancer, so some claims may be frivolous. .
  • It could plausibly understate true harm. Highly exposed populations with high occupational exposure – especially undocumented agricultural workers – are unlikely to pursue legal claims, for reasons unrelated to whether they were actually harmed.
  • 170,000 is simply too large a number to be accurate.
  • Bayer has settled most claims, so the true number of harmed people must be lower.
Q11) In 2023, Bayer began phasing glyphosate out of its U.S. residential lawn-and-garden products, while continuing to sell it for agricultural use. If glyphosate exposure really does drive non-Hodgkin lymphoma risk at levels common in residential application, what would you predict?

This is the distinguishing prediction, not just a justification after the fact. Bayer’s decision changed exposure for one group (residential users) but not the other (agricultural workers), for reasons unrelated to either group’s cancer risk. If glyphosate itself is driving risk, that predicts a specific, checkable pattern: a bigger drop in residential rates than agricultural rates over the same period. If we don’t see that pattern – say, rates drop equally in both groups – that’s evidence against glyphosate being the responsible agent, since something else would have to explain an equal drop in a group whose exposure didn’t change. Comparing the size of the change in an exposed group against an otherwise-similar unexposed group, over the same period, is a specific, well-known technique you’ll meet by name later in your statistical education.

📊 Glossary of Terms

  • Confounding variable: An unmodeled variable that distorts the relationship between an explanatory and response variable, making it unclear whether the explanatory variable, the confound, or both are responsible for an observed association.
  • Construct validity (aka measurement validity): How well the thing you measured actually captures the thing you meant to study.
  • Measurement reliability: How similar your measurement would be if you took it again – a question of consistency, separate from whether you’re measuring the right thing in the first place.
  • Internal validity: How well a study design lets you draw the specific causal conclusion you’re after, for the individuals and conditions used in that particular study.
  • External validity: How far a study’s results generalize beyond the specific study that produced them.
  • Ecological validity: Whether a treatment’s intensity falls within the range organisms actually experience in nature, rather than an artificially extreme dose or condition.
  • Statistical conclusion validity: Whether the statistical inference itself is sound – adequate power, appropriate handling of non-independence – separate from whether the design itself was valid.
  • Random assignment: Assigning treatment to experimental units by chance, so that any resulting group differences are attributable to the treatment rather than to pre-existing differences between units.
  • Experimental artifact: A result caused by some feature of the experimental manipulation itself, rather than by the treatment of actual interest.
  • Placebo effect: A change in outcome caused by the belief that one has received a treatment, rather than by the treatment itself.
  • Regression to the mean: The tendency for extreme observations to be closer to average on a second measurement, simply because part of what made them extreme was random noise unlikely to repeat.
  • Blocking: Deliberately grouping experimental units by a nuisance factor (like field site) and ensuring a balanced, complete set of treatments within each group, so that group-to-group variation can be modeled rather than left as noise.
  • Randomized Complete Block Design (RCBD): The formal name for a blocked design in which every block contains a complete, balanced set of the treatment levels.
  • Matching: Pairing units that are as similar as possible except for the variable of interest, so that other sources of variation cancel out within each pair.
  • Split-plot design: A design used when one factor can only be randomized at a coarser scale (a whole plot) than another factor, which can be randomized more finely within it (a split plot).
  • Stratifying: Comparing outcomes within levels of a suspected confounder, rather than pooling across all levels – the observational analog of blocking.
  • Natural experiment: A real-world situation in which something close to random assignment occurs without a researcher designing it, allowing for stronger causal inference than typical observational data.
  • Pseudoreplication: Treating non-independent observations as if they were independent replicates, which inflates apparent precision without actually adding independent information.
  • Power: The probability of correctly rejecting a false null hypothesis.
  • Effect size: The magnitude of the difference or relationship a study is trying to detect.
  • Triangulation: Using multiple lines of evidence with different likely failure modes to build a stronger case than any single study could support on its own.

R Packages Introduced

  • pwr: the most widely used R package for power analysis – finding the sample size needed to detect a given effect size with a given level of power, or vice versa.

  • presize: a complementary package (with an accompanying web app) for planning sample size around the precision of an estimate, rather than the power to reject a null.

Additional resources

Readings:

Book chapter:

  • Chapter 23: Sample Size Calculations with {pwr} – from Reproducible Medical Research with R (Higgins). A hands-on walkthrough of the pwr package with a full table mapping test type to function (t-tests, ANOVA, correlation, chi-squared, and more), plus worked clinical examples.

Web app:

Podcast:

  • Placebo – Radiolab (2009). The full-length episode behind the short clip already excerpted in Experimental Design – an hour on the neuroscience of belief, from the symbolic power of a doctor’s coat to a visit with a faith healer.

Videos: