How many participants? Sample size for imaging and digital-test studies

What drives sample size

Every sample-size calculation answers one question: how many participants does the study need to detect an effect of a given size, with a given probability, if the effect is real? That probability is statistical power, usually set at 80% or 90%.

Five inputs drive the answer. The effect size you need to detect: smaller effects need larger samples, and the effect should be the smallest one that would matter, not the one you hope to see. The variability of the measure: noisy measures need more people. The design: repeated measurements on the same people, matched groups and covariates can all reduce the number needed. Drop-out and missing data: the analysed sample is smaller than the recruited one. And multiple comparisons: testing many regions or outcomes needs a stricter threshold.

Underpowered studies are a common problem in neuroscience. They miss real effects, and the effects they do find tend to be exaggerated (Button et al., 2013). A credible sample size is therefore not a formality; it protects the study's conclusions.

Case 1: a group difference in a regional brain measure

Suppose a study compares the volume of one brain region between two groups. Published data suggest a standardized difference (Cohen's d) of 0.5. With a two-sided test at the 5% level and 80% power, a two-sample t-test needs about 64 participants per group.

If 15% of participants are expected to drop out or fail image quality control, recruitment rises to 64 / 0.85, about 76 per group. If the study tests ten regions and corrects for multiple comparisons, the required number rises again. Each assumption should be written down so a reviewer can check it.

Case 2: an ICC reliability study

A team wants to show that a digital test gives consistent results when taken twice. Reliability is measured with the ICC. Here the goal is not just a significant ICC (almost any test beats zero) but evidence that reliability exceeds a minimum acceptable value.

If the minimum acceptable ICC is 0.6, the expected ICC is 0.8, each participant is tested twice, and the test is one-sided at the 5% level with 80% power, a standard approximation gives about 39 participants. Practice effects, drop-out between sessions and subgroup analyses all increase that number. Guidelines for choosing and reporting the type of ICC are summarized by Koo and Li (2016).

Case 3: a Bland–Altman agreement study

A device or digital test is compared with a reference method. Agreement is shown with a Bland–Altman plot: the mean difference and the 95% limits of agreement. Bland and Altman (1986) showed that the standard error of each limit is about the square root of 3s²/n, where s is the standard deviation of the differences.

That gives a direct way to plan: with 50 participants, the 95% confidence interval around each limit extends about 0.48 standard deviations either side; with 100 participants, about 0.34; with 200, about 0.24. The team chooses the precision that makes the limits useful for their decision, and that sets the sample size.

When formulas fail: simulation

Formulas assume simple designs. Real studies often have several visits per person, missing scans, clustering by site, or a mixed-effects model as the main analysis. In those cases the honest approach is simulation: generate many artificial data sets under the planned design and assumptions, analyse each one exactly as planned, and count how often the effect is detected.

Simulation makes every assumption visible and testable. It also answers questions formulas cannot, such as how power changes if drop-out is higher at later visits. For prediction models, dedicated methods exist that also account for the number of predictors and the outcome rate (Riley et al., 2020).

What grant reviewers look for

Reviewers rarely object to a number; they object to an unexplained number. A strong sample-size section names the primary outcome, gives the effect size with its source, states the significance level and power, explains how drop-out and multiple comparisons were handled, and shows what happens if the assumptions are wrong. A short sensitivity table (power at a smaller effect, or at higher drop-out) answers most questions before they are asked.

Reviewers also look for consistency. The sample size should match the primary outcome named in the aims, the analysis described in the statistical analysis plan, and the budget for recruitment and imaging. A mismatch, such as a power calculation for a simple t-test when the analysis plan describes a mixed-effects model with covariates, is one of the most common reasons a methods section draws criticism. Writing the sample-size justification and the analysis plan together, from the same assumptions, avoids it.

A checklist

1. Name the primary outcome and the analysis that will test it.
2. Justify the effect size from published data or a pilot, and use the smallest effect that matters.
3. State the significance level, power and whether the test is one- or two-sided.
4. Inflate for expected drop-out and quality-control exclusions.
5. Account for multiple comparisons or co-primary outcomes.
6. For reliability and agreement studies, plan for the precision of the ICC or the limits of agreement.
7. Use simulation when the design or analysis is complex.
8. Show a short sensitivity analysis of the key assumptions.

References

1. Button KS, Ioannidis JPA, Mokrysz C, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013. doi:10.1038/nrn3475 (external site)
2. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016. doi:10.1016/j.jcm.2016.02.012 (external site)
3. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet. 1986. doi:10.1016/S0140-6736(86)90837-8 (external site)
4. Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020. doi:10.1136/bmj.m441 (external site)