Evaluating a digital cognitive test: reliability, agreement and fairness

Why evaluation matters

Digital cognitive tests can be taken on a tablet, a computer or a phone, often without a trained examiner. That makes them attractive for research and for reaching people who would not otherwise be assessed. It also raises new questions: does the device or the setting change the score, do people improve simply by repeating the test, and does the test work equally well for everyone who takes it?

Answering those questions takes a set of studies, each designed for one purpose. This note sets out the main ones and the statistics behind them.

Reliability and practice effects

test–retest reliability asks whether a person gets a similar score when tested twice under the same conditions. It is usually reported as an ICC. The type of ICC must match the design (for example, absolute agreement versus consistency), and it should be reported with a confidence interval. A common guideline treats values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good and above 0.9 as excellent (Koo and Li, 2016).

Cognitive tests have a specific complication: the practice effect. People often score higher the second time simply because they have seen the test before. A good reliability study measures the size of this effect, chooses a retest interval that fits the intended use, and, where alternate forms exist, checks whether they reduce it. Our own work on longitudinal reliability of brain-age measures follows the same logic (Beheshti et al., 2021).

Agreement: why correlation is not enough

A new digital test is often compared with an established paper-and-pencil test. The tempting summary is a correlation coefficient. But correlation measures whether scores rise and fall together, not whether they are the same. Two tests can correlate at 0.9 while one consistently scores several points higher.

The standard method is the Bland–Altman plot (Bland and Altman, 1986). It plots the difference between the two tests against their average and reports the mean difference (systematic bias) and the 95% limits of agreement. Whether those limits are acceptable is a judgement about the intended use, and it should be made before the data are analysed. When two methods are meant to be interchangeable, equivalence testing can test whether the difference is smaller than a pre-set margin (Lakens, 2017).

Accuracy: discrimination and calibration

If a test is meant to separate groups, for example people with and without mild cognitive impairment in a research sample, its accuracy is described by sensitivity and specificity at a chosen threshold and by the AUC across all thresholds.

When a test produces a probability or a risk score, calibration matters as much as discrimination: a predicted 30% should correspond to about 30% of people with the outcome. Calibration is often neglected and often poor (Van Calster et al., 2019). Accuracy estimates should come from data not used to build or tune the scoring, ideally with external validation in a different setting.

Norms

A raw score means little until it is compared with what is typical. normative data come from a reference group and show how scores vary with age, sex and education. Norms built from a convenience sample of highly educated volunteers can mislabel many people as impaired. A norms study plans the reference sample to match the population in which the test will be used and models how scores change across age and education rather than relying on a few coarse bands.

Norms also age. Populations change, devices and operating systems change, and a score that was typical ten years ago may not be typical today. A norms study should record when and how the data were collected, the device and software version used, and the exclusion rules, so that later users can judge whether the norms still apply. Where a test is used across countries or languages, separate norms, or at least a check that the same norms hold, are usually needed.

Fairness across groups

A test can be reliable and accurate on average while performing worse for some groups: older adults less familiar with touch screens, people tested in a second language, or people with fewer years of schooling. A fairness analysis compares reliability, agreement and accuracy across these groups and checks whether items behave the same way in each (measurement invariance).

Small subgroups give uncertain estimates. Good reporting says so plainly, shows confidence intervals for each group and identifies where more data are needed, rather than declaring a test fair or unfair from thin evidence.

The medical-device boundary

Software intended to screen for, diagnose or monitor a condition can be regulated as software as a medical device. In Canada, Health Canada decides whether a product is a medical device and which class applies; the developer is responsible for that determination and any licence. Evaluation studies produce evidence that informs those decisions, but they do not settle them.

For that reason, careful language matters. A report can say that a study found good test–retest reliability in a defined sample. It should not say that a test is "validated" or "diagnostic" unless that claim has been established through the appropriate regulatory route. Cedar designs and analyses evaluation studies; it does not certify tests or prepare regulatory submissions.

A checklist

1. Define the intended use and population before designing any study.
2. Plan a test–retest study with an interval that fits the use, and measure practice effects.
3. Report the ICC type, value and confidence interval.
4. Show agreement with a Bland–Altman analysis; set acceptable limits in advance.
5. Report discrimination and calibration on data not used to build the score.
6. Build norms from a reference sample that matches the intended population.
7. Compare performance across age, sex, language and education, with intervals.
8. Use careful language and leave regulatory claims to the regulatory process.

References

1. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016. doi:10.1016/j.jcm.2016.02.012 (external site)
2. Beheshti I, Potvin O, Duchesne S. Patch-wise brain age longitudinal reliability. Human Brain Mapping. 2021. doi:10.1002/hbm.25253 (external site)
3. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet. 1986. doi:10.1016/S0140-6736(86)90837-8 (external site)
4. Lakens D. Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science. 2017. doi:10.1177/1948550617697177 (external site)
5. Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019. doi:10.1186/s12916-019-1466-7 (external site)