Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Increasing the statistical power of animal experiments with historical control data

Nature Neuroscience volume 24, pages 470–477 (2021)Cite this article

Subjects

Abstract

Low statistical power reduces the reliability of animal research; yet, increasing sample sizes to increase statistical power is problematic for both ethical and practical reasons. We present an alternative solution using Bayesian priors based on historical control data, which capitalizes on the observation that control groups in general are expected to be similar to each other. In a simulation study, we show that including data from control groups of previous studies could halve the minimum sample size required to reach the canonical 80% power or increase power when using the same number of animals. We validated the approach on a dataset based on seven independent rodent studies on the cognitive effects of early-life adversity. We present an open-source tool, RePAIR, that can be widely used to apply this approach and increase statistical power, thereby improving the reliability of animal experiments.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Many animal experiments are severely underpowered.
Fig. 2: Historical controls can decrease the number of animals required for sufficiently powered research.
Fig. 3: Sensitivity simulation.

Similar content being viewed by others

Data availability

The data that support the findings of the current study can be downloaded from https://osf.io/wvs7m/.

Code availability

All code used in this manuscript is available at https://osf.io/wvs7m/.

References

  1. Ioannidis, J. P. A. Why most published research findings are false. PLoS Med. 2, e124 (2005).

    Article  Google Scholar 

  2. Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).

    Article  CAS  Google Scholar 

  3. Smaldino, P. E. & McElreath, R. The natural selection of bad science. R. Soc. Open Sci. 3, 160384 (2016).

  4. Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).

    Article  Google Scholar 

  5. Crabbe, J. C., Wahlsten, D. & Dudek, B. C. Genetics of mouse behavior: interactions with laboratory environment. Science 284, 1670–1672 (1999).

    Article  CAS  Google Scholar 

  6. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712–713 (2011).

    Article  CAS  Google Scholar 

  7. Voelkl, B. et al. Reproducibility of animal research in light of biological variation. Nat. Rev. Neurosci. 21, 384–393 (2020).

  8. Macleod, M. & Mohan, S. Reproducibility and rigor in animal-based research. ILAR J. 60, 17–23 (2019).

    Article  CAS  Google Scholar 

  9. Bonapersona, V. et al. The behavioral phenotype of early life adversity: a 3-level meta-analysis of rodent studies. Neurosci. Biobehav. Rev. 102, 299–307 (2019).

    Article  CAS  Google Scholar 

  10. Gurevitch, J., Koricheva, J., Nakagawa, S. & Stewart, G. Meta-analysis and the science of research synthesis. Nature 555, 175–182 (2018).

    Article  CAS  Google Scholar 

  11. Richter, S. H., Garner, J. P. & Würbel, H. Environmental standardization: cure or cause of poor reproducibility in animal experiments? Nat. Methods 6, 257–261 (2009).

    Article  CAS  Google Scholar 

  12. Karp, N. A. Reproducible preclinical research—is embracing variability the answer? PLoS Biol. 16, e2005413 (2018).

    Article  Google Scholar 

  13. Richter, S. H. et al. Effect of population heterogenization on the reproducibility of mouse behavior: a multi-laboratory study. PLoS ONE 6, e16461 (2011).

  14. Kramer, M. & Font, E. Reducing sample size in experiments with animals: historical controls and related strategies. Biol. Rev. Philos. Soc. 92, 431–445 (2017).

    Article  Google Scholar 

  15. Brakenhoff, T., Roes, K. & Nikolakopoulos, S. Bayesian sample size re-estimation using power priors. Stat. Methods Med. Res. 28, 1664–1675 (2018).

  16. Spiegelhalter, D. J., Abrams, K. R. & Myles, J. P. Bayesian Approaches to Clinical Trials and Health-Care Evaluation 1 (Wiley, 2004).

  17. Galwey, N. W. Supplementation of a clinical trial by historical control data: is the prospect of dynamic borrowing an illusion? Stat. Med. 36, 899–916 (2017).

    Article  CAS  Google Scholar 

  18. Mutsvari, T., Tytgat, D. & Walley, R. Addressing potential prior-data conflict when using informative priors in proof-of-concept studies. Pharm. Stat. 15, 28–36 (2016).

    Article  Google Scholar 

  19. Walley, R. et al. Using Bayesian analysis in repeated preclinical in vivo studies for a more effective use of animals. Pharm. Stat. 15, 277–285 (2016).

    Article  Google Scholar 

  20. Nord, C. L., Valton, V., Wood, J. & Roiser, J. P. Power-up: a reanalysis of ‘power failure’ in neuroscience using mixture modeling. J. Neurosci. 37, 8051–8061 (2017).

    Article  CAS  Google Scholar 

  21. Cohen, J. Statistical Power Analysis for the Behavioral Sciences (Lawrence Erlbaum Associates, 1977).

  22. Ibrahim, J. G. & Chen, M. H. Power prior distributions for regression models. Stat. Sci. 15, 46–60 (2000).

    Google Scholar 

  23. Rice, C. J., Sandman, C. A., Lenjavi, M. R. & Baram, T. Z. A novel mouse model for acute and long-lasting consequences of early life stress. Endocrinology 149, 4892–4900 (2008).

    Article  CAS  Google Scholar 

  24. Percie du Sert, N. et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. PLoS Biol. 18, e3000410 (2020).

    Article  CAS  Google Scholar 

  25. Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008).

    Article  Google Scholar 

  26. Rubin, E. J. & Fortune, S. M. Misunderstanding the goals of animal research. BMJ 360, 29321149 (2018).

    Google Scholar 

  27. Gelman, A. Objections to Bayesian statistics. Bayesian Anal. 3, 445–450 (2008).

    Article  Google Scholar 

  28. Viele, K. et al. Use of historical control data for assessing treatment effects in clinical trials. Pharm. Stat. 13, 41–54 (2014).

    Article  Google Scholar 

  29. Neuenschwander, B., Capkun-Niggli, G., Branson, M. & Spiegelhalter, D. J. Summarizing historical information on controls in clinical trials. Clin. Trials 7, 5–18 (2010).

    Article  Google Scholar 

  30. O’Hagan, A. Expert knowledge elicitation: subjective but scientific. Am. Stat. 73, 69–81 (2019).

    Article  Google Scholar 

  31. van der Naald, M., Wenker, S., Doevendans, P. A., Wever, K. E. & Chamuleau, S. A. J. Publication rate in preclinical research: a plea for preregistration. BMJ Open Sci. 4, e100051 (2020).

    Google Scholar 

  32. Du Sert, N. P. et al. The experimental design assistant. Nat. Methods 14, 1024–1025 (2017).

    Article  Google Scholar 

  33. Crabbe, J. C. & Phillips, T. J. Mother nature meets mother nurture. Nat. Neurosci. 6, 440–442 (2003).

    Article  CAS  Google Scholar 

  34. Kafkafi, N. et al. Addressing reproducibility in single-laboratory phenotyping experiments. Nat. Methods 14, 462–464 (2017).

    Article  CAS  Google Scholar 

  35. Shansky, R. M. Are hormones a ‘female problem’ for animal research? Science 364, 825–826 (2019).

    Article  CAS  Google Scholar 

  36. Bonapersona, V., Joëls, M. & Sarabdjitsingh, R. A. Effects of early life stress on biochemical indicators of the dopaminergic system: a 3 level meta-analysis of rodent studies. Neurosci. Biobehav. Rev. 95, 1–16 (2018).

    Article  CAS  Google Scholar 

  37. Wickham, H. et al. Welcome to the Tidyverse. J. Open Source Softw. 4, 1686 (2019).

    Article  Google Scholar 

  38. Chang, W., Cheng, J., Allaire, J., Xie, Y. & McPherson, J. shiny: Web Application Framework for R. R Package version 1.5.0 https://CRAN.R-project.org/package=shiny (2020).

  39. Ekstrøm, C. T. MESS: Miscellaneous Esoteric Statistical Scripts. R package version 0.5.6 https://CRAN.R-project.org/package=MESS (2019).

  40. Faul, F., Erdfelder, E., Lang, A. G. & Buchner, A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175–191 (2007).

    Article  Google Scholar 

  41. Lenth, R. V. Some practical guidelines for effective sample size determination. Am. Stat. 55, 187–193 (2001).

    Article  Google Scholar 

  42. Gelman, A., Carlin, J. B., Stern, H. S. & Rubin, D. B. Bayesian Data Analysis. (Chapman & Hall, 1995).

Download references

Acknowledgements

We thank J. Knop and M. Sep for helpful discussions and R. de Kloet for critically reviewing the manuscript. This work was supported by the Consortium of Individual Development (CID), which is funded by the Gravitation program of the Dutch Ministry of Education, Culture and Science and the Netherlands Organization for Scientific Research (NWO grant no. 024.001.003).

Author information

Author notes

  1. These authors contributed equally: R. A. Sarabdjitsingh, M. Joëls.

Authors and Affiliations

  1. Department of Translational Neuroscience, University Medical Center Utrecht Brain Center, Utrecht University, Utrecht, The Netherlands

    V. Bonapersona, J. Knop, R. A. Sarabdjitsingh & M. Joëls

  2. Department of Methodology and Statistics, Utrecht University, Utrecht, The Netherlands

    H. Hoijtink

  3. University of Groningen, University Medical Center Groningen, Groningen, The Netherlands

    M. Joëls

  4. Swammerdam Institute for Life Sciences, SILS-CNS, University of Amsterdam, Amsterdam, The Netherlands

    M. Abbinck, A. Korosi, H. J. Krugers, E. F. G. Naninck, K. Reemst, S. R. Ruigrok, E. H. L. Umeoka & K. Y. Yam

  5. Department of Anatomy/Neurobiology, University of California Irvine, Irvine, CA, USA

    T. Z. Baram & J. L. Bolton

  6. Department of Pediatrics, University of California Irvine, Irvine, CA, USA

    T. Z. Baram

  7. Department of Stress Neurobiology and Neurogenetics, Max Planck Institute of Psychiatry, Munich, Germany

    J. Bordes & M. V. Schmidt

  8. National Clinical Research Center for Mental Disorders, Peking University, Beijing, China

    J. T. Li

  9. Key Laboratory of Mental Health, Peking University, Beijing, China

    J. T. Li

  10. Faculty of Medicine, University Center Unicerrado, Goiatuba, Brazil

    E. H. L. Umeoka

  11. Department of Psychiatry, McGill University, Montreal, Quebec, Canada

    C. D. Walker

  12. Department of Neurobiology, Zhejiang University School of Medicine, Hangzhou, China

    X. D. Wang

Authors
  1. V. Bonapersona
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. H. Hoijtink
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. A. Sarabdjitsingh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. M. Joëls
    View author publications

    You can also search for this author in PubMed Google Scholar

Consortia

RELACS Consortium

  • M. Abbinck
  • , T. Z. Baram
  • , J. L. Bolton
  • , V. Bonapersona
  • , J. Bordes
  • , M. Joëls
  • , J. Knop
  • , A. Korosi
  • , H. J. Krugers
  • , J. T. Li
  • , E. F. G. Naninck
  • , K. Reemst
  • , S. R. Ruigrok
  • , R. A. Sarabdjitsingh
  • , M. V. Schmidt
  • , E. H. L. Umeoka
  • , C. D. Walker
  • , X. D. Wang
  •  & K. Y. Yam

Contributions

V.B. contributed to conceptualization, data curation, analysis, investigation, methodology, software, visualization and writing the manuscript; H.H. contributed to conceptualization, analysis, methodology, supervision and reviewing and editing the manuscript; members of the RELACS consortium provided the data; R.A.S. contributed to conceptualization, project administration, supervision and editing and reviewing the manuscript; M.J. contributed to conceptualization, funding acquisition, project administration, supervision and writing the manuscript.

Corresponding author

Correspondence to V. Bonapersona.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Neuroscience thanks Stanley Lazic, Malcolm MacLeod, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary Notes 1–4, Supplementary Fig. 1 and Supplementary Tables 1–4

Reporting Summary

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bonapersona, V., Hoijtink, H., RELACS Consortium. et al. Increasing the statistical power of animal experiments with historical control data. Nat Neurosci 24, 470–477 (2021). https://doi.org/10.1038/s41593-020-00792-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41593-020-00792-3

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing