Contemporary Use of Power Analysis in Hypothesis Testing

Abstract

Introduction

The power of a statistical test is the probability of obtaining statistically significant results when the alternative hypothesis is true [1]. The higher the power, the higher the chances of detecting an effect. Although numerous attempts of highlighting the importance of high-powered studies have been made, the prevalence of low powered studies remains a persistent issue in scientific research [2]. Low power not only reduces the likelihood of observing true effects but also minimises the likelihood that the obtained statistically significant results are true. Low power contributes to the production of exaggerated effect sizes, unreliable and nonreplicable results [2] and produces studies with overall low informational value [3]. Moreover, the conventional use of the .80 power value seems to be inadequate and lacks a solid justification [3].

Due to the presence of low-powered studies and neglect of power analysis in both prior [1] and recent history [2] we decided to conduct a comprehensive literature review examining contemporary use of power analysis in quantitative research in the field of cognitive science. Furthermore, we investigate the justifications provided for the utilised power values.

Aim

The study aims to address the following research questions: (1) How frequent is the use of power analysis? (2) What power values are employed? (3) What justifications are given, if any?

Methodology

A systematic review of 300 empirical articles from 2023 will be carried out. Articles will be chosen from six high-impact journals, forming three pairs, each representing a different methodological approach predominantly used in their respective publications. 

Individual studies within the articles will undergo separate evaluations. The evaluation criteria include the frequency and incorporation of power analysis, identification of the type utilised, validation of the presence of all necessary parameters and their provided justifications. Additionally, power values employed across studies will be documented.

A pilot evaluation of 20 articles will be carried out by two independent coders and inter-rater agreement will be assessed. The results will be compared across three fields using different methodological approaches: neuroimaging studies, behavioural experiments with healthy participants and neuropsychological research using patient data.

Expected results

A lack of use of any form of power analysis is anticipated and errors in reporting or calculating power are expected. Furthermore, a lack of justifications for the used parameters and continued reliance on the conventional .80 power benchmark are expected. Frequency and accuracy of use might differ depending on the employed methodologies. Overall, the results will give us an insight into the state of contemporary use of power analysis, with implications for reliability, validity, and replicability of scientific findings in the field of cognitive science.

References

[1] J. Cohen, “A power primer.,” Psychological Bulletin, vol. 112, no. 1, pp. 155–159, 1992. doi:10.1037//0033-2909.112.1.155

[2] K. S. Button et al., “Power failure: Why small sample size undermines the reliability of neuroscience,” Nature Reviews Neuroscience, vol. 14, no. 5, pp. 365–376, Apr. 2013. doi:10.1038/nrn3475

[3] D. Lakens, “Sample size justification,” Collabra: Psychology, vol. 8, no. 1, 2022. doi:10.1525/collabra.33267