Bonferroni's correction, not Tukey's, should be used to control the total number of false positives when making multiple pairwise comparisons in experiments with few replicates.

Abstract

Statistical tests can be used to help determine whether experimental manipulations produce effects. In tests of means, when more than two groups are compared the total number of Type 1 errors (false positive results) increases unless a correction is used. Tukeys test is thought to offer good control of the false positive rate and high statistical power when all pairwise comparisons are made. However, the number of replicates in laboratory experiments is often quite low, and small sample sizes can undermine assumptions underlying statistical methods. I used simulations to investigate how well Tukeys test controls the total number of false positives when there are 3-6 experimental groups and 2-6 experimental replicates, conditions that span the range of typical values, and found that it generates too many. I investigated 11 other approaches to controlling false positives and found that none is as effective as the simple Bonferroni correction or offers much more power. I conclude that researchers should not make all pairwise comparisons using Tukeys test but instead use Bonferronis correction on a limited number of pre-selected comparisons.