Sign of selection on mutation rate modifiers depends on population size

The influence of population size (N) on natural selection acting on alleles that affect fitness has been understood for over half a century1. As N declines, genetic drift overwhelms selection and alleles with direct fitness effects are rendered neutral. Often, though, alleles experience so called indirect selection, meaning they affect not the fitness of an individual but the fitness distribution of its offspring. Some of the best studied examples of indirect selection include alleles that modify aspects of the genetic system such as recombination2 and mutation3 rates. Here we use analytics, simulations and experimental populations of S. cerevisiae to show that modifiers that increase the genomic mutation rate (mutators) are favored by indirect selection in large populations but become disfavored as N declines. This surprising phenomenon of sign inversion in selective effect demonstrates that indirect selection on a mutator exhibits a qualitatively novel dependence on N. Sign inversion may help understand the relatively sporadic distribution of mutators in nature despite their frequent emergence in laboratory populations. More generally, sign inversion may be broadly applicable to other instances of indirect selection, suggesting a previously unappreciated but critical role of population size in evolution.