Role of epistasis on the fixation probability of a non-mutator in an adapted asexual population

•Fixation probability of a single nonmutator is studied.•Nonmutator has an indirect selective advantage.•A multitype branching process is used in the model.•Antagonistic epistasis lowers the fixation probability of nonmutator.•Synergistic epistasis enhances the chances of mutation rate reduction.

The mutation rate of a well adapted population is prone to reduction so as to have a lower mutational load. We aim to understand the role of epistatic interactions between the fitness affecting mutations in this process. Using a multitype branching process, the fixation probability of a single non-mutator emerging in a large asexual mutator population is analytically calculated here. The mutator population undergoes deleterious mutations at constant, but at a much higher rate than that of the non-mutator. We find that antagonistic epistasis lowers the chances of mutation rate reduction, while synergistic epistasis enhances it. Below a critical value of epistasis, the fixation probability behaves non-monotonically with variation in the mutation rate of the background population. Moreover, the variation of this critical value of the epistasis parameter with the strength of the mutator is discussed in the appendix. For synergistic epistasis, when selection is varied, the fixation probability reduces overall, with damped oscillations.