Replicator dynamics of cancer stem cell: Selection in the presence of differentiation and plasticity

• We study the heterogeneity and plasticity in cancer cell populations.
• We derive the replicator dynamics of a three-compartment Moran-type model.
• The effective fitness of CSCs depends on net proliferation and dedifferentiation.
• The phase diagram of advantageous and disadvantageous CSCs are obtained.

The cancer stem cell hypothesis has evolved into one of the most important paradigms in cancer research. According to cancer stem cell hypothesis, somatic mutations in a subpopulation of cells can transform them into cancer stem cells with the unique potential of tumour initiation. Stem cells have the potential to produce lineages of non-stem cell populations (differentiated cells) via a ubiquitous hierarchal division scheme. Differentiation of a stem cell into (partially) differentiated cells can happen either symmetrically or asymmetrically. The selection dynamics of a mutant cancer stem cell should be investigated in the light of a stem cell proliferation hierarchy and presence of a non-stem cell population. By constructing a three-compartment Moran-type model composed of normal stem cells, mutant (cancer) stem cells and differentiated cells, we derive the replicator dynamics of stem cell frequencies where asymmetric differentiation and differentiated cell death rates are included in the model. We determine how these new factors change the conditions for a successful mutant invasion and discuss the variation on the steady state fraction of the population as different model parameters are changed. By including the phenotypic plasticity/dedifferentiation, in which a progenitor/differentiated cell can transform back into a cancer stem cell, we show that the effective fitness of mutant stem cells is not only determined by their proliferation and death rates but also according to their dedifferentiation potential. By numerically solving the model we derive the phase diagram of the advantageous and disadvantageous phases of cancer stem cells in the space of proliferation and dedifferentiation potentials. The result shows that at high enough dedifferentiation rates even a previously disadvantageous mutant can take over the population of normal stem cells. This observation has implications in different areas of cancer research including experimental observations that imply metastatic cancer stem cell types might have lower proliferation potential than other stem cell phenotypes while showing much more phenotypic plasticity and can undergo clonal expansion.