A new protocol for intermittent androgen suppression therapy of prostate cancer with unstable saddle-point dynamics

• We propose a new protocol for intermittent hormonal therapy for prostate cancer.
• Tumor cell population dynamics are modeled by a hybrid dynamical system.
• Dynamics are determined by event-driven transition between two unstable subsystems.
• Saddle-point dynamics of the subsystems are deeply involved in the stabilization.
• Clever use of the stable manifold provides robust stability and successful therapy.

Intermittent androgen suppression (IAS) therapy is a class of hormonal treatment for prostate cancer, in which a drug-induced androgen deprivation can reduce the population of prostate cancer cells. In IAS therapy, drugs are administrated only in on-treatment periods that are separated intermittently by off-treatment periods. The presence of off-treatment periods may be beneficial for maintaining the sensitivity of the tumor cells to androgen deprivation. Thus, IAS can be superior to continuous androgen suppression (CAS) for delaying or possibly preventing relapse of a tumor. IAS therapy usually monitors the level of serum prostate-specific antigen (PSA), which is related to the population of tumor cells. Each on-treatment period begins when the PSA level is greater than an upper threshold; treatment results in a decrease in the PSA level. The on-treatment period is suspended when the PSA level falls below a lower threshold; the PSA level then rises again until the beginning of the next on-treatment period. To determine the transitions between on- and off-treatment periods, we propose a new IAS protocol that uses a model-based estimate of the state point in the phase space of the tumor dynamics. We show that the proposed protocol is effective if, in each of the on- and off-treatment periods, the tumor dynamics exhibits a saddle-point instability accompanied by a stable manifold. Mathematical analysis reveals that tumor dynamics can be controlled in a more effective and robust manner with the proposed protocol than with conventional IAS. We also discuss the clinical feasibility of the proposed protocol as an alternative to conventional IAS therapy.