Stochastic finite-state systems in control theory

This paper is concerned, in the main, with a problem of pursuit–evasion in the context of a stochastic finite-state system. Two cases are considered: (a) non-competitive pursuit in which the target does not try to evade the pursuer; and (b) a competitive case in which the aim of the target is to maximize the time of interception, and that of the pursuer is to minimize it. Employing dynamic programming, it is shown that determination of optimal policies for the target and pursuer reduce to solution of a functional equation involving the expected time of interception vector. Furthermore, it is shown that the functional equation is a contraction mapping. Optimal solution is obtained through iterated contraction. Convergence of iteration is established through the use of the Banach fixed-point theorem.