First Passage Problems for Nonstationary Discrete-Time Stochastic Control Systems

This paper concerns first passage problems for nonstationary nonlinear discrete-time stochastic control systems. The state and control spaces are general Borel spaces, the transition probabilities are nonstationary, the costs/rewards are time-varying and may be unbounded. The optimal control problem is to minimize the expected discounted costs incurred until the first passage time to some target set, in which the discount factors are allowed to be both time- and state-dependent. Under reasonably mild conditions we establish the so-called first passage optimality equations, and prove that the optimal cost/reward functions satisfy the optimality equations. Furthermore, from the optimality equations we show the existence of optimal Markov policies.