Convergence of controlled models and finite-state approximation for discounted continuous-time Markov decision processes with constraints

• Convergence of the optimal values for discounted constrained continuous-time Markov decision processes (CTMDP).
• Convergence of optimal policies for discounted constrained CTMDP.
• Finite-state approximation to countable-state discounted constrained CTMDP.
• Applied examples and convergence rates.

In this paper we consider the convergence of a sequence {Mn}{Mn} of the models of discounted continuous-time constrained   Markov decision processes (MDP) to the “limit” one, denoted by M∞M∞. For the models with denumerable states and unbounded transition rates, under reasonably mild conditions we prove that the (constrained) optimal policies and the optimal values of {Mn}{Mn} converge to those of M∞M∞, respectively, using a technique of occupation measures. As an application of the convergence result developed here, we show that an optimal policy and the optimal value for countable-state continuous-time MDP can be approximated by those of finite-state continuous-time MDP. Finally, we further illustrate such finite-state approximation by solving numerically a controlled birth-and-death system and also give the corresponding error bound of the approximation.