Process control using finite Markov chains with iterative clustering

•Finite state MDP provide a viable alternative for process control.•An iterative re-discretization is proposed based on clustering of closed-loop data.•An efficient hierarchical modification of the k-means clustering is proposed.•CSTR process control benchmark requirements are met by the suggested control design.

Finite state Markov Decision Processes (MDP) for process control are considered. MDP provide robust tools to perform optimization in closed-loop, and their finite state description enables an easy implementation of Bayesian state estimation. An approach to tackle the curse of dimensionality problem, yet retaining the benefits of the finite state MDP in control and estimator design, is proposed. The suggested approach uses iterative re-discretization based on clustering of closed-loop data. An efficient modification of the k-means clustering technique is proposed. The performance of the approach is demonstrated using a challenging benchmark from chemical engineering, the van der Vusse continuous stirred tank reactor control problem. It is shown that the requirements of the benchmark are met, and that the suggested iterated clustering significantly improves the performance. It is concluded that the finite state MDP approach is a viable alternative for small-to-medium scale problems of practical process control and state estimation.