Optimality and convergence of adaptive optimal control by reinforcement synthesis

Adaptive Optimal Control (AOC) by reinforcement synthesis is proposed to facilitate the application of optimal control theory in feedback controls. Reinforcement synthesis uses the critic–actor architecture of reinforcement learning to carry out sequential optimization. Optimality conditions for AOC are formulated using the discrete minimum principle. A proof of the convergence conditions for the reinforcement synthesis algorithm is presented. As the final time extends to infinity, the reinforcement synthesis algorithm is equivalent to the Dual Heuristic dynamic Programming (DHP) algorithm, a version of approximate dynamic programming. Thus, formulating DHP with the AOC approach has rigorous proofs of optimality and convergence. The efficacy of AOC by reinforcement synthesis is demonstrated by solving a linear quadratic regulator problem.