Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach

In this paper we propose an online Q-learning algorithm to solve the infinite-horizon optimal control problem of a linear time invariant system with completely uncertain/unknown dynamics. We first formulate the Q-function by using the Hamiltonian and the optimal cost. An integral reinforcement learning approach is used to develop an actor/critic approximator structure to estimate the parameters of the Q-function online while also guaranteeing closed-loop asymptotic stability and convergence to the optimal solution.