Numerical Methods for Optimal Controls for Nonlinear Stochastic Systems With Delays, and Applications to Internet Regulation

The Markov chain approximation method, a primary approach for computing optimal values and controls for stochastic systems, was extended to nonlinear diffusions with delays in a recent book. The convergence of many forms of algorithms was proved. The path, control and/or reflection terms can all be delayed. Reflection terms occur in communications models, where they correspond to buffer overflows. If the control and/or reflection terms are delayed, the memory requirements can make the problem intractable. Recasting the problem in terms of a “wave equation” yields practical algorithms with much reduced computational needs. We outline the approach, concentrating on forms motivated by applications to communications, and give data illustrating the potential.