Simulating autonomous agents in augmented reality

In many critical applications such as airport operations (for capacity planning), military simulations (for tactical training and planning), and medical simulations (for the planning of medical treatment and surgical operations), it is very useful to conduct simulations within physically accurate and visually realistic settings that are represented by real video imaging sequences. Furthermore, it is important that the simulated entities conduct autonomous actions which are realistic and which follow plans of action or intelligent behavior in reaction to current situations. We describe the research we have conducted to incorporate synthetic objects in a visually realistic manner in video sequences representing a real scene. We also discuss how the synthetic objects can be designed to conduct intelligent behavior within an augmented reality setting. The paper discusses both the computer vision aspects that we have addressed and solved, and the issues related to the insertion of intelligent autonomous objects within an augmented reality simulation.