Multimodal Fusion, Fission and Virtual Reality Simulation for an Ambient Robotic Intelligence

In this paper, we present an architecture that demonstrates multimodal fusion and fission using semantic agents and web services that interacts with worldwide-web computers, services and users. This solution extracts the meaning of a situation using the semantic memory of agents to manage the interaction process involved. The fusion of values from different sensors produces an event that needs implementation. The fission process suggests a detailed set of actions that are for implementation. Before such actions are implemented by actuators, these actions are first evaluated in a virtual environment which mimics the real-world environment. If no danger arises from such virtual evaluation, then implementation is feasible. Otherwise, there might be a need to add one or more smaller actions to render the action safe and free from danger. Our work presents the following contributions: (i) a design of agent memory and a model of the world environment using a knowledge representation language that is compatible with the current standards, (ii) creation of a pervasive architecture with several scenarios of composition and adaptation, (iii) presentation of how agents and services interact to provide support in a real-world, and (iv) simulation of an event in a virtual environment to assess the feasibility of the event's implementation.