Application-based anomaly intrusion detection with dynamic information flow analysis

This paper presents a new approach to detecting software security failures, whose primary goal is facilitating identification and repair of security vulnerabilities rather than permitting online response to attacks. The approach is based on online capture of executions and offline execution replay, profiling, and analysis. It employs fine-grained dynamic information flow analysis in conjunction with anomaly detection. This approach, which we call information flow anomaly detection, is capable of detecting a variety of security failures, including both ones that involve violations of confidentiality or integrity requirements and ones that do not. A prototype tool called DynFlow implementing the approach has been developed for use with Java byte code programs. To illustrate the potential of the approach, it is applied to detect security failures of four open source systems. Also, its effectiveness is compared to the effectiveness of an approach to anomaly detection that is based on analyzing method call stacks.