On early detection of application-level resource exhaustion and starvation

Software systems are often engineered and tested for functionality under normal rather than worst-case conditions. This makes the systems vulnerable to denial-of-service attacks, where attackers engineer conditions that result in overconsumption of resources or starvation and stalling of execution. While the security community is well familiar with volumetric resource exhaustion attacks at the network and transport layers, application-specific attacks pose a challenging threat. In this paper, we present Radmin, a novel system for early detection of application-level resource exhaustion and starvation attacks. Radmin works directly on compiled binaries. It learns and executes multiple probabilistic finite automata from benign runs of target programs. Radmin confines the resource usage of target programs to the learned automata and detects resource usage anomalies at their early stages. We demonstrate the effectiveness of Radmin by testing it using a variety of synthetic and in-the-wild attacks. We provide a theoretical analysis of the attacker's knowledge of Radmin and provide a metric for the degree of vulnerability of a program that is protected by Radmin. Finally, we also compare the accuracy and effectiveness of two different architectures, Radmin which works in both the user and kernel spaces, and URadmin which works solely in user space.