Towards a trusted HDFS storage platform: Mitigating threats to Hadoop infrastructures using hardware-accelerated encryption with TPM-rooted key protection

As a follow-on to the authors' previous work, this paper further expands on the concept of creating a trusted Apache Hadoop Distributed File System (HDFS). We discuss our motivation and evaluate a threat model for HDFS, and address a set of common security concerns within HDFS through infrastructure and software involving data-at-rest encryption and integrity validation. To accomplish these goals, we make use of technology from the Trusted Computing Group, such as the pervasively available Trusted Platform Module. In addition, we discuss our design considerations in building an encryption framework for Hadoop in a trustworthy manner, and describe the results of our experiments creating an encryption scheme for Hadoop utilizing hardware key protections and AES-NI for encryption acceleration. As part of this design we evaluate the recently implemented crypto framework for Hadoop and independently test the performance claims of AES-NI regarding mitigating performance overhead.