Logging kernel events on clusters

We present tools for recording and analysing kernel events on Linux clusters. The tools provide cluster-wide event logging at system clock accuracy. We demonstrate the usefulness of our tools by verifying an implementation of a simple remote scheduling feature with interesting results regarding schedule responsiveness, by analysing local clock drift in a cluster, and by observing the effect of a kernel read-ahead policy upon sequential file read. With our GUI-based Java application, data recorded on multiple hosts is integrated for visualization. These tools can be used for analysis of cluster schedulers such as gang schedulers, SMP scheduling affinity, or the interaction between user level applications and kernel policies such as file read-ahead. Our work can be easily extended for integration with application level logging.