An Empirical Survey of Functions and Configurations of Open-Source Capture the Flag (CTF) Environments

Capture the Flag (CTF) is a computer security competition that is generally used to give participants experience in securing (virtual) machines and responding to cyber attacks. CTF contests have been getting larger and are receiving many participants every year (e.g., DEFCON, NYU-CSAW). CTF competitions are typically hosted in virtual environments, specifically set up to fulfill the goals and scenarios of the CTF. This article investigates the underlying infrastructures and CTF environments, specifically open-source CTF environments. A systematic review is conducted to assess functionality and game configuration in CTF environments where the source code is available on the web (i.e., open-source software). In particular, from out of 28 CTF platforms, we found 12 open-source CTF environments. As four platforms were not installable for several reasons, we finally examined 8 open-source CTF environments (PicoCTF, FacebookCTF, HackTheArch, WrathCTF, Pedagogic-CTF, RootTheBox, CTFd and Mellivora) regarding their features and functions for hosting CTFs (e.g., scoring, statistics or supported challenge types) and providing game configurations (e.g., multiple flags, points, hint penalities). Surprisingly, while many platforms provide similar base functionality, game configurations between the platforms varied strongly. For example, hint penalty, time frames for solving challenges, limited number of attempts or dependencies between challenges are game options that might be relevant for potential CTF organizers and for choosing a technology. This article contributes to the general understanding of CTF software configurations and technology design and implementation. Potential CTF organizers and participants may use this as a reference for challenge configurations and technology utilization. Based on our analysis, we would like to further review commercial and other platforms in order to establish a golden standard for CTF environments and further contribute to a better understanding of CTF design and development.