Automate the pain

DM For Hints

Sunday, August 2 2020

The year is 2020. We have all become avatars on virtual display. Conferences that required budget for travel, housing, and coordination have since become available online for all to participate. A great deal of those conferences have canceled with more created in their place. I refer to this as the Year of the Virtual Conference.

Rather than burning out in both work calls and virtual conference planning and participation, I took the approach of viewing only the conferences of interest, and dedicating my time for continued learning. This had influenced me to becoming a Capture The Flag (CTF) challenge creator. A CTF requires a different approach to coordination, planning, scheduling, and testing than other types of events. I am a novice when it comes to playing in CTFs, however, I found that I perform exceptionally well when challenges are system’s administration, operations-focused, and I have found that I also have a particular knack for the steganography, and cryptology.

Kreator

Creating the challenges themselves required more effort than I had originally anticipated. The research and planning involved ranged from having challenges that were created in an hour, and other challenges that required many hours more. I prefer to theme my challenges on real-world scenarios. Learning the history, usage, and the techniques to make the experience more productive for the time invested, while also being able to share that fun-facts learned with the players. The types of steganography challenges for NahamCon CTF were pre-selected before I joined the team, though I was able to theme and ultimately create them. For HackerOne’s H@acktivityCon’s CTF, I had complete ownership of the challenges and was able to plan, design, and create at will.

I knew that I did not want my challenges to be solved quickly with automated tools or online solutions. I had designed my challenges to include subtle hints, depending on the challenge difficulty, to be indicated in the challenge descriptions. The easier challenges were as simple as finding an article of an outfit to identify the technique that they used, or being given a key to a known an older known cipher. Some challenges were in combinations with a multi-step approach to retrieving the flag. How I selected the scenarios of the challenges I found to be the most interesting. The ideas came from joint-reports released by Government agencies that had identified active and ongoing campaigns that were communicating publicly, yet discreetly, through social media or other means. I also had the guidance of two individuals who specialize in cryptanalysis. I found that some users were not as interested in learning the history or usage of techniques as I was, and felt the challenges required more guessing than research.

The Day of the Za

In addition to dedicating time to creating the challenges, there is also quite a bit of coordination involved for the other aspects of hosting a CTF. The majority of the team is centrally located in the DC area, while other team members are located in other time zones. Those of us in the DC area would plan to meet each Friday afternoon to enjoy the za and get to work. We began to call this DEVDAY.

After NahamCon CTF, we had already established which roles we were going to be responsible for. These roles are challenge creation, infrastructure management, and hype. We knew that John and Kaitlyn were not going to be available during live, and he needed to focus on coordination with HackerOne, creating challenges, and being the hype man. The rest of us focused on creating challenges, designing infrastructure, or a combination of both. When the CTF is live we all have the responsibility of being available for the users while managing the infrastructure.

As the go-live date nears, we begin the infrastructure build, testing, and any design considerations in addition to the existing architecture. Much of the architecture be viewed from a talk previously given. The preferred Cloud Service Provider (CSP) by John is DigitalOcean, and was used for the two previous CTFs. However, with H@cktivityCon, the CSP used was the Google Cloud Platform (GCP) that was provided by HackerOne. Configuration of new baselines for the build scripts were performed by Caleb with few configuration considerations from John and I. Caleb also handle the rest of build-and-deploy while the rest of us configured the Dockerfile or yml configurations of our own challenges.

cgroups, martial scripts, and ulimit

Already having two CTFs under his belt, John knew that he wanted to continue his standardized approach to using Docker, and this time with the infrastructure to be cloud-agnostic. I had shared with him the capabilities I had architected into Government modernization efforts. These capabilities were agnostic solutions, automated provisioning, automated implementation with configuration hardening, and self-healing (corrective changes). Caleb had taken on a lot of this effort to build and configure the infrastructure for the H@cktivityCon CTF. His efforts reduced the numbers needed to manage infrastructure from four to two people by developing a Challenge Server Management plugin that automatically distributes actions across all challenge servers with parallel-ssh (or pssh) to install, run, update, or manage the challenges themselves. He also developed a CTF Monitor plugin that monitors the health and takes automatic action on performance of the challenges themselves all while providing to us metrics and notification of any failures. Together, we were able to quickly check the resource utilization of the challenge servers and quickly update / restart any docker containers that where live changes needed to be made. The CTF Monitor had a large role in providing the self-healing aspects by controlling the health of the containers and their challenges. Having limited visibility into HackerOne’s GCP for load balance and other infrastructure monitoring, we were able to supplement by having these customized capabilities in place.

Despite our best efforts with all those capabilities in place, we experienced an on-going :(){ :|:& };: attack Thursday beginning 9:12 EDT. With system resources constrained the automated challenge announce-and-release was unable to execute at its scheduled time. The attack was cleverly annoying. I had taken to reviewing the logs on on targeted challenge servers to understand what was happening while Caleb took the action of ensuring challenge availability for the users. We had 10 challenge servers available for balancing the user experience. We discovered that, after restarting a hard-locked server, that the attack would immediately start again thus determining the attack to be somewhat automated. We learned that the containers that experienced this issue either did not have nsjail isolation, or lacked user-specific least privilege configuration from the Dockerfile. We also learned from the IoT Village staff that they were experiencing the same attack allowing us to begin cross-referencing user information between the villages to take action. This action, along with updating the configurations to our container images with user limitations, we were able to come back to high availability and provide the anticipated user experience. The total time was nearly two hours, which was unfortunate.

Break It Down

When it came to interacting with the users, I can only imagine what it was like for them to interact with me. My interaction with users was based on their modesty of CTF experience, quality of questions for hints, and their technical capability in general. Users either received a meme with a hint, a meme without a hint, or a meme with snark. The snark grew greatly as the CTF and sleep deprivation carried on.

I have analyzed this information over time and established user categories. These do not reflect all kinds of users, but the ones that have stood out the most. These categories are the Students and First Timer’s, the I know nothing, for I am Jon Snow, the I’ll DM every admin, and The Knowitalls.

The Students and First Timer’s are very easy to work with. They genuinely want to learn for themselves to either gain a competitive edge in their career or to simply learn a new skill. These are the users that arrive with their ego checked and let us know that they don’t want it handed to them. They want the hints and the nudges. The students and First Timer’s know that they need to ask for help when they have learned that they may have found the metaphorical rabbit.

Like the Students and First Timer’s, the Jon Snow’s come with much less technical experience. They may have just learned about the CTF and wanted to try something new, or they are attempting the transition from a clicker to a keybind and felt that this was the approach that they were going to take. These types of users require much less hints about the challenge and more direction on technology available for that type of challenge.

The users that direct message (DM) every admin within one minute are the worst. They just are. They are not to be confused with the other players that do DM multiple administrators because of unavailability. I am referring to the users that want the win easy, early, and often. These types users have caused us to ask our team members if we need to help or ignore a user that messages. The worst.

Then there are the users that there are no pleasing. The user that knows it all. These are the users that register to play with entitlement. What they make up for in not needing hints is the ability to actively tell everyone in the public channel that a challenges is either too easy, hard, “guessy”, or their netcat connection isn’t fast enough, or they aren’t happy with not having 24/7 administrator availability on a free event. The upside is that they don’t need our help.

While I do categorize the users, and we do talk about specific users in our team private channel, we also do our best to care about every user and their experience. We will have both John’s user base in his discord, and the Discord provided by the event coordinator to manage. It is not simple to have an immediate response to incoming messages across two large communities. This presents the challenge of being just as active in the public channels. If Caleb and I have to focus on the infrastructure, then we are forcing the other team members to deplete their spoons much faster than desired.

👁👮👹👷👡👹

Overall, it has been a great being a host than a participant. Having experienced both sides does provide greater insight and appreciation. I have began to see the same players coming back to play our CTF while also meeting new players. There is a hugely talented team that I was invited to work with, and this could not have been as successful from start-to-finish without them.

Appreciate you, team.