A Winning Approach to Boss of the SOC (BOTS)

At .conf 2024 Deepwatch again had a top ten team for the Boss of the SOC (BOTS) contest. Deepwatch participates amongst hundreds of competitors, and has had a top ten team for three years running. Started in 2016, BOTS allows Splunk users to show off their security operations skill set in teams of four, focusing on helping identify security breaches at the fictitious “Frothy Brewing Company” by reviewing logs utilizing Splunk tools and capabilities. This “capture the flag” exercise is open to anyone who registers a team and is participating onsite at a BOTS event.

Recently, Deepwatch gathered members of this top-tier team to discuss the BOTS competition and share insights and thoughts for anyone interested in BOTS. What follows is an abridged version of that conversation. The original conversation can be viewed in its entirety via the accompanying video.

Deepwatch Marketing: Alan, what are you allowed to bring in terms of Deepwatch capabilities and Deepwatch technologies to help you with the BOTS activity? Or are you really just kind of stuck with a clean interface to Splunk, and whatever you can pack between your 2 ears?

Alan Reichner: You can bring outside tools on your laptop, and you can bring outside Splunk queries, things like that. In fact, you will need outside tools for some of the questions, not all the answers are actually found in Splunk itself. Sometimes you have to take data from there and then use an outside tool to gather information that wasn’t necessarily available in the logs itself. But you can’t really install Splunk apps or anything like that. You’re kind of limited to what they give you on these Splunk instances or their other products that they’re having you work through. You definitely have to bring a lot of knowledge as well.

Deepwatch Marketing: Tyler, Alan mentioned some outside tools. Give me an example of what he might have been referring to.

Tyler Wright: Last year, one of the challenges that we faced was that we needed to go to a GitHub repo and actually look at the content in a GitHub repo. We’re trying to look at the problem within Splunk logs, which are fantastic to work with, but not as great as just going to the repo itself for a better understanding of what was happening and trying to identify the answer.

Deepwatch Marketing: Kyle, Deepwatch has participated in BOTS for a while. Can you give us a quick history on Deepwatch’s BOTS participation over the years?

Kyle Shaffer: Deepwatch employes have participated in the BOTS competition since as early as 2019, but 2021 is when we formed the first official Deepwatch BOTS team which placed in top 10, and then for every year after we place in top 10 since, sending a Deepwatch official team to Boss of the SOC to compete. We’ve sent one or two teams some years, and we’ve had different people cycle out between the teams just based on the changes in the cybersecurity environment, particular skill sets, and availability. Since 2021, it’s been in person only in Las Vegas and that does limit some people who aren’t able to travel.

Deepwatch Marketing: So, Kyle, as somebody who’s been on more than one of these teams, what would you say it takes to be a successful team at a BOTS competition?

Kyle: Communication is definitely a big portion of it. Understanding where people’s strengths and weaknesses are and where to put people in specific modules. Each module ranges wildly in terms of different specialties, and so assigning the right individuals to specific modules allows you to accomplish them quickly. So, understanding who’s on your team and kind of where their strong suits are, and putting them in the right spot is definitely one portion of it. That allows you to succeed. Then it’s a time based competition. So the faster that you can knock things out the better. You can’t do everything alone. And you gotta know when to, you know, rely on your teammates.

Deepwatch Marketing: How would you say BOTS tests your Splunk capabilities? How does it actually test what you know how to do within Splunk?

Tyler: One of the fun things about BOTS is they do trial and test out some of their new tools and capabilities. So while you do have the typical activities to do in Splunk, you do get access to some of their other product offerings. This year, they had a tack analyzer so we could submit samples and see what those reports would look like. You get to see, access, and play around with tools that you don’t normally get to play with. I think if you’re a typical analyst and you’re working in Splunk ES you’re set for the traditional ES portions. But it’s really gonna test your abilities to figure things out on the fly. You know, it’s not always as difficult as you make it out to be.

Alan: They introduce tools that you may not have seen before, because they’re brand new, so this year they did that and we really had to start getting into it and looking at some of these new tools that they have, and really investigating things. Also, provides a great learning experience as well because of that.

Deepwatch Marketing: How does your experience at Deepwatch enable you to achieve that top 10 finish at BOTS?  

Tyler: First and foremost, I always consider myself an analyst – think you’re always an analyst at heart, at least in the cybersecurity space. My day-to-day job is building out detections and fine tuning our detections. So it’s really neat to see what tricks the adversary is using, trying to bypass techniques and detection capabilities. What should I be looking out for when I’m building my detections to enable the analyst to do their jobs as effectively as possible?

Kyle: As Director of SOC operations, I would also consider myself to be an analyst as well. It is definitely a core competency of mine to make sure that the analysts on my team are able to perform analysis. Quick, effective, and accurate, and that’s the kind of skill set that’s brought to this competition that allows us to hit Top 10. Speed is a big factor in this, and the ability to walk through the analysis section and quickly identify a key host. And here’s the alerts that fired for this host, the notables and pivoting each one is a huge portion of that, and then being able to effectively search in Splunk environments. You have to be able to craft effective searches, to find those logs even in the day-to-day here at Deepwatch, when time is a factor, just like it is for the BOTS competition. If you just do “index equals star” in the BOTS competition, you just sit there and wait and wait and wait. You’re going to have to know what you’re looking for. You are 10,000 points behind the rest of the people who knew exactly where they were going to look in the exact index and then the exact source type. And then what you’re looking for. Knowing what tools and technologies do, what kinds of logs they generate, when you’re looking in logs is important. Let’s say, for example, logs are coming from a Cisco ASA. If you don’t know what that is, as a source type, you have to search for that on the Internet. That’s time spent. We at Deepwatch work with these source types day in and day out, and can quickly identify them from experience. I know what all those source types are. I know what they do. I need to look inside this one. It’s a quick and easy pivot. So those are a few things that the day-to-day at Deepwatch brings over to the skill set here.

Alan: I’m a detection platform architect here at Deepwatch, and I spend a lot of time investigating new ways to enhance our detection platform. And so I spend a lot of time looking at tools that Splunk provides that can help us leverage our detection platform. It’s really great when you go into BOTS, and you start seeing some of these tools that you may have looked at before starting to need to understand them. You need to know how to search things. It’s really important in Splunk, knowing the right searches, they also limit you on certain searches. They don’t want you doing really inefficient searches, because lots of people are using these environments. So you have to do searches that are efficient, but also find the answers you need fast.

Deepwatch Marketing: How does BOTS compare to the work you do every day at Deepwatch?

Alan: You have to look at a lot of logs in BOTS. They provide a lot of data across multiple platforms. And that’s something I do a lot. I’m constantly looking at data that we have from our detections and from our detection platform to try to understand things as well as investigate things. When you start looking at some of the other tools like attack analyzer or SOAR and you really have to dig into what these playbooks are doing, what they’re grabbing. And also some of the code that is being executed on the SOAR platform. As an example, during this BOTS competition I worked on the SOAR platform, and they had me dig into some of the python code that they have, and change things a little bit to see what happens when I do it. And so I really had to take some of my day-to-day knowledge that I gathered at Deepwatch and really use that for BOTS.

Tyler: I think working on our detections and building out our detections provides a unique experience and opportunity. So, when I’m building out detections for Deepwatch I’m trying to make effective searches that aren’t going to be CPU intensive on our customer search heads and they provide the analyst exactly what they need to know. I take some of that knowledge and experience when I’m working, you know, in a BOTS competition to make sure that I’m searching effectively, providing the answers as quickly and effectively as possible.

Kyle: One thing I do everyday at Deepwatch is I try to optimize the analyst experience, to make the “time the value” as short as possible. How fast can analysts get information they need, and then make a concise decision on that to be able to give value to customers? So, that’s one of the big pieces, and that directly relates to BOTS, because every year the four of us are going to sit back down and talk about how we’re going to optimize this experience to make the next year even more effective, as soon as we are done with the competition. I was probably up until 3 o’clock in the morning, just going over everything that we could have done as a hyper-efficiency change and when we go back into it for the next year it’ll be even more effective. And then we’re also able to take those lessons learned from BOTS and take them directly into Deepwatch to just have a better experience for our customers and the analysts. There’s an entire section this year, specifically, just on cloud computing logs, an entire section. A lot of things were learned from that section from one single instance of a ton of cloud computing logs that we can then take and use to enrich the experience for Deepwatch analysts as a whole and detections and everything else.

Deepwatch Marketing: What tips or tricks would you share with others who want to participate in BOTS?

Kyle: The faster you answer questions the faster you get points in Boss of the SOC. Every minute the questions go down by a percentage of their value. So the quicker you’re able to score points, the quicker you get the full value for points. Don’t be scared of using hints or waiting for hints. Did you try googling it? Third, of course, is assigning the right modules to people you think will be most successful in those. Fourth tip is don’t over complicate the answer. Cybersecurity isn’t always that complicated. And we all should know this. Attackers use the path of least resistance often in order to maneuver, and oftentimes those are the easiest answers as well. If you’ve never done it before – and everyone should do a Boss of the SOC competition – even if you don’t think that you’re going to win, you shouldn’t go in there to win, you should go there to experience. Everyone should have an opportunity to try and compete in this, because it’s just really eye opening. You know what it looks like to walk through an entire attack. 

Tyler: Just have fun, it’s a great learning opportunity. Lots to gain from it. So just have fun, don’t go in expecting to win. As the competition progresses we range all over in the top 10 for a while. It’s not until probably the last, you know 30-45 min that you start seeing where everybody shakes out.

Alan: Read the questions closely. They are not multiple choice questions. If they say it’s going to be a number, they will provide you the rounding places that they want that number to be rounded to. Don’t feel like you have to go here to compete at BOTS. I met numerous first-time analysts there at the BOTS competition, who are actually just there to learn, and afterwards they may not have gotten the best score, but they got a lot of information. They got a lot of great experience that they could definitely use in future BOTS competitions, as well as in their careers as analysts.

Deepwatch encourages anyone who uses Splunk for cybersecurity operations and who wants to improve or challenge themselves to consider participating in a BOTS competition. We hope this interview has provided you insights for – and interest in – the BOTS competition, and insight into why the Deepwatch team is a consistent top-ten finisher every year.

↑