Standing up a modern, well-functioning security operations center (SOC) is not something that typically is—or should be—done overnight, nor all at once. Organizations often start this multi-phase journey by putting in place the core threat detection and response capabilities required, such as threat intelligence and incident handling functions. Then as the organization grows, maturing the security operations is actualized and facilitated by layering in key enablement functions such as security architecture and data management functions.
These are the themes we explored in the first blog and second blog of this three-part series aimed at defining the various functions of a modern, effective SOC, as visualized above. But maintaining a modern SOC calls for consistent testing and regular refinement. In this third and final post in the series, let’s discuss the importance of testing and validation as tools you can leverage to identify opportunities for improvement as well as chart your progress.
Testing & validating security operations
As you implement any system or process, and particularly as it scales, you need to be able to validate the results. In the context of security operations, we engage in a number of activities aimed at achieving some level of validation. Some of these activities are forms of testing performed by internal or external teams. Other activities are focused first on validation—through the review of evidence—often in the form of the various outputs that security operations produce.
This is to say that there are a wide range of approaches to testing and validating security operations. And this article and list are not intended to be exhaustive, but rather, capture some of the most common methods:
- Red teaming
- Penetration testing
- Atomic testing
- Scenarios and exercises
- Compliance
Let’s examine each of these, taking a brief look at their definition and purpose, and how they can be used in concert to provide assurance that your program is functioning as intended.
Red teaming and penetration testing are two of the most common approaches to testing a security operation’s effectiveness. And there are dozens, if not hundreds, of articles that debate the difference between these terms, so we’re not going to settle that here. For our purposes, we’ll treat penetration testing as performing tests that are more limited in time and scope, primarily focused on testing software and controls. And we’ll treat a red team engagement as a test that occurs over time that is broader in scope and more representative of a real-world threat, with a focus on testing overall effectiveness of detection and response. Both are highly functional in nature, meaning that they affect operational systems, and in most cases should result in operational responses. Some of the specific questions that these types of tests answer include:
- Are the systems that provide us with visibility operating continuously and as expected?
- Are the analytics that alert us to suspicious or malicious activity functional? Are they providing value?
- Are we confident in our detection coverage, as a function of visibility and analytics but also taking into account threats?
Red teaming
When thinking about the ways you might test your security operations program, a red team engagement is most closely aligned with operational effectiveness. The red team should be treated like an adversary, which means that a valuable red team engagement is likely to result in a number of observable activities:
- Detection
- Triage and investigation
- Threat identification and classification
- Incident handling and response
A red team engagement should have objectives that reflect the organization’s threat model. If the most valuable thing that your organization produces is the formula for a vaccine, then obtaining this formula might be a worthwhile objective for the red team. Similarly, given that ransomware and extortion are included in just about everyone’s threat model, challenging the red team to execute a “friendly” ransomware attack may be a valuable exercise as well.
The red team is responsible for introducing or being a threat, thus ensuring that your security operations team has some work to do to close gaps and eliminate deficiencies. As a result, it’s a good opportunity to understand some of the most foundational security operations measures (noting that some of these can be calculated when events occur while some may not be known until incident post mortems are complete):
- Time from occurrence to detection
- Time from detection to acknowledgement
- Time from acknowledgement to confirmation of the threat
- Time from threat identification to threat response
- Time from threat response to recovery or remediation
These measures should tell a story about the incident: From the time that the adversary took some observable action, how long did it take to sound the alarm? How about to investigate the threat? Respond to it? And finally, how long did it take to return to normal business operations?
Other valuable learnings exposed by this type of test include the root causes of the intrusion, ranging from initial access to the eventual impact, as well as the techniques that the adversary leveraged along the way. Each of these points is an opportunity to prevent, detect, or disrupt an adversary who chooses a similar path in the future.
Penetration testing
Penetration testing is generally narrower in scope than a red team engagement and can be a more effective means of identifying or validating details like the attack surface of systems and their susceptibility to software or configuration vulnerabilities. In the context of the target system, components, and business operations, penetration tests may help you understand:
- The number and severity of vulnerabilities identified
- The percentage of identified vulnerabilities that were remediated
- The effectiveness of your remediation
Even with a limited scope, however, you hope to see your preventive or detective controls engaged during a penetration test. Activities ranging from attack surface discovery to service enumeration and more in-depth probing of software for vulnerabilities are all opportunities for your technology or your team to identify suspicious or malicious activity.
Atomic testing
Atomic testing is the practice of performing small, narrowly scoped tests aimed at exercising adversary techniques for the purposes of understanding the techniques themselves, the signals that are generated as a result and, ultimately, how we can validate or improve on our ability to detect them. Of the testing methodologies discussed thus far, this one is often the fastest, simplest, and least expensive, and it doesn’t require extensive subject matter expertise.
Getting started with atomic testing is as simple as visiting Atomic Red Team, an open source project that is the product of thousands of community contributions, executing one or more tests, and looking at the results.
Atomic testing can help you test, validate, or implement:
- Visibility, including everything from the presence of events in local logs to exercising more complex data pipelines
- Detection, including whether corresponding analytics or native product alerts fired
- Preventative controls, which you expect to block or otherwise mitigate the activity
- Incident response processes or actions, manual or automated
Because atomic testing is intended to be very approachable and easy, you should be able to perform these tests regularly. One of the simplest approaches may be to use something like this tracking template we recently shared, which you can use to keep track of your tests and how the results change over time.
💡 Simple, measurable @MITREattack testing using Atomic Red Team.
🔢 Pick a technique, execute tests, and track whether you:
1. Observed it
2. Detected it
3. Mitigated it⚛️ Atomic testshttps://t.co/VNAk8jQvOk
📊 Tracking template (ATT&CK v12.1)https://t.co/LTSbCnQjNe pic.twitter.com/s8zXavepjS
— Keith (@kwm) January 13, 2023
Scenarios and exercises
All of the testing methods listed above are forms of scenarios and exercises, but there are many more, and you can be as creative with these as you wish. Your organization’s ability to detect and respond to threats is like a muscle, and the most important thing is that you exercise it regularly. A good exercise takes into account the many moving parts of your team and system and affects them in realistic ways for the purposes of observing, evaluating, and improving response.
Common scenarios include table top exercises, which are discussion-based and involve a facilitator presenting an incident or other emergency to the team, and then talking through and documenting questions asked by the team, decisions the team makes, and actions the team takes.
Table top exercises can be a fun, non-disruptive means of validating that your team has processes for handling an incident and can communicate and make decisions effectively under duress. Further, these exercises provide the opportunity to build and refine playbooks for specific incident types or events.
Purple teaming is another approach to testing that is gaining in popularity, but it isn’t so much a testing methodology as it is a way of working. Rather than offering up our own definition, we’ll refer to this concise definition from Daniel Miessler:
Purple teams exist to ensure and maximize the effectiveness of the Red and Blue teams. They do this by integrating the defensive tactics and controls from the Blue Team with the threats and vulnerabilities found by the Red Team into a single narrative that maximizes both. Ideally Purple shouldn’t be a team at all, but rather a permanent dynamic between Red and Blue.
Purple teaming may be transactional in the sense that you plan an exercise or other event for the express purpose of bringing a red team and blue team together. But aspirationally, your security operations organization should function naturally such that you test your detection and response capabilities regularly, and both the testers and the tested develop lightweight, highly effective systems of measurement and feedback aimed at continuously improving.
Compliance
Let’s face it: Compliance isn’t everyone’s favorite topic. However, I would argue that compliance is invaluable for validating that you have the necessary processes in place and they are being followed. This is usually accomplished in the context of certifications, which provide a framework for the minimum set of policies and systems that you must have in place, coupled with regular audits that look at evidence of use.
Compliance and certifications are, at best, often treated as very abstract concepts and, at worst, as an impetus for getting all the boxes checked. As is the case with most things, you can game the system and get by, or you can put thought and care into your compliance program and get out a lot of value.
A few examples of very practical, operationally critical items that an audit might help to uncover include whether:
- Privileged access to systems or data is properly assigned, reviewed, and revoked
- Data and assets are accounted for properly
- The many applications or services you rely on to do your work are known to your information technology and/or security teams, and whether they are properly configured
- Your incidents are properly classified and documented
- Corrective or preventive actions are identified, adjudicated, and implemented following an incident
If done well, compliance programs foster thoughtfulness and accountability, encourage consistency and repeatability, and ultimately improve the maturity of not only your security operations program but your organization as a whole.
Putting it all together
You can read or re-read the first article in this series here, which covers the core threat detection and response functions of the SOC. The second article in the series covers enablement functions. When taken together, these three articles give a fairly comprehensive look into the different functions of a modern SOC and how these functions all connect and relate.
Lastly, keep in mind that security operations is but one component of effective risk management—a discipline grounded in compromise. This model is intentionally very simple, and thus very flexible. Ultimately, your order of operations—including whether you choose to engage in some of these activities at all—should be informed first by business objectives. Given clear objectives, you can focus intently and effectively on prioritization, implementation, and refinement.
Related Articles
How Red Canary works to create an accessible coding environment for all