ATT&CK™ Is Only as Good as Its Implementation: Avoiding Five Common Pitfalls

MITRE’s ATT&CK framework has been a tremendously useful resource for security teams seeking to identify coverage gaps, increase visibility, detect new threats, and bolster their organization’s security. However, implementing ATT&CK into a security program is not a trivial undertaking. As we mapped Red Canary’s detection and response platform to ATT&CK over the last year-and-a-half or so, we encountered (and overcame) a variety of pitfalls. I walked through these in a presentation at MITRE’s first annual ATT&CKcon and wanted to share them for those who couldn’t attend.

Here are five pitfalls to watch out for and strategies for avoiding them.

#1 Don’t Assume All Techniques Are Equal

TL;DR: Some ATT&CK techniques are specific and others are generic, so focus on what’s specific first, then increase your scope from there.

Problem

The size and scope of each ATT&CK technique varies dramatically. Some techniques are very specific. There are a limited number of ways an adversary can apply or implement them. Others are incredibly broad. PowerShell, for example, is just a programming language. You can do a lot of things with it, and it’s applicable to nearly every tactic in the matrix. When we started mapping our detectors to ATT&CK, a massive number of them fell under both PowerShell and other techniques.

The matrix is loaded with broad techniques like this. File deletion is another example. If you’re responding to a destructive attack that used this technique, you’re going to want to know what was deleted. However, you probably don’t want to create alerts for every time a file gets deleted as a normal function of the operating system. Bottom line: it’s critical to understand that not everything in the ATT&CK matrix lends itself to detection or testability.

Solution

To overcome this pitfall, we recommend that teams focus on the most specific techniques first. Focus on the techniques for which you can readily create and implement reliable detections before moving onto the broader techniques.

Our detection engineering team approached ATT&CK techniques in the context of detection, measurement, and coverage. With PowerShell, for instance, we first focused on download cradles, then on obfuscation, isolating the constituent pieces of the broader techniques, and addressing them one at a time. The key was to determine what adversaries were doing with the broader technique and how that behavior revealed itself in the data sources on an organization’s network. We found this was far more effective than attempting to run through the matrix and map out detections left-to-right or at random.

#2 Don’t Try Building Alerts for Every Technique

TL;DR: You don’t need to alert on every technique in the matrix, so focus on those techniques that are more readily detectable before moving on to the more complex ones.

Problem

It can be easy to fall into this this trap if you’re constantly trying to assess whether you can detect this technique or that. However, you can’t simply funnel the entire matrix into alerts because not every technique has to be an alert. Jack Crook, an incident responder from GE, talks about looking at things in clusters of detection techniques. If you group some of the broader techniques together into a cluster, then you might be able to generate an alert for them.

One approach is to organize the techniques into buckets. First, there are alertable techniques that you can build readily tunable alerts around. Then there is another bucket filled with things that are contextually or forensically important—but maybe not great for alerting.

Techniques like file deletion are very important from a contextual perspective in disruptive attacks. Directory discovery is useful for attackers that are enumerating your network. You definitely want your analyst to know about these things, but you probably don’t want to alert on these techniques because there are legitimate scripts that run across hundreds of thousands of systems that do these sorts of things every day (or even every couple hours). If you build detections for these sorts of techniques, then you’ll overwhelm your analysts with largely meaningless alerts.

Solution

Focus on providing high fidelity detections with the very specific techniques while working with the less specific techniques to provide enrichment for your analysts. In this way, everyone can spend less time churning on alerts and more time building more complex detections to catch the more evasive adversarial techniques.

For example, if you are working on creating detection coverage for MSBuild (“T1127″), you want to know everything about how MSBuild is used benignly and maliciously. If you truly understand what is normal (and probably benign) for MSBuild and what is anomalous (and potentially malicious), then you extract the anomalous behaviors and turn them into detectors.

#3 Don’t Misunderstand Your Coverage

TL;DR: Each technique contains boundless possibilities, so measure the efficacy of the techniques you can detect, not the unknown.

Problem

We get a lot of questions about what we can and can’t cover. The problem with answering this question is that with many techniques, the possibilities are boundless.

For example, you might build a coverage map where little green squares ostensibly illustrate that you have coverage for a given technique. But as you examine each technique, you realize there is a seemingly infinite universe of possibilities or variations within many techniques in the matrix. You might think you have a technique fully covered, but then some researcher will publish something on Twitter, and you’ll have to go and add to an existing detection.

At Red Canary, we measure every detector that we’ve ever implemented, gathering metrics on whether they’re generating true positives or false positives, what category of detection they’re generating detections for, and how much time analysts are spending on events for that specific detector. It helps us to track the performance of our detectors and determine which are causing analysis problems for our Cyber Incident Response Team (CIRT). It ultimately allows us to decrease the amount of time our analysts spend on each event, but it also offers a dynamic way of looking at coverage.

Solution

Measure your detectors and understand that ATT&CK techniques aren’t static. Don’t waste time trying to measure the unknowns within each technique in an effort to define your coverage, but keep your shareholders up-to-date with how your coverage improves as you add new detections. Explain how you closed one part of the gap and are working next on a related gap. Drive home the fact that you’re continuously improving. At the end of the day, that’s really what the goal is: to continually improve your security.

#4 Don’t Stay in the Matrix

TL;DR: Adversaries move faster than models, so you have to be proactive about finding ways to detect emerging threats.

Problem

It’s easy to become myopic and focus exclusively on the techniques in the matrix. ATT&CK is great, but ATT&CK is not all inclusive. It’s certainly very comprehensive, but there are definitely things that fall outside of ATT&CK—some of which will be added to the matrix as it grows. If you stay in the matrix all the time, then you’ll fall behind because adversaries are innovative, move quickly, and are constantly developing new techniques or new ways of leveraging existing ones. Models and standards, on the other hand, take time to update.

Solution

Remember to keep an eye on what is going on outside of ATT&CK. An emerging technique may pose a more critical risk to your business than anything in the matrix. Stay up-to-date with the latest research so you can develop coverage for techniques that aren’t yet included in ATT&CK.

#5 Don’t Forget the Fundamentals

TL;DR: ATT&CK is a great repository for adversarial behaviors, but you have to be careful not to lose track of fundamental security concepts like security awareness training, vulnerability management, and the principle of least privilege.

Problem

As our team spent the better part of two years mapping to and operationalizing ATT&CK, we found ourselves getting sucked into focusing exclusively on ATT&CK and on detection and response. However, security is much broader than that. There are many important fundamentals that exist outside the scope of ATT&CK. You can have the best ATT&CK coverage on the planet but suffer a breach because of poor IT hygiene. Alternatively, maybe your security team is great at finding evil, but lacks a clear incident response plan to guide them on responding to evil.

Solution

Leverage ATT&CK where it makes sense, but continue to also work on those other security controls that are important to you and your organization. Segment your network, limit host-to-host communications, maintain software and system inventories, install patches, control user permissions, and have a tested incident response plan. These types of fundamentals will always have a significant impact on security.

Conclusion

We are big (HUGE!) fans of ATT&CK at Red Canary, and we highly encourage security teams to use it as a resource to improve their organization’s security posture. Just remember not to lose yourself in the matrix or become disproportionately focused on the impossible or the insignificant. Hopefully these tips can help you stay focused on delivering better security outcomes without getting lost in any ATT&CK technique rabbit holes.