Skip Navigation
Request Demo
 
 
 
behind the scenes

Methodology

Since 2013, Red Canary has delivered high-quality threat detection to organizations of all sizes. Our platform collects as much as a petabyte of security telemetry every day and leverages a library of roughly 3,000 detection analytics to surface potential threats that are analyzed by our Cyber Incident Response Team (CIRT). Confirmed threats are tied to corresponding MITRE ATT&CK® techniques and specific threats to help our customers clearly understand what is happening in their environments. A significant portion of this report is a summary of confirmed threats derived from this data.
Creating metrics around techniques and threats is a challenge for any organization. To help you better understand the data behind this report and to guide you as you create your own metrics, we wanted to share some details about our methodology.

Behind the data

1PB telemetry per day 14M investigative leads 33k confirmed threats 1 Threat Detection Report

To understand our data, you need to understand how we detect malicious and suspicious behavior in the first place. We gather telemetry from our customers’ environments and feed it through a constantly evolving library of detection analytics. Our detection analytics are mapped to one or more ATT&CK techniques and sub-techniques, as appropriate. When telemetry matches the logic in one of our detection analytics, an event is generated for review by our detection engineers.

When a detection engineer determines that one or more events for a specific endpoint surpasses the threshold of suspicious or malicious behavior, they create a confirmed threat documenting the activity on that endpoint. These confirmed threats inherit the ATT&CK techniques that were mapped to the analytics that first alerted us to the malicious or suspicious behaviors.

It’s important to understand that the techniques and sub-techniques we’re counting are based on our analytics—and not on the individual review performed by our detection engineers, during which they add more context into detections. We’ve chosen this approach to maximize efficiency and consistency. However, the limitation of this approach is that context gleaned during the investigation of a threat does not inform its technique mapping, and by extension, some small percentage of threats may be mapped incorrectly or incompletely. That said, we continually review these confirmed threats, and we do not believe that there are a significant number of mapping errors in our dataset.

How do you count?

You may be wondering how we tally the scores for the Threat Detection Report. Our methodology for counting technique prevalence has largely remained consistent since our first Threat Detection Report in 2019. For each malicious or suspicious detection we published during the year, we incremented the count for each technique reflected by a detection analytic that contributed to that detection (excluding data from detections of unwanted software). If that detection was remediated and the host was reinfected at a later date, a new detection would be created, thus incrementing the counts again. While this method of counting tends to overemphasize techniques that get reused across multiple hosts in a single environment (such as when a laterally moving adversary generates multiple detections within a single environment), this gives appropriate weight to the techniques you are most likely to encounter as a defender.

For the purposes of this report, we decided to set our rankings based on techniques, even though the majority of our analysis and detection guidance will be based on sub-techniques. This seemed to be the most reasonable approach, considering the following:

  • Sometimes we map to a technique that doesn’t have sub-techniques
  • Sometimes we map to sub-techniques
  • Sometimes we map generally to a technique but not to its sub-techniques

In cases where a parent technique has no subs or subs that we don’t map to, we will analyze the parent technique on its own and provide appropriate detection guidance. However, in cases where sub-technique detections are rampant for a given parent technique, we will focus our analysis and detection guidance entirely on sub-techniques that meet our requirements for minimum detection volume. To that point, we decided to analyze sub-techniques that represented at least 20 percent of the total detection volume for a given technique. If no sub-technique reached the 20 percent mark, then we analyzed the parent.

What about threats?

Our Intelligence team seeks to provide additional context about threats to help improve decision-making. By understanding what threats are present in a detection, customers can better understand how they should respond. Throughout 2021, the Intelligence team sought to improve how we identified and associated threats in detections. We chose to define “threats” broadly as malware, tools, threat groups, or activity clusters. We took two main approaches to associating a detection to a threat: automatically associating them based on patterns identified for each specific threat and manually associating them based on analysts’ assessments conducted while reviewing each detection.

In contrast to our technique methodology, we counted threats by the unique environments affected. Whereas for techniques we counted multiple detections within the same customer environment as distinct tallies, for threats we decided to only count by the number of customers who encountered that threat during 2021. This is due to the heavy skew introduced by incident response engagements for laterally moving threats that affect nearly every endpoint in an environment (think ransomware).

Had we counted threats by individual detections, ransomware—and the laterally moving threats that lead up to it (e.g., Cobalt Strike)—would have been disproportionately represented in our data. We believe our approach to counting gives an appropriate measure of how likely each threat is to affect any given organization, absent more specific threat modeling details for that organization. It also serves as a check against the acknowledged bias in the way we count technique prevalence.

Limitations

There are a few limitations to our methodology for counting threats, as there are for any approach. Due to the nature of our visibility (i.e., that we predominantly leverage endpoint detection and response data), our perspective tends to weigh more heavily on threats that made it through the external defenses—such as email and firewall gateways—and were able to gain some level of execution on victim machines. As such, our results are likely different than what you may see from other vendors focused more on network or email-based detection.

While the top threats are worth focusing on, they are not the only threats to consider, since other impactful ones may be unidentified and therefore underreported. The analysis and detection guidance in this report is reflective of the overall landscape, and, if implemented, offers a great deal of defense-in-depth against the threats that most organizations are likely to encounter.

Knowing the limitations of any methodology is important as you determine what threats your team should focus on. While we hope our top 10 threats and detection opportunities help you and your team prioritize, we recommend building your own threat model by comparing the top threats we share in our report with what other teams publish and what you observe in your own environment.

 
Back to Top