What is threat detection and response?

The evolving threat landscape continues to burden businesses, requiring them to develop effective threat detection and response capabilities to counteract opportunistic adversaries leveraging commoditized malware to compromise and monetize whatever systems and data they can. The threat landscape is overwhelming, and determining an organization’s threat model within it is a complex exercise. Even when an organization understands the threats that pose a serious risk to them and the ways their IT infrastructure exposes them to those risks, there is still the monumental challenge of figuring out how to mitigate those risks with an ever-evolving array of security controls. Organizations may choose to build their own security infrastructure, an expensive option made more difficult by the well-established cybersecurity skills gap. Organizations may also choose to pay for services and products to help fortify their security, but the information security product landscape is every bit as complex as the threat landscape.

As you consider your path forward and what makes the most sense for your organization, we hope to provide you greater insight into the concept of threat detection and how your teams can best design, implement, and maintain your systems to gain an advantage over adversaries. In this article, we’ll define threat detection and response—and provide an illustrative example of what it entails. Additionally, we will explain the importance of detection engineering as a critical component of threat detection and response and explore the MITRE ATT&CK® framework, an invaluable resource quantifying detection coverage and a common language for categorizing threats and the adversary techniques they leverage.

Threat detection is a broad concept that describes an organization’s ability to monitor events in its environment and to identify suspicious activity. It encompasses the tools, people, and procedures involved in uncovering threats. Effective threat detection depends on the maturity of your security operations center (SOC) and the tools at your disposal. Historically, prevention technologies like firewalls and antivirus were primarily responsible for protecting organizations against cyber attacks. However, as the IT landscape changed to support distributed, remote workforces that are more reliant on cloud infrastructure and less constrained by traditional network perimeters, threats changed too. And now, effective security demands advanced tools and processes that defend against the growing diversity of threats that target an ever-expanding attack surface.

In response, a number of detection and response tools have emerged, including expansive threat detection and response platforms that can comprise varied detection and response capabilities, such as user and entity behavior analytics (UEBA), threat intelligence, threat hunting, detection engineering, security information and event management (SIEM), incident response, network traffic analysis (NTA), security orchestration, automation, and response (SOAR), and more. Some examples of detection and response platforms include:

• Endpoint detection and response (EDR)
• Network detection and response (NDR)
• Cloud detection and response (CDR)
• Identity threat detection and response
• Extended detection and response (XDR)
• Managed detection and response (MDR)

For illustrative purposes, let’s start with an example of a workflow that covers the entire process:

Effective threat detection starts with data. This data originates from various sources like cloud systems, endpoints, network security tools, mobile devices, and countless other kinds of IT equipment. Oftentimes this data is forwarded to some kind of log or alert aggregation tool like a SIEM. At the risk of oversimplification, the data is generally available in two broad forms: raw security telemetry (logs and events) and pre-processed alerts generated by the tools themselves. You can leverage this data in combination to identify suspicious activity.

Once collected and stored in a centralized location, this data is then analyzed for signs of suspicious or malicious activity. Automated threat detection tools and human experts often perform this analysis in a collaborative manner. The goal is twofold: identify known malicious activity that represents a clear and definite risk to an organization and uncover suspicious behaviors that deviate from normal user or system behavior and may represent a risk to an organization.

When the investigative process identifies a threat, some kind of an alert is generated to inform the affected organization of the threat, and the detection or incident response process kicks off.

Threat response is the set of actions taken to contain, eradicate, and recover from a security incident. It’s often referred to as incident handling or incident response, and it’s the reactive phase that comes after an organization detects a threat.

Once a threat is detected, incident responders or handlers investigate all of the relevant context available to them, typically based on priority, confirming that it’s a true positive (an actual threat) and not a false positive (a harmless event mistaken for a threat). If the investigation confirms the veracity of a threat, incident handlers or responders initiate a series of manual or automated actions to contain and remediate the threat and mitigate any potential future damage. This response might involve:

• isolating affected devices or user accounts
• blocking connections to IP addresses to limit the flow of malicious traffic
• blocking hashes associated with malicious tools or malware
• deploying security patches to address vulnerabilities that may have enabled malicious activity
• removing malware or other malicious artifacts from affected systems
• wiping systems when defenders can’t confidently evict a given threat

The final stage is recovery. This includes restoring normal operations and recovering any lost data. Depending on the severity of the incident, recovery may look like restoring from backups, rebuilding systems, or implementing new security protocols to prevent similar incidents in the future.

It’s important to note that threat detection and response is a cyclical process, thriving on lessons learned from past incidents for continuous improvement. In the next section, we’ll explore the critical role detection engineering plays in this ongoing process.

Detection engineering is the practice of developing analytics or alerts that uncover suspicious or malicious activity, as well as the continuous process of tuning, suppressing, and otherwise nurturing a library of detective controls. Threat intelligence is a critical input for detection engineers to identify and prioritize relevant threats and then design, implement, and maintain automated systems for detecting and contextualizing those threats. Threat research is another key input into any detection engineering program that will provide additional context to threat intelligence, identify emerging threats that are looming on the horizon but may require proactive attention, assess blindspots, and ensure detective coverage is thorough and resilient. The primary output of a detection engineering team is detection analytics.

So what is a detection analytic? Put simply, detection analytics are sets of rules or logic that continuously analyze data and alerts from tools like detection and response platforms, cloud infrastructure, identity providers, and other IT and security tools, looking for specific patterns of malicious or suspicious behavior. Detection analytics parse telemetry and alerts automatically, saving analysts from manually sifting through an endless stream of raw log data and alerts. In turn, detection engineers are constantly tuning detection analytics and suppressing alerts to help reduce false negatives and false positives, improving the fidelity of detections, and providing the actionable insights security teams need to effectively triage, investigate, remediate, and prevent threats in the future.

What does detection engineering look like in practice?

Detecting engineering is cyclical by nature, meaning it’s a continuous process that repeats itself. While the discipline still isn’t well-defined, the cycle usually looks something like this: define detection priorities, assess visibility, collect data, develop detection analytics, monitor events, suppress or tune detection analytics, repeat. This never-ending loop helps ensure detection analytics continue to effectively catch threats, aren’t overly noisy, and expand coverage for unknown or novel threats.

Here’s a more in-depth look at this cycle:

Define detection priorities

Threat intelligence and threat modeling are key prerequisites to defining an organization’s detection priorities. Organizations must be able to identify threats that are likely to target them based on their IT hygiene, the IT systems they use, the types of data they store, and the ways they store and access that data. They must strike a balance between threats that are highly prevalent and opportunistic and threats that are less prevalent, more targeted, and more risky. It’s a constant balancing act, weighing low-impact, high-volume threats against high-impact, low-volume threats.. Frameworks like MITRE ATT&CK (we’ll cover this below) can help security teams quantify and track relevant adversary techniques and perform gap analysis.

Detection prioritization might involve proactively sitting down with your team and strategically assessing threat intelligence to develop detection hypotheses. Or it might be a reactive response to a recently missed detection. Ultimately, the goal here is to determine clear objectives for your detection engineering efforts based on business needs and risk exposure.

The MITRE ATT&CK Framework

As previously mentioned, the MITRE ATT&CK framework is a valuable tool that can help you organize and track your detection objectives. But what exactly is MITRE ATT&CK?

MITRE describes their framework as “a curated knowledge base and model for cyber adversary behavior, reflecting the various phases of an adversary’s lifecycle and the platforms they are known to target.” ATT&CK is open, community-supported, and comprehensive. Critically, it is provided by MITRE as a resource to the community and is not operated or sponsored by any vendor. Anyone is free to leverage it, and everyone is free to contribute as they discover new tactics and techniques.

MITRE ATT&CK categorizes attacker tactics, techniques, and procedures (TTP). It takes what was once a nebulous, ill-defined threat landscape and organizes it into a largely agreed-upon set of TTPs communicated in a common language. By understanding this framework, you can gain a comprehensive understanding of the different ways adversaries might infiltrate, establish persistence, escalate privileges, steal data, and disrupt operations. Even so, not all ATT&CK techniques warrant detecting, so understanding your threat model in the context of ATT&CK is crucial for defining what kind of threats and techniques you ought to be most concerned about.

Data collection

Your ability to detect threats relies heavily on the data that gives you visibility into the systems where threats manifest. And if you don’t have that data, then you’re not going to be able to detect threats. So once you’ve defined your detection objectives, you also have to assess your visibility via the logs and data sources available to you. Is there a log or data source missing but necessary for your use case? How can you collect that log or data source?

The bigger variety of sources you have, the more ground your detectors can cover and the more threats you can detect. However, having more data isn’t always better. Organizations should strive to only collect data that is strategically useful and helps them accomplish their detection priorities.

Detection analytic development

Now that you’ve defined your objectives and you’re collecting the necessary logs and data, you can move forward with writing detection analytics. These can vary in breadth or depth. For example, they can be based on specific indicators of compromise (IOC) or on behavioral patterns. It’s important, however, to find a balance between overly broad detection analytics that generate tons of noise and overly precise analytics that are too specific. As we explained on the Red Canary blog, “a great detection engineering team works every day on objectives that are at odds with one another. First, don’t miss things. Build detection coverage that has both breadth and depth, looking for as many adversary techniques as you can, and looking for each in ways that are increasingly thorough and resistant to evasion. At the same time, don’t inundate the investigative team with low quality investigative leads, false positives, or a lead that lacks context.”

After developing a detection analytic, an organization must test it.Think about what should trigger the detector, but also test cases that should not trigger the detector. This is an important way to ensure the detection analytic works and does not incorrectly flag the wrong activity.

Continuous testing, validation, and monitoring

Once refined and deployed into production, your detection analytics require ongoing maintenance and optimization. In this stage, continuous monitoring, tuning, suppression, and testing ensure the analytics are functioning properly, detecting what they were designed to detect, and not generating too much noise.

Testing your detection analytics in a variety of ways helps you validate they are working as designed and allows you to identify gaps and deficiencies that may not have existed—or you may not have realized existed—when the original analytic was created. Based on your requirements, you may choose one or all of these test types:

• red teaming
• penetration testing
• atomic testing
• scenarios and table-top exercises
• compliance testing

For the purposes of brevity, we’ll define and share more information on red team testing, as it’s one of the most common approaches to testing a security operations effectiveness. To learn more about the other forms of testing, refer to this article on Testing and validation in the modern security operations center.

Red teaming occurs when professionals from outside your team or organization (e.g., ethical hackers) try to break in, only with your permission. For example, if you are in an industry highly susceptible to ransomware attacks and you don’t have an internal red team, you may hire or consult with an organization that offers red teaming to emulate a ransomware attack as an exercise to surface potential gaps and help you eliminate deficiencies. Red teaming exercises provide insight into meaningful security operations measures, like time from occurrence to detection and time from threat response to recovery or remediation. You’ll also practice the execution of your incident response plan and ensure the humans on your team are better prepared when a real threat emerges.

The threat landscape continues to evolve at a quickening pace, making adopting the right tools and approach difficult. Selecting a combination of tools that individually defend against attacks may seem like a sound approach initially, but these tools rarely work well together. This increases operational overhead and complexity for your already resource-constrained team. If you can unify the data and signals from these disparate tools in a way that allows you to take action, you’ll be ahead of the curve. By optimizing your operations over time with threat intelligence, you can stay ahead. Keep in mind, most organizations can’t do it by themselves. Finding a trusted partner to help augment your team is a popular strategy—it keeps your team focused on what’s unique to your business.

Red Canary pioneered managed detection and response (MDR) and has 4,000+ detection analytics in production that we’ve tuned over our 10 years, working with organizations of all sizes and industries. These analytics span endpoint, network, cloud, IoT, and identity—ensuring customers are protected from threats wherever they occur. Our detection engineering team works alongside our threat hunters to continuously improve detection coverage based on evolving adversarial techniques.

We offer a full suite of response capabilities. For machine-speed response, we offer automated playbooks. We can also work directly with you on a guided response to ensure you have the context about the threat and, more importantly, understand how to act on it. Finally, we offer active remediation—wherein we respond to a detected threat directly on your behalf. These options aren’t mutually exclusive and together help ensure you can respond to threats effectively.