The three keys to threat hunting

“Threat hunting” is among the great buzzwords of cybersecurity. Everyone feels like they need to do it, but there’s no universally agreed-upon definition of what threat hunting entails. Is it sweeping an environment for indicators of compromise (IOC)? Reviewing alerts? Or does it necessarily require querying your way through disparate log sources and correlating aggregate data to find the threat needles in the IT infrastructure haystack?

In this blog, we’ll briefly describe how threat hunting has evolved from its spare-time, reactionary origins to its current status as a dedicated discipline of security operations. Then we’ll explore a few foundational principals that organizations should consider if they’re starting, evaluating, or rehabilitating a threat hunting program. We hope that readers can leverage these ideas and concepts to improve their threat hunting program in particular and their organization’s security operations more generally.

What is threat hunting anyway?

Early on, threat hunting was typically ad hoc and involved reading an intelligence report and then grepping raw syslog data for evidence of malicious or suspicious activity—often during free time when analysts weren’t investigating alerts. The fundamental purpose of threat hunting was to manually search for threats that might have slipped past your organization’s existing security controls. This hasn’t exactly changed, but the systems in which an organization can hunt have expanded and the practice of threat hunting has become more sophisticated.

Hunch-based, ad hoc hunting by SOC analysts with a bit of free time still exists (and is great for organizations with limited resources), but threat hunting has also evolved to include hypothesis-driven, programmatic hunts performed by dedicated teams of threat hunting specialists. Mature teams tailor a threat hunt program to their individual organization, with particular consideration to its unique IT infrastructure, threat model, and the risks they’re most concerned about. Beyond that, quality hunts follow repeatable processes with clear success criteria. With all this in mind, we tend to organize hunt activity into three categories (not to be confused with the three principles we’re going to discuss throughout this blog):

There’s no single correct way to implement a threat hunting program, since all good threat hunting programs address an organization’s unique needs. However, after a decade of threat hunting across thousands of organizations, we’ve identified three key principles that any organization can follow to threat hunt with purpose.

Threat hunting with purpose

Threat hunting should be a deliberate, proactive, and iterative process to confirm or disprove hypotheses about latent malicious or suspicious activity.

Threat hunting is deliberate

Hunts should have a clear goal and a repeatable means of accomplishing that goal.

Without intention, it is difficult to produce repeatable results that align with specific business goals. Disparate, aimless hunts can lead to redundant findings that aren’t reproducible and don’t necessarily align with your organization’s threat model, risk profile, or security needs. However, given specific parameters to consider (e.g., threats, risks, or IT systems you’re concerned about) and repeatable processes to follow (that include documenting your findings), you can avoid redundancy, address actual security priorities, ensure hunts are reproducible, and generate outcomes that actually benefit your organization.

The first (and most important part) of being deliberate is understanding your organization’s security priorities and allowing those to guide the focus of your threat hunting. Being deliberate also requires threat hunting teams to deeply understand their IT infrastructure, ensuring that they are collecting logs from data sources that offer them visibility into the threats and risks they’re actually concerned about—whether it’s a set of MITRE ATT&CK® techniques, specific threats, a broader risk to your organization like business email compromise, or a shadowy part of your IT environment that gives you the scaries.

Lastly, you can’t be deliberate if you don’t have a clear idea of what success looks like—and success can look like a lot of things—from validating that your SOC fully remediated a threat to searching for one you just read about in a breaking threat intelligence report to testing out new detection ideas.

You can’t be deliberate if you don’t have a clear idea of what success looks like.

For our part, one of the key priorities of our threat hunting program is to complement our detection coverage. We deeply understand the strengths and weaknesses of detection engineering at Red Canary scale and try to focus our hunts on places and activity that aren’t effectively detectable by analytics. Threat hunters may seek to discover threats in places where high-fidelity detection is a challenge, like enumeration for example, which is incredibly noisy and can cause serious false positive problems if detected or alerted on programmatically. We do this both across our customer base, addressing broad coverage gaps for the tooling we support, and within individual customers.

Your goals may be different, but it’s massively beneficial to understand where your coverage is strong and weak—this can pertain to specific threats or entire parts of your IT infrastructure—and develop a hypothesis-driven hunt strategy to search for unknown threats that may be emergent or merely lurking in less well-monitored parts of your environment. The following shows some examples of potential threat hunting outputs:

Ultimately, take the time to understand what you want to accomplish and how you aim to accomplish it in order to avoid meandering threat hunts that waste time and produce dubious results. Successful threat hunts don’t reliably come by chance. They come from understanding your goals, the challenges you face, and establishing clear criteria for success.

Threat hunting is proactive

Hunts should actively seek to uncover risks or threats that evade your existing security controls.

Even if you’re undertaking a reactive or exploratory hunt, your approach should still be proactive. Threat hunting exists to add a layer of defense in depth and aid in early detection of threats because traditional security controls are not always enough to consistently detect malicious activity. Adversaries change tactics, abuse misconfigured tools, exploit unpatched vulnerabilities, and otherwise evade security measures in unpredictable ways that are, by definition, not going to be detected or prevented by your existing security controls that are designed to catch known threats or mitigate known risks.

Threat hunting does not rely on the tools or detections that are in place, rather threat hunters proactively look across the environment to identify threats that may have evaded detection, adopting new tools and log sources as needed. Counterintuitively, some hunt operations may be reacting to new information or merely exploratory, but they should be proactive in the sense that you’re actively seeking a solution—not merely waiting for some other security control to solve the problem for you.

Threat hunting does not rely on the tools or detections that are in place, rather threat hunters proactively look across the environment to identify threats that may have evaded detection, adopting new tools and log sources as needed.

The job of a threat hunter, after all, is to identify malicious or suspicious activity instead of waiting to be informed by a notification or alert—or to allow unknown threats to linger in an organization’s IT infrastructure. Threat hunting is intended to provide defense in depth beyond that which is provided by detective, preventive, and other security controls.

Threat hunting is iterative

Hunts should be based on a process that adapts to new information and past performance.

It would be nice to say we conducted a threat hunt and the organization is now secure. However, we know that the security landscape and the organizations we protect are always evolving. As time goes by, new threats emerge, attack surfaces expand, and new technologies emerge, which requires threat hunting to be both continual and a malleable process capable of addressing new requirements and challenges on the fly.

Adversaries are largely advantageous, and so a change in email infrastructure or cloud migration, for example, may expose an organization to new threats before they are able to develop compensatory security controls for them. Threat hunting is a great stopgap for combatting these new threats. Effective threat hunting programs are constantly learning lessons from previous successes or failures and applying them to new hunts that address emerging needs. Documenting previous experiences will empower your threat hunting function to retain whatever queries or code they develop, report their findings consistently, and develop improved processes for future hunts. As an added bonus, threat hunts can serve as a feeder for developing new security controls, like when a threat hunt query is adapted into a new detection analytic.

Effective threat hunting programs are constantly learning lessons from previous successes or failures and applying them to new hunts that address emerging needs.

A flexible, iterative approach to threat hunting will simultaneously protect your threat hunting team from reinventing the wheel with each subsequent hunt—while also enabling them to hunt across new technologies for novel or evasive threats and techniques. Your team may not be experts in all domains or able to hunt across all data sources at this point in time. However, having an intentional process and feedback loop for lessons learned will enable your threat hunting program to mature and routinely identify potential threats missed by other security controls.

The following shows a mock plan for how you might organize a hunt that is deliberate, proactive, and iterative:

How does threat hunting fit in your organization?

Threat hunting can improve outcomes for organizations all across the spectrum of security operations maturity, regardless of the size of your organization or the skills and experience of the personnel working in your SOC. Anyone can start by striving to better understand their environment, setting goals for the value you hope to gain from threat hunting, and making a plan for how your team can approach hunting across your datasets.

Some important factors to consider as you build out your organization’s threat hunting capability:

• threat profile
• risk tolerance
• network infrastructure
• IT assets
• available resources, spanning from people to technology

For most, threats encompass much more than malware and might include the introduction of risks caused by lackadaisical software hygiene, lackluster patching, or infrastructure challenges. Further, these threats and risks can impact servers, endpoints, applications, and all varieties of IT systems. Understanding how to gain visibility into all of these different kinds of assets is a challenge on its own, and figuring out where to store all of their corresponding logs or telemetry presents another layer of difficulty. However, these are not insurmountable challenges.

Effective threat hunting is meant to help your organization understand things that it may not know about itself. For example, if you’re an organization that has controls around USB use but still allows your corporate laptops to connect to home network-attached storage devices—which potentially presents susceptibility to worms, exfiltration points, and remote code execution—that’s a valuable narrative to understand and explore to improve your organization’s security posture. In this way, you can use what you learn about your IT infrastructure while threat hunting to provide valuable insights to your organization’s security leaders.

Effective threat hunting is meant to help your organization understand things that it may not know about itself.

One thing we’ve heard frequently from CISOs and security leaders over the years is that it’s not the problems they know about that keep them up at night but the problems they don’t know about. Threat hunting should instill confidence in organizations that they’ve got a team that is deliberately, proactively, and iteratively looking for unknowns across their entire IT infrastructure.

Conclusion

Whether you have a robust threat hunting program that you want to improve or you’re conducting ad hoc threat hunts as time permits, here are a few good things to keep in mind:

• Understand your reason for hunting, what the desired outcomes are, and be deliberate in your approach to hunting based on your understanding of the businesses needs and environment
• Be proactive and consistent in threat hunting to identify threats and risks
• Take lessons learned to continue to iterate and build new and improved threat hunting capabilities

A challenge worth undertaking, threat hunting provides awareness and supportive value to your security practice, no matter how advanced your capabilities are.