There’s Gold in Those Endpoints: Threat Mining vs Threat Hunting

As the leader of Red Canary’s Cyber Incident Response Team (CIRT), I get a lot of questions about how our detection engineers use threat hunting to find threats on behalf of our customers. My team eats, sleeps, and breathes process telemetry data. We spend our days digging into what is happening on our customers’ endpoints. I want to emphasize the term “digging” because the way we work through data and raise alerts to our customers is more akin to mining than it is to hunting. You could say we perform threat mining as a service (because, hey, if the security industry needs anything, it is a new buzzword or marketing term).

In this article, I’ll walk through:

• How Red Canary uses threat hunting and finds threats on behalf of our customers
• How we continually evolve our detection capabilities
• Why the service we provide is more appropriately considered “threat mining” than threat hunting:

Using Threat Hunting to Improve Detection

As I discussed in Threat Hunting Is Not a Magical Unicorn, threat hunting is a technique that teams should use to improve their detection. At the same time, threat hunting should not be the only (or even the primary) technique used to identify threats. “Threat hunting as a service” is not scalable if the service’s hunting effort is always manual. In order to be efficient, there needs to be some level of automation. This is where I begin delineating threat hunting versus threat mining.

Our SOC team uses threat hunting to help us build and test new behavior signatures, which we internally refer to as detectors. Each detector looks for a specific behavior/attacker technique, indicator set, or metadata concept.

Some examples would be:

• Behavior: A Windows shell making a network connection to an external host
• Indicator set: Network connection to a domain from the Farsight Newly Observed Domain feed
• Metadata concept: A process where the executing binary has never been observed in any Red Canary customer environment

We primarily use threat hunting to develop new behavioral detectors using the following process:

Red Canary’s Approach to Threat Hunting:

Threat Mining: the Next Step in Driving Efficiency

Here’s where this gets more interesting: we intentionally expect false positives from our detectors.

With threat hunting, going through a hunt and not finding something is okay. On the other hand, most threat hunting processes do not have a way to handle false positives. In a pure threat hunting model, the hunter either has to maintain documentation or have an intimate knowledge of the environment to deal with false positives. This is often the barrier to fully automating the logic or hypothesis of the hunt.

With threat mining, we expect false positives to be raised regularly. In true mining terms, these would be called “tailings.” One of the challenges with mining operations is how to deal with the tailings. Red Canary built a process for this. By planning for false positives, we are able to automate the findings of a hunt. We build automation via our detectors, each representing something potentially interesting to the analyst.

Every event raised is not gold, but sometimes you get interesting events that we would not have found otherwise. The pure tailings (false positives) can be “tuned” on the fly by suppressing the event based on a variety of criteria. This suppression reduces the number of processes the analyst has to review in the future without negatively impacting the ability to identify future threats.

Striking Gold

By using this combination of hunting, automation, humans, and continuous feedback into the platform, we are able to consume significantly larger volumes of data than a typical SOC. I’ll also go on record saying that while we use threat hunting as part of our service offering, that is only a small piece of what we do. A better description is that we mine our customers’ endpoints for anomalous and malicious activity. On the plus side, it’s one of the more environmentally friendly mining operations as our tailings do not have any negative impact on the planet (other than a little CPU burn).

Want to improve your threat hunting skills? Browse our most popular threat hunting articles and resources.