Grand Finale! Building a Mature Threat Hunting Program with MITRE ATT&CK

In the final part of our Threat Hunting with ATT&CK webinar series, we provide an inside look at how sophisticated security teams build mature threat hunting programs. Red Canary’s director of applied research, Casey Smith, guides the conversation as a panel of experts share real-world stories and practical advice to help you develop a level 4 threat hunting program.

Missed the live webinar? Receive on-demand recordings of all three parts of the “Threat Hunting With ATT&CK” series.

Casey and the panelists—Brenden Smith of First Bank, Tony Lambert of Red Canary, and Brian Baskin of Carbon Black—describe how the incorporation of concepts like automation and rigorous data analysis can help you and your team become threat hunting leaders. In the article below, Casey, Tony, Brenden, and Brian answer some fundamental questions about what a security team needs to do to take its threat hunting program to the next level.

Q: What role does automation play in a mature threat hunting program?

Automation can be super helpful in the threat hunting process because it allows computer systems to do what they do best: analyze large quantities of data, bringing interesting findings to the surface and suppressing the noisy, not-so-interesting findings. This, in turn, allows the human brain to do what it does best: spot patterns and anomalies. That said, while automation is important you don’t want to lean too heavily on automation.

Using automation allows human hunters to focus on the investigation of suspicious or evil behaviors, rather than toiling through searches and processes that can be easily performed by computers. Automation can help bring suspicious events to the attention of humans and lead them into an investigation with additional context. Essentially, automation can give humans the tools to make better decisions about suspicious behavior in an environment.

Q: What are some of the strengths and limitations of automation in the hunting process?

On the positive side, automation removes the time and resource-intensive processes away from analysts, allowing teams to utilize past intelligence and known context to minimize false positives so that the focus is on new threats. However, automation requires good intelligence to be effective. A poorly constructed query, or reliance upon a badly researched report, will create issues with potential attacks being overlooked or downgraded in severity.

The detection lifecycle really lets us build out playbooks that are assisted by automation so that we can handle incidents the same way every time. It’s very frustrating to have similar events occurring without a standardized way to respond to them. The important thing for us is that we have a framework, and we are consistent about the way we operationalize our threat hunting. We have a big team, so if we don’t have a formal structure, we can get a bit lost.

The limitations of automated hunting are the strengths of human-based hunting and vice versa: machines are bad at improvising and good at recognizing unexpected behavior, whereas humans are great at improvising and not as reliable at recognizing unexpected behavior. Automation is amazing at taking a repeatable process and running with it so humans don’t have to bear the load they would otherwise. Humans can use their previous experience and knowledge to improvise and recognize behaviors that are unexpected, like when trusted processes perform evil actions. To achieve the same effect with automation, an intense amount of coding, learning, and baselining would be needed.

Q: What are some of the key behavioral indicators that analysts can focus on to separate the wheat from the chaff during the hunt process?

One that immediately comes to mind, which was mentioned by Jimmy Astle of Carbon Black in the second part of this webinar series, is “account reuse.” You essentially are asking yourself, “Why is this user using this machine?” The really interesting behavioral detections are often found in accounts used in lateral movement or interactive service accounts logins.

In the case of lateral movement, a good behavior to seek is network communication using administration tools that are not approved and communication between hosts that are not part of authorized administration methods. If PsExec isn’t allowed on your network, hunt for it! At best you might find a policy violation; at the worst, you might find an adversary moving between systems. This notion shouldn’t be limited only to endpoint analysis. You can also perform the same type of hunts by analyzing network traffic between hosts and network segments.

Q: How do you demonstrate the value of time spent threat hunting? What are your indicators of success, especially when a hunt takes up time but turns up nothing?

One successful indicator of proper threat hunting is that you gain rich context about the environment in which you are hunting. Any behavior that may be considered unusual is assessed, catalogued, and queryable. This allows for the production of proactive security reporting, which could be useful in a variety of cases, including when a new TTP has been disclosed publicly, for example. A security team can quickly assess the threat, compare it to known behavior in their environment, and create a trusted report. A second point is that threat hunting is meant to be tested. Mature organizations undergo routine, professional penetrating tests from outside organizations. A competent threat hunting team should be able to identify such attacks and work in a “hands-off” way to identify such tests as they occur. A successful threat hunt should then be able to assign every reported attack by the red team to observed artifacts as if it were a real attack.

Threat hunting is an odd value proposition. If a hunt is successful in finding evil, the organization burns more resources to be rid of the evil. On the other hand, a hunt without finding evil gives the illusion of using resources with no return. To push back against this notion, you can demonstrate value through the improvement of processes when you complete an “unsuccessful” hunt. Every hunt should teach you something about your environment. In the cases where you don’t find evil, you learn what looks normal in your environment and things you’re definitely not interested in detecting. A good example of this would be a MS Word macro launching a legitimate script to manage files (we see this a lot in manufacturing). You can use that knowledge to tune future hunts and alerts so less time is used during a hunt. In these cases, a good indicator of actual success is the amount of time used to perform a hunt. For subsequent hunts of the same tactics, less time should be needed as your tools are tuned.

We’ve found that mapping is a great way to demonstrate consistency in your threat hunting program, and I often use these maps as educational resources when I’m talking to the board, executives, or even regulators.

Q: Are there any brief stories that summarize the importance of an organization with a mature hunting program in place?

Along with Casey’s “Why is this user here?” question, I’ve had great results in simply asking “Why is this executable here?” This is especially useful when potentially non-standard LOLBins are in use. In an incident response, a large organization had whitelisted Putty Link (PLink) to create encrypted network tunnels from IT systems to servers. However, the same tool is also deployed by adversaries to mask their internal movement. The organization had a mature software inventory and an endpoint providing rich metadata, which meant they were able to quickly assess different versions of the same tool, determine which were previously approved by IT, and respond quickly to systems with unauthorized versions. This provided a method to identify systems that were previously compromised and staged, even though they were not used in active attacks.

Q: What are the most important aspects of a sophisticated threat hunting program?

The best threat hunting practices tend to emerge from teams that have unconstrained time and resources to develop theories, understand visibility and collection gaps, and develop custom tooling to enable hunts. No matter what your constraints are, it’s important to remember that not all hunts find interesting things, but you’ll almost always learn something when you try to answer a question about your environment.

I think there are three guiding principles that can enable the best hunting programs. As such, threat hunting programs should 1) allow the flexibility and freedom to anticipate upcoming threats instead of being reactive to current and previous styles of attack, 2) focus on expanding visibility, ensuring that there are minimal, if any, gaps for threats to exist in, and 3) be continually testable, either by intentional or unintentional sources, so that you can detect gaps while allowing proper time to address improvements.

I always want to make sure that the security analysts in my organization are getting the right context and information. Beyond that, the good threat hunting units are able to optimize team efforts so that they are as efficient as possible. As a rule, we want them to be able to get to the real threats as quickly as possible when we’re hunting.

Register to watch Part 3 on demand and get access to the full “Threat Hunting With ATT&CK” webinar series.