Q&A: Visibility, Testing Critically Important for Hunting

In Part 2 of our ongoing “Threat Hunting with ATT&CK” webinar series, Joe Moles and Adam Mathis of Red Canary joined Jimmy Astle of Carbon Black to dig into the critical nature of testing and maintaining a high level of visibility in your network. Topics ranged from running Atomic Red Team tests to becoming familiar with your specific environment and networking fundamentals.

With more than 350 attendees, we weren’t able to answer everyone’s questions during the live Q&A. Fortunately, Joe, Adam, and Jimmy were kind enough to circle back and address the remaining questions afterward. Find their answers below.

Missed the live webinar? Watch it on-demand and receive access to all three parts in the “Threat Hunting With ATT&CK” webinar series.

Q: Are there any prerequisites before learning about threat hunting, like programming or operating system knowledge?

You definitely want to build your base knowledge of the systems you are hunting in and the types of threats you’re looking for in those systems. If you are operating in a standard enterprise environment, then make sure you have solid, systems administrator-level of knowledge about the operating systems therein, the broader network, and tooling deployed on it. You’ll also want to read up on tactics and techniques related to that environment.

If you don’t know what “normal” looks like for a given platform or system, then it’s difficult to identify what is “abnormal” for the same platform or system. For example, SANS has some good resources regarding standard parent-child process relationships. Understanding these expected behaviors will help you more quickly eliminate the normal, presumably benign activity and identify that which is abnormal and more likely to be malicious during your hunts.

An understanding of networking fundamentals like open system interaction (OSI) layers, operating systems internals, and adversary behaviors are all topics to cover down on in the context of threat hunting. You almost need to put your IT or bad guy hat on to really dig into a threat hunter’s mindset.

Q: Should I just open the ATT&CK framework and pick any TTP to start looking for it in the environment?

I would take that same mindset, but start looking where you have the right tooling and visibility. For example, I would not start out with Account Manipulation if you don’t have the proper auditing policies and log aggregation tools in place. It just wouldn’t make sense, and you’d basically be setting yourself up to fail. Another area where the ATT&CK framework is really useful is in identifying areas of investment in terms of technology and processes where you might currently have gaps.

Prioritize what you are looking for based on what you want to achieve. Develop a question or hypothesis, then move to gathering the information to answer that question or confirm or disprove that hypothesis. Threat hunting is very much like running a scientific experiment: frame what your hypothesis is, why the answer or idea is important, and then gather the data needed to investigate.

Some good initial questions to address include: do I have access to the requisite data necessary to examine this aspect of ATT&CK? What systems can an adversary actually execute these techniques on? I recommend writing all of this information down and then tracking your findings as you move along; that way you can go back and reference what you’ve done and learned. This will then help as you frame out where you want to go next.

It can certainly be overwhelming to start the process of systematically hunting through all of the documented tradecraft in ATT&CK. Depending on the data available to you, it may make more sense to prioritize one tactic over others. For example, if you don’t have a great historical process data set, you can still look for signs of compromise from existing persistence mechanisms in use on your machines. Do what you can now versus waiting until you have perfect visibility.

Q: When and where do you draw the line between threat hunting and incident response procedures when you are performing atomic testing or a hunting engagement?

Don’t go hunting without being prepared to go directly into incident response. So make sure your hunters know your incident response plan (IRP). And don’t set out on a threat hunt if you are not ready to respond to an incident. After all, threat hunting can end up proactively discovering a security incident.

Q: How is threat hunting helpful if we are using a tool like Cb Response, which is designed to automatically identify the majority of these ATT&CK-related techniques?

To me, threat hunting isn’t just about looking at things from the eyes of an attacker, but also about trying to continuously monitor your environment for anomalous behaviors. These behaviors can come from an adversary, but, more often than not, they come from your IT team, business processes, developers, and other normal parts of everyday business operations. Endpoint is only one place to collect and correlate data against ATT&CK, which happens to provide the richest breadth of coverage. Network (DHCP/DNS, ARP Tables, etc.), intrusion detection and prevention systems (IDS/IPS), proxies, firewalls, netflow, and other IT and security tooling are super rich data-sources that will help you paint that entire picture of ATT&CK coverage and visibility.

Threat hunting, by most accepted definitions, makes a distinction between hunting and responding to an alert. If you have systems in place to automatically detect and alert on observed activity, which is definitely your goal, hunt for things not already covered in this monitoring footprint. Testing your detection capabilities will help you decide if there is additional hunting/detection building you can focus on for a given technique.

Regarding privilege escalation, what do you think would be good indicators to look for? For example, vision of process permissions, change of permissions, weird processes with weird permissions, different sessions, etc. Where do you start?

There is no silver bullet in detecting privilege escalation, but ensuring you have visibility into the avenues where privilege escalation can take place is super important. So an EDR solution on the endpoint and file integrity monitoring (FIM) on server systems where you know files should not be changing (think: webshells) and then correlating user account logins across your endpoints are a few good places to look. To be honest, ATT&CK is a great place to start to look for these types of behaviors. I would start there!

Most privilege escalation is accomplished by abusing misconfigurations. ATT&CK documents some of the most common misconfiguration issues (ambiguous service paths, scheduled tasks, etc.), and is an excellent place to begin building your plan.

Q: Should you learn to test in multiple environments like regular systems, virtual machine systems, and servers, albeit in restricted systems that do not impact production?

Yes. You want to validate your visibility in various different system types and areas of your network. I also think there is value in doing non-destructive testing in your production environment. You can create or execute a test that will validate your visibility without impacting your systems. Use a combination of unit and functional tests.

Test anywhere you intended to find evil, as running tests in a vacuum on a lab machine will only get you so far. Often you’ll find that controls may be disabled or visibility is missing for specific production systems, and you might miss this without live fire testing.

Q: How do you decide which technique to test? Would you recommend starting from the early stages of the ATT&CK framework (initial access) and working your way through from left to right?

Tests should align to your monitoring and detection priorities. I think of this similar to classifying vulnerabilities: look at the places where you would have the highest likelihood of a breach and the highest impact first if there was a compromise. Then work down the priority order from there. Do not try to solve all the things at once; work in a methodical and prioritized fashion. This way when you find a problem or gap, you can address it and show incremental improvement.

If you’re starting out and all options appear equal, I’d take a look at the Mandiant Cyber Attack Lifecycle. You’ll notice that, on average, attackers spend a disproportionate amount of time escalating privilege levels, performing reconnaissance, moving laterally, and establishing persistence. This is a good place to start!

Q: Can you provide some good resources or reference books for learning about threat hunting?

There is a collection of blogs, articles, and books that you can use to jump-start your threat hunting process here: https://www.threathunting.net/reading-list. And (shameless plug!) Red Canary also has a good selection of resources at redcanary.com/threat-hunting.

Watch the recording of this webinar, along with Part 1 and Part 2 of the “Threat Hunting With ATT&CK” webinar series.