Red Canary’s Senior Manager of Threat Hunting Ingrid Parker, MITRE’s Kathryn Knerler, and Microsoft’s Carson Zimmerman have been exploring top strategies for taking your security operations center to the world-class level. In fact, they’ve teamed up for a book and a full season of podcasts on the subject.
In their recent appearance on the SANS “Wait Just an Infosec” podcast, the conversation turned to strategies for building threat hunting capabilities in a security operations center (SOC) and common problems that hold enterprises back from building out their threat hunting teams.
A theme that Ingrid and her co-authors underscore is the importance of taking a proactive approach to threat hunting. If your organization has been stuck in neutral from a threat hunting perspective, or you’re looking for validation on your threat hunting approach, this excerpt is rich with advice for the best frameworks and business cases for threat hunting, We’ve embedded the entire podcast, as it’s worth a listen, but the conversation with Ingrid, Kathryn, and Carson starts at 13:20.
You can read a transcript of the conversation below, which has been edited for clarity:
For teams just getting started with threat hunting
John Hubbard
Any thoughts on—in your experience and especially as it pertains to the book and some of the guidance in there—what is it that is potentially holding teams back (from building out threat hunting teams) and what should people be thinking about at this stage in their threat hunting capability?
Kathryn Knerler
So I’ll start because I forgot to mention something very important. First of all, I work for MITRE still, and this is a book that we’ve been talking about for a while, as John mentioned, and it is available for free. You can also buy it if you prefer it in print. Threat hunting is in Chapter 11 of the book and Carson, Ingrid, and I spent a lot of time talking about it. It’s a very exciting topic, so I want to point that out,just to start it off and then I’m sure Ingrid and Carson will jump in.
Threat hunting has become an increasingly more important function for your security operations. Excited to talk about that because it’s not good enough to just respond to those things coming in. Threat hunting is about looking for stuff proactively, so to the question, what’s holding people back? My experience has been the people. It requires skill to start doing threat hunting, and so I’m really encouraging everyone to kind of set a low bar and just try, you know, looking at log files, if nothing else, just to jump in and get started. But people tend to be what holds it back. There’s just not enough resources to do things over.
Ingrid Parker
Building from people, the challenge is if you’re in operations, you’re dealing with the fires, a lot of times you’re dealing with the alerts that come in, you’re dealing with what incidents happen, and threat hunting requires that you be able to take a little bit of time to step back and think creatively about the data that you have available, what you can do with it, what you’re trying to find, and it’s very easy to get caught up in , “Oh, I’ll just start looking for things”, in that very ad hoc way and feel like you’re not making progress.
So I think it’s a combination of needing to decide that you as an organization are at a point where this is going to be beneficial to you. That is going to add to the things that you’re trying to do, being clear about a couple outcomes. And by outcomes, I don’t mean that you’ve found the badness, because there are great outcomes from threat hunting that aren’t about just finding incidents.They’re about understanding your environment or knowing what’s not happening in your environment. And then figuring out some way to dedicate some time to that so that you’re not just trying to do hunting in five-minute increments in between the latest alert that has popped up on your screen.
What’s holding threat hunting teams back
John Hubbard
Let’s get to our second poll question where we are actually going to ask people exactly that. “On your team, what is it that’s preventing you from taking your threat hunting to the next level?”
And what I’m trying to get at here is to maybe coalesce around some certain themes and questions that may be getting in the way of your threat hunting team.
(Referring to the poll) Money. That’s an interesting one..
…so team budget training, budget funding, expertise, money. Anyone surprised by any of these things?
Carson Zimmerman
No, I think we’ve all got feelings about that. I’ll look at that expertise and the team budget. I think it’s probably worth starting to address some of these. When people think about hunting, they often think about this big grandiose, super formal, super hard to get into high-end activity. And I would lower the bar a little bit. We’ve talked about this before as a team; hunting is not aimlessly wandering through data, nor is it ordinary incident investigation. That said, hunting can start from very little, from very humble beginnings. You can start, for example, from your incident work and thinking about, “Oh, how do I pull the string and pull the thread on the thing I saw in an incident but didn’t have time for it later?” So I would say, you know, just not knowing where to start seems to get in people’s way and they put that bar too high in both budget and expertise. I’ll leave it there.
John Hubbard
Any other thoughts on any of the words that are popping up here?
Kathryn Knerler
I’ll elaborate on the people part because I meant exactly what’s coming up here. The expertise and the number of people too. When you say team budget, it’s not just the skills, it’s the tools that go along with it, but as Carson was mentioning, you definitely can start with very little with humble beginnings. You can just start with looking at your previous incidents. I’m not saying incident investigation is the same as threat hunting, but you can certainly leverage what you’ve learned from the incidents that you’ve had and start thinking about what an adversary might look like in your data. So, look for what is an adversary, what’s weird? What’s an adversary interested in?
So expertise is definitely a bar, but if you’ve done an incident investigation, you can begin by leveraging what you already know and certainly look at the ATT&CK framework. I don’t want to be an advertisement for MITRE, but attack.mitre.org is is a great place to start. It’s free information on TTPs that have been curated. These are actual TTPs of adversaries that are out there and you can start by looking at “what does an adversary look like in your environment?”
Ingrid Parker
I think about the budget perspective, and that expertise perspective, and gaining buy-in from leadership to spend time on this. And this is where I think it’s really important that even if you’re early, you think about those outcomes and they don’t have to be this big grandiose, you know, “we’re going to find everything.” It literally can be: “Hey, we know that we need some new detectors around *this* because we think it’s important to our environment.” So let’s go in. Let’s do some hunting. Let’s find out what’s there. Let’s look at the ATT&CK model. Let’s figure out the TTPs that are important to us, and focus on hunting in this way to find out what is important to us.
Is it something we see in our environment? Should we have better detectors around it? That can be a very small loop that you can work on and then be able to show those metrics of how you’re improving your organization’s security posture overall, which has a really powerful story to it that helps mitigate what I’ve seen as some of the challenges with leadership, of them going, “Well, you’re off hunting, you’re just playing in data.” Which yeah, we’re playing the data because it’s fun, but really you’re doing it with a purpose.
So I would say even if you only have a little bit of time each week, figure out what your objective is. Do it consistently. If it’s truly, “Hey, we can only spend two hours a week doing this to start.” Then do it two hours every week, have a plan for that and come out with those objectives so that you can start to show the progress and show the value that this is going to bring to your team.
What should we hunt for?
John Hubbard
Awesome. All great stuff. I do want to get to the next question here as well while we keep going on this. The second poll to the audience is going to be a little bit about where you get the ideas for what you are going to go threat hunt for. So if you walk into the office and you’re like, “Today I’m going to go on a threat hunting trip,” what is the core input that you are using for that? Is that going to be maybe something you read on the news that day? Is that going to come from your cyber threat Intelligence team? Is that going to be based on previous threat hunts? Is that going to be something based on whatever your CISO or your manager or your director told you to hunt for that day?
Do you have plans for hunts where you have a backlog of things you want to look for? Different reports that may be coming up, things like that. We’d love to know what your primary source of ideas for threat hunting is.
To my co-hosts here, what do you think are some of the most valuable resources, especially if you’re early on in your threat hunting, where should people be looking for inspiration and priorities for threat hunting?
Carson Zimmerman
I think one of the major pieces here is other people’s data. And that actually goes to one of the funding pieces. Yes, there’s the people issue we’ve identified that. However, a lot of people when they’re doing hunting may feel like, oh, “I have to get all of the data into my own SIEM or my own log analytics repository” or whatever it is that you’re focusing your analysts on.
And I would push back a little bit and say, you know, we all agree that the SOC should have as much important data as it should. However, some of the best SOCs and some of the best analysts I’ve met spent most of their time in other people’s data. So the point is, we can use our inspiration and mitigate some aspects of the funding problem by being inspired by other people’s data where it exists.
Kathryn Knerler
Building on that, where can you get other people’s data? Certainly by joining forums of various kinds—some are in person, some are online. There’s various ways that you can start talking to other people, especially if they’re in an industry that’s related to yours, because a lot of times adversaries will target similar kinds of organizations. And some of the powerful forums that I’ve seen over time are where companies have gotten together and worked, combining and sharing the TTPs and the incidents that they’ve had within their organization. And I think it goes without saying too: Your own data is a great place to start for cyber threat intelligence. We tend to overlook it sometimes because we already have that data, so we want new data, but make sure you use your own data. That’s cyber threat incident data and also what’s coming at you, what you’re seeing from adversaries.
John Hubbard
For sure. So let’s take a look at some of the other intelligence sources in our audience poll. We’ve got risk posture. We got newsletters and threat intelligence, the dark web, CTI, zero days, Twitter, OSINT, MITRE, a lot of great options here. It looks like we got some threat intelligence vendors, previous knowledge. That’s part of it as well. Looking back at what has affected you in the past and just kind of saying like, what have we fallen victim to? What do we know we’re still potentially vulnerable to? Where are there potential gaps in our coverage? And you can use that as a trigger to say that this is one of our priority attack groups that we’re paying attention to. We know they do this thing. We know they’re not that good at finding it. And therefore, it would probably make a lot of sense to maybe focus some effort on undoing that sort of thing.
I saw a comment go by, “My opinion on threat hunting. Threat hunting should culminate in enhancing the detection in the SIEM, custom detections in EDR, and other detective controls.” Absolutely right. If you go threat hunting and you find a thing that works, an analytic, some kind of measure, making sure that feeds back and just becomes the way you do things from there on out and you don’t have to threat hunt in that specific way. Again, just continuous process improvement through threat hunting as well, taking what did work, maybe feeding it back to the team and using that and maybe a little kind of presentation to the team are tactics I’ve used in the past to make sure we can start to build up that knowledge on how to do threat hunting across the entire team.
Any other kind of thoughts or comments on this from my co-hosts here on our process for finding inspiration for threat hunting?
Ingrid Parker
One of the things I notice is there’s a lot of ideas here, and I think that’s where it can become really challenging when you start to say, “Oh my goodness, I need to go check Twitter and I’ve got to read the intelligence feeds.” And if you’re lucky enough to have an intelligence team, you should be talking to them. “Oh, and I’ve got my own data and I’ve got other data as and I’m part of it…” It just adds up. And so it’s one of those where, like with everything we do, take a deep breath, step back, find a thing that works for you. Don’t feel like you have to do everything. If there’s a particular threat intelligence report from a vendor or SANS or anybody else, something you look at every week anyway, then start from there. If you’ve got a really good history of your own incident data, start from there. Just find something. Don’t feel like you have to do all of this to start, because I’ve worked with some really large great hunting teams, and they don’t look at everything. They’re very selective in where they choose to start.
Carson Zimmerman
John, there’s something you mentioned that I wanted to double down on, and also it goes off what Ingrid was saying, and that is emphasizing proactive hunting as a highly collaborative activity, both within the SOC. This is a way to have analysts, investigators, responders learn from each other, learn about their data estate, learn about analytic techniques, learn about the business, etc. It is also a collaborative opportunity for the SOC to engage some of its key users, key business owners, key stakeholders who are cyber interested, cyber curious, or don’t know how to participate. Hunting is one of the ways we can bring them in and make them not just feel loved and not just incorporate their ideas, but really bring their ideas into how we’re looking for threats. And then that, as you were stating earlier, kind of dovetails into, hopefully, some detections later on.
What about attribution?
John Hubbard
Yeah. Detections and potentially some building knowledge of the types of groups and the tactics that they’re using when they attack you. One of the comments I just saw: “How important is attribution?” That’s another good question that constantly comes up. I see some smiles from the cohosts here, and that’s exactly what I want to ask about: How important is it to be able to map what you’re seeing and what you’re finding in your threat hunts back to a specific threat?
Kathryn Knerler
This is actually one of my favorite questions. We deal with this a lot of time. If you’re looking at traditional intelligence, they’ll tell you that you need perfect attribution. You need to know exactly who’s coming at you to be able to anticipate. And while that would be wonderful if we had that kind of perfect information, when you work in the security operations, sometimes there’s a “good enough.”
So if you kind of know basically what’s happening and you can kind of attribute it to a particular adversary, it might help you anticipate some of the moves they may make against you and where they may go into your environment, into your intellectual property, what they’re going after. So it is useful to have attribution, or more of an association, so adversary associations, knowing what’s happening without the perfect attribution. Attribution is a big argument in security operations, threat hunting, cyber threat intelligence and traditional intelligence.
Ingrid Parker
Where we came down to in the book, as Kathryn is saying, it’s association. It’s important to know that it is similar to this named group, if somebody named it, or to this type of activity you’ve seen before because it can help you understand what might be coming next or what else you should look for. But it’s not the kind of attribution where you can put the named person behind the keyboard in the country of your choice, because that doesn’t help you as a defender. Make the decisions you need to make about your own environment.
Kathryn Knerler
Yeah, and attribution can be very costly. And that’s the big argument in security operations. We don’t have the resources or the time to go figure out exactly who this is.
Ingrid Parker
And a lot of times we don’t have the data.
Kathryn Knerler
And we don’t have the data.
John Hubbard
I’m sure I have the perfect crowd for this question on group naming conventions. You know, let’s say you’re threat hunting, you find a domain name, you type in a search, and you find that was APT “A” or whatever it is. Are we done there or do we need to start kind of collecting additional different names? And how might we do that? Any thoughts you have on APT naming conventions and tracking your threat intelligence over time?
Ingrid Parker
I love them and I hate them because it’s like, you know, we as humans cannot keep all these different data points in our mind without some way to structure and organize them. Naming conventions try and do that. They try and help us create the shortcuts we need to keep a lot of complex information available to us in an easier way.
Where analysts struggle—and this is not the fault of this concept—but where they struggle is because we don’t always know exactly what went into the analysis of how that organization gave that name. We may not be making the same associations within our own environment to make the same kind of analytical decisions about why they grouped it. This is something we see a lot, where somebody will name it one thing, somebody will name it another, and we go, “Oh, this is the same.” But if you have that opportunity to dive in, or maybe you can talk to the analysts at both teams, you realize that they actually had different data they used to come to their assumptions and it isn’t exactly the same.
There are some great diagrams out there, especially about some of the Iranian groups that show all the different names. And there’s a one bubble diagram I always think of that overlaps all of the different pieces.You can see how it’s never a 1:1 match. So I think like everything, associations can be helpful to get you a shortcut, but don’t jump to the conclusion of, “Oh, I’ve seen these three factors. So it’s absolutely this.”
Make sure you understand what went into that naming and if it’s not relevant to you, consider naming it yourself internally until you have a better idea. And if you look at a lot of the companies out there that name these intelligence groups, they actually track undefined groups. They actually have groups that are in progress. They are continually reevaluating what they’re doing so that it’s not just a static of, “Oh, we named it this, it is always this.” Because those groups, within countries, within criminal organizations are changing, too. So they might not be the same group that they were five years ago.
John Hubbard
Yeah, all excellent points. And that’s exactly what I wanted everyone to be sure to hear. If they hadn’t heard that before; this is always in flux. Things are changing, there’s overlap. There’s stuff that seems like it might be the same and not. So you have to be careful to try to track associated group names. But know it’s not a perfect science because every vendor, every kind of threat intelligence group gets their own piece of the pie.
The third question I want to jump to here before we run out of time here is a little bit about what helps you threat hunt. So what kind of frameworks do you use? What kind of tools do you use? This is our final audience question of the day, “What frameworks or tools help you organize, execute and measure threat hunts?” If you’re out there threat hunting right now, I would love to know, succinct word tool, whatever it happens to be—what’s helping your team actually get this done? Figure out the outcomes of what you’re getting out of the activity of threat hunting and those sorts of things.
On zero days
John Hammond
One thing I saw go by in the meantime here I wanted to ask everyone about as well. There was a comment about threat hunting for zero days being incredibly difficult or impossible to do. You know, in very real terms, zero day is defined as we don’t know what it looks like. So to my cohosts: if you’re worried about zero days, is there anything in the realm of threat hunting that you can think of that might help people find those kind of activities that we, by definition, have no direct signature for?
Ingrid Parker
One of the things I think about with zero days is that the zero day is probably a vulnerability in some particular application piece of hardware, whatever it is, but that’s only one phase of the lifecycle an adversary takes. And so even if that vulnerability is something that you’re maybe not detecting right now, the adversary is going to take next step actions. They’re going to create command and control back to their organization, some home base. They’re going to do a lateral movement, they’re going to be doing something else that you probably have some detectors for or that you could be hunting for. That’s going to be really powerful.
I know during our own podcast earlier, we were talking about supply chain. What can you do about supply chain attacks because it’s so important, and you don’t see it until it hits you? But the fact is, after it hits you, you get to see all of these other indicators. So I would worry less about, “can we identify this particular zero day?” and more about if you can hunt across all the different phases of the lifecycle so you don’t get caught short if you miss it in one particular place.
John Hubbard
Yeah I love that point. And you know, that’s one of those things I’m constantly trying to stress as well, especially with supply chain attacks, is people wondering how they’re ever going to catch the next SolarWinds-style trusted vendor attack, something embedded in a DLL? Well, you might not catch that phase of it, but that is just the one phase, right?
Attackers aren’t magic. Just because they got in doesn’t mean they’re immediately out the door with everything, right? That’s the delivery stage. That’s maybe the exploit stage. But nearly any attack that’s high impact is going to be something that’s complex, multi-stage, played out over weeks and months. And so while you not may not be able to threat hunt—and I’m not even sure that this is even true—but while you may not be able to threat hunt directly for zero days, there’s still every other stage, every other tactic, every other technique that’s used. And all of those things have to work for the attack to actually be successful. So if you catch any of those things, you still stop the attack. And that’s truly what matters, right?
Threat hunting frameworks and tools
Carson Zimmerman
John, I’d like to take a somewhat controversial stance given the words from the audience we see popping up here. And by the way, flattery will get the audience everywhere, it seems. MITRE ATT&CK is the biggest…I don’t even work at MITRE anymore, but this is great. The controversial stance I’ll take is yeah, SIEM is important, we all agree with that. But there’s a bunch of other tools on here that I love. I would argue the most important tool for hunting is where you take your notes. Where are you capturing the queries that you’ve run, the analytic conclusions you’ve drawn, the notes you’ve had from one day to the next? Because I guarantee you’re going to forget half of it.
And considering that hunting is generally a highly collaborative activity, we want to think about capturing those.
Kathryn Knerler
To build on that, threat hunting is an evolution, right? It’s a learning process and you’re never done. So it’s the journey, not the destination. Adversaries will change up what they’re doing anyway. So what Carson said is absolutely right, I don’t think it’s super controversial. Believe it or not, I think notes and a history of what you’re doing is a super, super excellent idea.
And back to the zero days. Looking for zero days is a—I’ll just be really bold and say a little bit of a fool’s errand because you don’t know. There are thousands, millions of vulnerabilities out there. And we don’t know which vulnerabilities people will build. Adversaries will build tools for (vulnerabilities) in advance.
But keep on top of it; if you hear about a zero day, certainly be proactive. That’s a great place for your threat hunting team to build a case around. If there’s a zero day out there and it does affect your environment, be proactive about it.
Ingrid Parker
One thing I want to bring up that I didn’t see on this list was TaHiTI, which is a framework for how to think about hunting if you are newer in that process. They’ve got three phases: initiate, hunt, finalize. I think all the notes Carson was talking about definitely fall in that finalize part. But it’s the process parts that go behind hunting, and it’s a great resource that’s available out there for everybody to look at.
John Hubbard
There’s a whole nice kind of write up of what’s going on in all the stages and what to do and even an Excel spreadsheet and things that you can track along the way to help make sure you know all of the value that you’ve created, all the things that you’ve discovered, anything that has happened during your hunting sessions, get organized, get captured, and then get rolled up into a bigger level metric of why you’re threat hunting and hopefully justifying the time being spent.
That brings me to another question and a comment I saw roll by. Looks like Elliot asked, “Curious how SOCs tie threat hunting resource costs back to budgeting and value to the organization?” Any thoughts on how to specifically tie metrics around threat hunting and how you can show that direct value to the organization for the time spent? I know it came up a little bit earlier, but I wanted to directly answer that question because I know a lot of people ask about that.
Making it all count with metrics
Ingrid Parker
We all take a deep breath because metrics are hard, but this is actually where I think consistency is important. When you’re thinking about metrics, consider not just the zero, one, yes, no, this number, whatever else. Don’t focus too much on how long a hunt takes or exactly how many you do; try and tie it to the kinds of outputs that you’re working on.
So, “Hey, we ran X number of hunts and of that, this percentage resulted in new detectors. This percentage resulted in confirmation that we are not vulnerable to this new zero day. This percentage resulted in discovery of the fact that we’re running, 16 remote management tools, and perhaps we only need one, if any,” you know, whatever it is.
Make sure you’re showing this information over time. Because if you get caught up in the loop of, “Oh, we did five hunts this month, the next month we did seven hunts,” that’s not going to be successful and it’s not going to set your executives up for asking the right questions about the outputs that you’re achieving.
Kathryn Knerler
Yeah. And specifically, when I’ve briefed executives on this, what I’ve said is “we’ve blocked X number of adversaries that we did not previously know were going to be an adversary for us.” So being able to create a block upfront even though you haven’t been hit is one way to show the value.
Carson Zimmerman
Ingrid did a great job of reading my mind, as she sometimes does. I think there are a couple of things I would add here: One of them is, we should think about hunts coming in very different sizes, which is why we don’t want to over-fixate or over-index on the number of hunts we run every month. And by the way, that number is usually pretty darn low, even if you’re a big team. So I would encourage the audience to separate out hypotheses proven or disproven coming out of a hunt from the, “Hey, we found random bad stuff.” You’re always going to find random bad stuff in a hunt, even if it’s totally unsuccessful in finding an adversary, and you will be unsuccessful in many of your hunts finding an adversary. You will also find serious hygiene issues.
So one of the things to think about in value return to the business is not just our ability to find adversaries or get better at detecting them, but also along the way what did we discover about the enterprises, about our user base, our defended services, etc., and how did we return those discoveries back to the businesses, either changes in posture or or some other aspect of the overall cybersecurity apparatus?
John Hubbard
Yep. Perfect kind of summary and kind of direction.