Skip Navigation
Get a Demo
 
 
 
 
 
 
 
 
 
Resources Blog Security operations

Navigating the cloud security landscape

Watch one of Red Canary’s co-founders take stock of the cloud security landscape on the CISO Tradecraft podcast

Chris Rothe
Originally published . Last modified .

Red Canary Co-founder Chris Rothe recently stopped by the CISO Tradecraft podcast for a wide-ranging conversation with host G Mark Hardy about the cloud security landscape, the evolution of managed services, the cybersecurity talent gap, and much more.

Watch the interview below or read the transcript, which has been edited for clarity.

 

 

G Mark Hardy:

Hello, and welcome to another episode of CISO Tradecraft, the podcast that provides you with the information, knowledge, and wisdom to be a more effective cybersecurity leader. I’m your host, G. Mark Hardy, and today I’m going to talk with a very interesting fellow with whom I’ve had an opportunity to work with for a number of years.

Chris Rothe, who is with Red Canary, and not only with Red Canary, but was one of the founders. Now, a little bit of disclaimer up front is that I’ve been a customer of theirs for a while. And as a result, I am quite familiar with it. From time to time, you might even see this Red Canary swag back here that they have as a jacket.

I have participated as an uncompensated member of their CISO Advisory Council, and was even on one of their CISO Debate Series. So, Red Canary has been around for quite a while in terms of just adding value to the community. I think it’s one of the better companies that I’ve seen out there.

Although it’s not an infomercial, as you guys know, this is an educational program, but we’re going to talk about cloud: cloud security, how to manage that, how to monitor that, and the direction that we’re going in the future. So with all that, Chris, welcome to the show.

 

Chris Rothe:

Well, thank you. And, thanks for the kind words. Appreciate it.

 

Red Canary’s origin story


G Mark Hardy:

I think you’ve heard them. Can you tell me a little bit about, or tell our audience a little bit about yourself and what got you into this and how’d you end up co-founding Red Canary?

 

Chris Rothe:

Sure. Well, I’m an electrical engineer, loved computers since I was very young and ended up starting my career in the national defense space. I worked for Lockheed Martin for a number of years. Through a series of very fortunate things in my life, I ended up connecting up with two other gentlemen who wanted to start this company.

We worked inside of a company where Carbon Black was also incubated and spun out, so kind of early in operationalizing endpoint detection response. I was just really fortunate to meet the right folks at the right time and bring some of the practices and operational techniques that we used in the national security space into the commercial sector to help organizations.

So that’s a short version of the story.

 

G Mark Hardy:

It sounds like what we’re talking about here is being able to take an expertise you developed in some sort of a work role, but at some point saying, hey, we’re going to leave the nest and then we’ll do our own thing. Which brings me to the question: why the name Red Canary? It’s kind of cool and I love the logo, but why Red Canary?

 

Chris Rothe:

Canaries were detection devices for miners, and we’re a detection and response company. It fit from that perspective. Red is meant to communicate a level of seriousness. We all know that false positives are the bane of existence in security, especially in the MSSP space and MDR space. So, the connotation is a red alert. If you hear from us, you likely need to take action. That’s a little bit of a nod to all three of our backgrounds in the national security space.

 

G Mark Hardy:

Well, it sounds great. And for people who aren’t familiar with the role of a canary, the miners didn’t take them down to the mine shaft because they like to listen to the noise. Canaries are very sensitive to oxygen loss. And if the bird’s singing away when you’re there, chipping away down there like the seven dwarves, you’re fine. But when the bird goes silent, you get worried. And when it’s an ex-canary, you better get out of there because there’s a problem. And so that’s where the term came from.

 

Chris Rothe:

Yeah, I used to try to suggest to our marketing team that we should use the tagline: “let us die for you.” But that never really caught on.

 

The view from the cloud


G Mark Hardy:

Yeah, I can see that. Give your marketing team some credit. Maybe something like “stop chirping” or whatever, but I’m tempted to think of a “Monty Python” skit with the ex-parrot and things like that, but we’ll move on from there.

When a lot of us started working in IT, there really wasn’t a cloud. And today, I don’t know of any organization that can operate without it. But, unfortunately, a lot of the investments that we made in security technologies over the years for the enterprise—witness my piece of artwork here, my old four XP machines—don’t scale into the cloud. A lot of the enterprise solutions are fixed solutions.

When some of us started doing security, it was really endpoint focused. I’ve been around a while, and I remember when VirusScan came out in 1987. It was created by a guy who used to be a programmer at NASA, worked for Booz Allen in the early to mid 80s, and then he was an employee at Lockheed Martin. Sounded familiar. He read about the Brain computer virus and decided, hey, there’s a business opportunity here. And the rest, as they say, is history, but unfortunately so was that founder, who was John McAfee. So this industry goes back quite a while, but we’ve seen sort of a generational approach to security in the cloud.

We’d set up with the CIS benchmarks, for example. At this, the Center for Internet Security had the top 20. We’re now at 18 critical security controls to do things like stop configuration attacks. It lets you make sure that you’ve set your defenses correctly, but in a way you’re just positioning your chess pieces on the board.

It’s going to miss things like identity abuse. If somebody goes ahead and grabs a real credential, and then starts doing something with that real credential, the traditional approach doesn’t work. So, the next level we went into is anomaly analysis, where we look for known attacks. A DDoS attack or some DLP alert goes off saying somebody is trying to exfill your entire customer database out. You want to be alerted on that.

But we’re getting now into a more mature approach for an EDR, if you will, in the cloud environments. If somebody steals a single session cookie for AWS and you’re there doing the monitoring, what would you observe? With respect to, let’s say, an S3 bucket exfil through a valid user’s access.

Is that enough to set off alarms, or do you just miss that until all of a sudden you get a little note saying, oops, we own all your files?

 

Chris Rothe:

Think of the cloud control plane as an operating system. That’s the analogy. If you think about a traditional endpoint, it’s got networking services, storage services, compute services, the same things you get by way of the cloud environment.

From that perspective, there’s sort of a rhyming that happens there. And core there is: what do you use to monitor the things that are happening on that operating system? In the cloud, it’s not EDR telemetry necessarily for that control plane. Instead, it’s the activity logs of the cloud platform.

So in AWS CloudTrail, or in Azure and GCP, it’s effectively the API telemetry, everything that someone is doing, someone, or an automated script is doing against that control plane. And the good news is that those are readily available. We don’t have to install any third-party technology. They’re provided by the cloud platform. So that’s a leg up over what we had traditionally with endpoints where we had to jam in kernel filter drivers and all kinds of stuff in order to get the telemetry that we needed at that operating system layer. So that’s kind of the approach.

So with your approach of someone stealing a session token and accessing this account and downloading data out of an S3 bucket or maybe making it open publicly, that’s something that’s observable by way of the telemetry that comes through the activity layer.

Think of the cloud control plane as an operating system.

 

G Mark Hardy:

And that’s a very good point, because for the most part, the logging that’s available in apps today will let you know what’s going on, but they’re not letting you know in real time. It’s great when you have to do a report to go into the board and say, well, at exactly 2:13 AM on Sunday is when we went bankrupt. Because that’s when all of our crown jewels got exfiltrated out the backdoor.

Not very helpful, though, in terms of being preventative. And so for organizations that have the time, the money, the resources, and the demand for it, they can stock up an entire security operations center (SOC). And you get a big SOC running, and it’s your own private SOC, well that’s great. Because now you’ve got analysts, you’ve got people that are working there, and they’re there 24×7. But even then, the issue is: is there enough to keep them interested? We’ll come back to manpower in a little bit, but look also at the idea of working with these third parties.

Way back in 2000, I worked for a startup MSSP called Guardant. And I remember thinking, why would anybody allow their most sensitive information to be shared with and monitored by a third party? Well, the answer of course is that a lot of organizations can’t scale their security function as fast as they can grow their IT environment. Remember, this is just before the dot com crash, and we survived.

I had another job opportunity that I did not take. It was mostly because my wife didn’t want to go out and live in a smaller house out in Silicon Valley, but that was to go work with Bruce Schneier at Counterpane at the time, to run his managed security services. I helped Bruce review one of his books, Secrets and Lies.

We get together from time to time, and he’s done some amazing stuff in the community with his writing and his blogging. I can’t say enough good things about him. But what I found out there is that by creating this MSSP, those gaps that are going to be unable to be monitored can be filled by somebody who knows what’s going on.

Now, what is going to motivate a company to pull in a third-party provider as compared to just doing it yourself? And then how would a CISO know when it’s the right time with the growth of a company or the complexity of their IT infrastructure to know that they ought to explore that option?

 

The choice to outsource security operations


Chris Rothe:

Yeah. In my mind, there’s a pretty simple trade off, which is: Hey, we know as a business that we have a lot to protect, whether it’s our money, it’s our IP, et cetera. and we’re not a large enough company to justify the security expenditure to staff up a full security operation. If you think about what it takes to actually build out a 24x7x365 detection response capability—to run three shifts of  8-12 people—that’s pretty out of reach for most midsize and smaller organizations.

And frankly, they won’t have enough to do. There aren’t that many things to monitor. Do you really want to staff that team and have them stare at empty dashboards most of the time? So that’s one of the benefits of the MSSP or MDR model. Being able to concentrate those resources so that the same 8-12 people can operate across 10, 100, 1,000 companies as opposed to just one, and you get some of the herd immunity across that as well. So that’s the traditional reason: Hey, we know we need security, but we don’t have the budget or the desire to build out a full mature security team in-house.

 

G Mark Hardy:

Got it. I just, I’m writing down a little concept, a herd of canaries. It’s interesting.

 

Chris Rothe

We use the word “flock” internally.

 

G Mark Hardy:

A lot of us talk about MSSPs and MDRs and things like that, but sometimes skip past that. So for the benefit of our listeners who think they might know, but aren’t absolutely sure, are embarrassed to ask: how would you explain the difference between an MSSP and MDR?

 

Chris Rothe:

Sure. We’re on this bird thing. Their genus and species are’t too far away from each other. They’re in the same family, but the traditional MSSP capability was: We’ll run your firewalls for you. We’ll run your antivirus for you. We’ll make sure it’s running. Maybe we’ll be the help desk when you need to make a firewall change. We’re there with an incident response service if you do have a breach. and then we’ll do tier one triage of your security alerts. So when something goes off, we’ll do the basic sorting and filtering. Maybe also we’ll run your logging program. We’ll store your logs for compliance purposes or whatever else.

The way we like to think about it in modern times is it’s a pretty thin, but broad layer of security for an organization. And it’s important. There’s certainly a place for it for companies that don’t want to run any of their own security infrastructure.

The difference between that and MDR from our perspective is that if you’re going to hire someone to do detection and response for you, their goal should be to detect everything that is bad in your environment, and then take action on it or help you take action on it. So it’s a narrower service. It’s more focused on detection and response. We’re not going to run firewalls for you. We’re not going to do your vuln management program, those types of things. But we are going to be super deep in detection across the areas that matter most in your environment.

Where did that come from? That came from the early 2010s when companies had gotten pretty good at securing their perimeters, pretty good at basic security programs, but breaches were still happening because it was relatively straightforward for an adversary to get a payload onto an end user’s endpoint, and then pivot from there into other parts of their environment.

So that really led to the rise of EDR. And then, to make use of the vast, vast telemetry that EDR creates, the MDR layer was important on top of that. That’s the difference between MSSP and MDR: broad and thin versus narrow and deep.

That’s the difference between MSSP and MDR: broad and thin versus narrow and deep.

 

G Mark Hardy:

So for a doctor, it’s a difference between a GP and a specialist. And if you know that you have a concern, this is great. But I want somebody who really practices this, who does it day after day, and that’s their expertise. And so I think of an MSSP as great if you have little-to-no resources, but it really doesn’t give you the follow through. They will help point you to the log entry that said you went bankrupt at 2:13 AM on a Sunday, but as a managed detection or response service, you’re actually going to be creating the alerts and not just telling people what’s going on, but doing something about it. As I had mentioned before at the beginning of the show, I’m a Red Canary customer and so I have things set up where I will get an alert.

And just as a little nod to certain video games, when I get my alert, it says “This is Red Canary calling. Wake up and smell the ashes. Something has just caught fire in your network.” And it works, and I’ve done a couple tests, and I found that the response rate is sometimes frighteningly fast. It’s like, wow, these guys are just sitting there, men and women, doing it.

But if we think about it, if all you do is ingest security alerts and then filter some out and send them back to you. You basically say: hey, here’s, here’s your alert stack and here’s the thing you care about. That’s kind of great, but it sounds like you go beyond that with an MDR. You’re actually doing some analysis and putting a brain onto it, saying yeah, not that one, or this one didn’t rank high, but something’s low and slow.

How do you differentiate, how do you get people to actually figure that stuff out? Is it all by AI now? Is everybody AI or do you just have a lot of smart people?

 

Chris Rothe:

I should say about MDR: there’s not a single definition and there isn’t a single approach that all managed detection and response vendors take. Our approach is really to take the idea of we’re not going to let you get breached and make that promise to our customers. That means we have to find threats that your other security products don’t. And so to us, it’s the mix of the detection products in your network, in your environment already. They send over alerts. We want to make sure we look at those and correlate them and filter down to the ones that are really important.

That’s one leg of it. But honestly, that’s the shorter leg. The stronger leg is detection analytics based on the source data for those environments. So endpoint detection, response, telemetry, the activity telemetry on every endpoint that’s fundamental in your traditional endpoint environment, which also extends into the cloud.

I mentioned before monitoring the cloud control plane, but it’s also super important to monitor your workloads in the cloud, the things that look like endpoints, running on Linux or Windows or whatever. We need to do the same detection response there. And then moving over into the identity space, monitoring the login and activity telemetry there to make sure that we can detect impossible travel and other things like that compromised identity.

So, from our perspective, it’s the alerts that come out of your security products. That’s great. It’s great context, but we apply our own detection analytics to really drive out the vast majority of these threats.

The reason sometimes people have a hard time understanding this: Well, wait a minute. Why wouldn’t the security product or the detection product do a better job than than you at detecting things? The answer is that all those products have to be tuned to a certain level of false positives, right? If you’re a piece of security software and you are just bombarding your users with lots and lots of alerts, and a lot of them are false positives, pretty soon they’re going to get dissatisfied with your product and turn it off or churn out as a customer.

We have this advantage because we have a security operations team that is in seat 24×7 working in our platform, which allows them to do their job and do their investigations really, really efficiently because we can crank up that sensitivity of detection to 11 and really look for those things. A lot of them are false positives, but they’re false positives that don’t get to the customer. They land on our team to investigate and decide what to do with. And so that’s the model.

You made a medical analogy, here’s another one for you: if you think about testing for diseases, let’s say, say cancer, there are tests that will determine if you have cancer, but a lot of time those don’t get run because they’re not specific enough. Like they may be 50 percent accurate and you don’t want to run a test on somebody and say, Hey, there’s a 50 percent chance you have cancer, unless you have some other indicator. So that’s the way to look at it. We’re happy to run those 50 percent tests all day, every day and deal with the baggage that comes with that, whereas most detection companies are like, no, no, no, we don’t want to deal with that. We only want to deal with the highly sensitive specific tests that tell us that it is cancer. Kind of a weird analogy if you can follow it, but that’s the way we think about it.

 

G Mark Hardy:

I get that. And again, for those who just came back from RSA, you go there and see the phonebook-size directory of all the different vendors out there. And everybody’s suddenly got a little bit here, a little bit here, a little here…these days it’s almost impossible to completely fund the entire vertical stack.

Okay, Apple did it years ago and they created their own ecosystem. But you don’t have a closed ecosystem, do you? If I already have an EDR solution that I like, like a Carbon Black, or if I’m using Microsoft and using those tools, like Defender, any reason I can’t just plug them into your system and let you guys watch? Or does that require a whole bunch of effort to get there?

 

Chris Rothe:

No, I mean, our approach is that we support the products that create the data that we need, right? And so in the EDR space, in the endpoint space, whether you’re using Carbon Black, CrowdStrike, Defender, SentinelOne, Palo Alto, all of those are supported by Red Canary. The bar for us is: do they create telemetry? Do they actually create telemetry at the level that we can apply our detection analytics? Cause there’s a long tail of EDR products out there that just don’t. So that’s the criteria there. So certainly if someone already has it, bring it in. If you’re looking for it, we can help you select one. If you’re looking for that new thing, we can give our opinion on where you might want to go. And then similarly, in the identity and cloud space, we support the major cloud vendors, and the major identity platforms. That’s our view of the world.

We don’t want to create a bunch of our own sensors and our own technology that forces people to deploy or force you to use our one specific technology. We look at it as a best-of-breed solution.

 

Challenges in staffing


G Mark Hardy:

And that makes good sense. Since some organizations have already invested in these technologies, it’s great that they have them, but again, there’s no way to follow up. If you’ve activated Defender, I’ll wake up in the morning and I’ll get some emails saying oh, we detected this, we detected that, and I get so many false positives from there, I’m not even going to worry about it. But when you do get a bad one that comes in, how do you get that to break out of the noise?

It sounds like by using a managed detection and response service, you’ve got somebody else who’s going to be up at two in the morning anyway. It’s part of their job, they’re working the night shift. They go: yeah, not that.

Today I’m interviewing people for a job. I’ve got nine interviews scheduled, trying to find somebody. Anybody out there wants to do IT support in Washington, DC, let me know. One of the things I’m finding is that there’s a lot of job openings, but the ratio of job openings to people in the industry is a little bit different than it is in some other lines of business. Where do you think we stand from that perspective? And is that holding organizations back from just saying, we’re going to do it ourselves, and actually driving business toward organizations like yours, where you get to, if you will, be the fire department and manage security for a bunch of folks?

And then does that even attract the best people? Because they want to see all those different varieties of things instead of just like the Maytag repairman waiting for something to break.

 

Chris Rothe:

You’re hitting on some key themes of MDR. One way to look at today’s security challenges is that if you zoom out several levels, they’re aren’t enough qualified people to do information security for the businesses and in this world.

You mentioned staffing; that’s one thing I like to track is how many open job recs are there in information security versus how many people are actually employed in information security. Right now the ratio is like 0.5 to one. So every person employed in security could have another half of a job.

It’s a pretty unhealthy ratio. A healthy ratio of supply and demand for jobs like sales or software engineering tends to be sort of five to one to seven to one. and so we’re, we’re like 0.5 to one. If my math is correct, that’s not really sustainable. The core problem is that even if you have the budget and the buy-in from your board to go build out an 8-12 person team to monitor your environment, the chances of finding those qualified people is pretty tough. And the likelihood that they’re going to leave pretty soon after you hire them is also pretty high.

In our world, we don’t have challenges keeping security analysts on our team because they love—like you said, working across a lot of different customers—but they also love working with security people really closely in a company that values security.

If you’re going to be a security analyst for a bank or something like that, you’re not necessarily someone there that everyone is excited to work with all the time. Sometimes you’re the Department of No, and it’s super valuable, super important. And we love those people. But there is a different side of it where you can work for a security company. So for us, the staffing problem isn’t as acute as it is for a lot of organizations. And that is fundamentally why you look to solutions like MDR.

The other aspect of it is, that in the end, the threats that we all face.are largely the same. There are certainly sector-specific threats and and organizations need to take care of those. But at the fundamental level, most people are using Windows, Mac, or Linux for their endpoints; GCP, AWS, or Azure for the cloud; Okta or Microsoft for identity; your productivity suite’s going to be one of a small number of choices. And so what that means is that an adversary’s attention is focused there. The threats that land in those environments are common. In a lot of ways, doesn’t make sense for most organizations to go out and build their own bank of detection analytics. That’s going to look exactly like the set that the company down the street does. You can allow folks like us to be that common layer, and then you can move up the stack and focus on more strategic things, more business-specific things. So that’s another way to look at it.

 

G Mark Hardy:

You mentioned this earlier from a staffing perspective, because I used to stand watches in the military. And so when you had a 24×7 watch, 365 days a year. You do the math and you divide by 40-hour weeks, but it doesn’t really work that way.Because what you’ll find out is if you go 40, 80, 120, 160…but you’ve got a couple weeks vacation, so at a minimum you need five people to keep a single desk seat warm. And even that’s pretty intense because you don’t get any flexibility. You can’t go off and do your conferences and your training, and when you throw in all the mandatory stuff thatcorporations throw at you for activities, especially in the military, it’s almost six to one.

Well, okay, that’s fine, but then we got this double multiplier on there and that is the cost of actually having someone on a payroll. And so somebody comes in at, let’s pick a random number, $100,000 a year. You’re going to have to pay the FICA and the social security on that and the Medicare. Then there’s going to be unemployment, which is not that much necessarily, but it depends on what state you’re in and your loss experience.

But then healthcare, which could really be a killer, depending upon what premiums are, plus the fact that I keep the lights on and you issue box of business cards and you have coffee in the coffee machine and things like that, and when you add it all up…I know that when I was at Booz Allen years ago we thought a minimum multiple was about 1.7-something, that is to say it would cost us 170 percent of a salary just to break even. So if we were billing out at 175, you are on a razor-thin margin and you better have nobody show up late for work, nobody miss a day, nobody quit, and no interruptions. The reality is you need more than that.

And a lot of people don’t necessarily think about it that way. I’ve talked to executives who say, Oh yeah, it’s like 1.1, 1.5, , 1.15. No, it’s not. And so to a certain extent, when you have a job, it’s like being in a fire department, where not all the houses are going to catch fire at once. It’s perfectly adequate to have one fire department that’s looking out for a whole neighborhood.

And if it has to, you can ring a second alarm and bring somebody else in, and stuff like that happens. And so what you’ve articulated here is a value proposition of not only being able to attract some of the most qualified people because they got an interesting environment to work in, but you retain them. I’m going to suspect, but I can’t speak for you, that you’ve got probably a lower than average turnover right there. Because it just sounds like a really cool job to be doing.

 

Chris Rothe:

We pride ourselves on building tools that make analysts really efficient. We also have sort of a unique model in our security operations center. We don’t actually call them “analysts.” We call them “detection engineers,” and that’s a really explicit title because not only are they investigating potential threats and writing them up for customers, but they also are meant to look at those threats and say, what’d we miss here? They then build out new analytics that immediately get fed back into the, into the engine. So, there isn’t a separation between the people who write the detection and the people have to deal with the alerts. It’s a cohesive unit. And that’s a really attractive job to a lot of folks who maybe have been in a SOC and were just kind of like been banging their way through alerts for a long time.

So, that’s the way we think about it. Love our security team. They are the beating heart of our business. We don’t necessarily treat them like royalty, but they are what ends up delivering the value to our customers. And, the fact that we can deploy them across a wide range is really valuable.

 

G Mark Hardy:

And again, I’ve had good interaction with these folks. It’s like, you give them a well done, and oh, by the way, you get a follow up, how did we do, and things like that. And this is in general for people who are listening to the show: If somebody does a good job for you, whether they work for you, whether they work for a company, you get these little surveys that come in and I don’t want it.

I don’t have time to answer a 37-question survey about how good was your support. But if they said happy smiley down to frown and then click the happy smiley face and said, great, we’re glad you love it. Optional. You don’t want to say anything more. Nope. Submit. That’s a good feedback loop. It keeps it simple.

You’re likely to get a second and a third and fourth response because they know it’s not going to eat up a lot of time. And, that works out well. So I like what you’ve done there.

Another cool thing I think you’ve done at Red Canary is creating the Atomic Red Team. And that of course has been a really cool concept, and I’ve got the t-shirt somewhere in here. What was the idea behind that when you started it and what does it do for people today?

 

Shout out to Atomic Red Team


Chris Rothe:

Well, brief intro. Atomic Red Team is an open source project. Around 2017, we noticed people would try to test our detection capabilities and proofs of concept, even customers that were working with us. The typical pattern was to go VirusTotal and download 50 malware samples, run them all on the same endpoint, and then come back to Red Canary and say, Hey, you only caught 12.

But that’s not how adversaries work, that’s not really a realistic test. We needed to change the way people thought about that. And so the concept behind Atomic Red Team was: let’s make it super easy to run these really small tests in any environment. For most of them, you can open up a command prompt, copy/paste out of Atomic Red Team into that command prompt, run it, and that’ll test some technique.

Around the same time, maybe a little bit later, MITRE ATT&CK came along and that excited us a lot because we had our own sort of taxonomy of classifying threats and techniques and now we were able to use one that was out there in public and well maintained, thanks to all the great work of that team. So we refactored Atomic Red Team to be aligned to ATT&CK techniques and and from there it’s just grown.

Red Canary is still the core maintainer of Atomic Red Team, but we have others from large security teams, large companies, and many others. It’s got a really, really broad community that supports it and a lot of derivative product projects around it to automate the execution of tests and things like that. It’s really a cool community started really just from our own need to help organizations test us better and, and more so than us, test their own security.

 

G Mark Hardy:

It reminds me of antivirus. They created the ICAR with the European Institute for Computer Antivirus Research, where you’d have this character string and you submit that and then all antiviruses are going to agree that, Hey, we will detect that. And so then, what you can do is alert on that. But what you’re saying is sort of a similar thing where you could go ahead and do some tests.

Kind of a quick side note: I remember a couple of years back doing a test where we used one of those testing tools that had all these scenarios.I’m not going to mention the company’s name. If you want me to mention it, they can come and put out their own episode. But what we did is we spaced out a number of tests over a period of time. And then we would get responses saying, Hey, we alerted on that. Well, what I thought was interesting is last year we had an intern in, and she wanted to go ahead and run some tests. And I said, well, here’s the test tool that we have. We have a license for it. And if you run these tests, then what we should be doing is getting all these alerts from Red Canary. And what was interesting is, I don’t know whether she just didn’t update the database because she wasn’t familiar with it, but she went, boom! Instead of doing this once every few minutes, she just fired off 20-some odd attacks back to back to back to back. And it’s like, oh my goodness, you should have lit up the switchboard. And I got no alerts. And I’m thinking, oh, come on, what’s going on, right? I’m getting ready to call up and say,, Whiskey Tango Foxtrot.

But I checked my logs and it turned out that every single one of those attacks had been blocked by Microsoft. So the test environment has to continually update. You can’t use last year’s tests this year. As you point out, it’s not a realistic type of scenario for an attacker. But more importantly, it can give you either a false sense of security, saying, hey, it blocked everything.

Why do I need an MDR service when my tools are working perfectly? And yet we take a look at the 8Ks that get filed by companies, we find out that, yeah, there are a lot of things that do get through, and we have to be careful about that.

The other thing I think that you produce is of great value, and I’m comparing it to the Verizon DBIR, their go-to document every year for breach information. This year’s report I read through, it’s a hundred pages. Part of Red Canary publishes an annual threat detection report, which, in my opinion, is a lot more relevant to CISOs because it shows the threats to their own enterprise and not just necessarily something that went wrong somewhere else, that might have gone through a mechanism that you wouldn’t even have been exposed to. Plus, you’ve got embedded music clips for enhanced reading. I noticed that. That’s pretty cool.

What do you think are the most important insights that your team has found this past year and any top-level recommendation you could offer for security leaders?

 

Scrolling through Threat Detection Report


Chris Rothe:

First and foremost, the Verizon DBI report is a legend and I read it every year. It’s super useful. So, I certainly don’t want to compare the two and say ours is better, but we’re really proud of what we produce on an annual basis with the Threat Detection Report.

 

And the idea of it is it just tells you about the trends and everything that we saw over the last year of detection in our array of customers. I saw a Twitter back-and-forth happening when we launched the Threat Detection Report this year about how PowerShell continues to be a very, very common technique that we see leading to threats. There’s a discussion on X, I guess, about how, well, wait a minute, I thought PowerShell was dead, and that led to some really good back and forth that we saw out there. While the defense and preventative tooling have gotten better at stopping it and better at disabling things that it can do, that doesn’t mean that it’s necessarily dead. It’s still something that can get through if you don’t have things configured the right way.

So the way I think about it is, as the detection and prevention and all that tooling gets better, that’s great, but it doesn’t mean we just go turn off all that detection that we’ve built up over time. Like all those things are still relevant. Those techniques are still relevant. And at some point, it’s like, okay, cool, but we need to make sure and detect the next set of things for sure. But if you don’t have that baseline of everything, at some point, somebody is going to accidentally leave a machine open, not have it configured in the way that the woman working for you did and it is going to get through.

So anyway, what I like about the Threat Detection Report is that it’s practical and it’s like, look, we’re not trying to highlight just the most advanced techniques here. We’re telling you this is what is getting through. This is what got through in the last year. Here are the different ways that adversaries are using it, different examples, and then practical advice about what you should do in terms of detecting it. So, I think it’s a really valuable resource.

For our customers, we certainly don’t wait a year to share these type of insights with them within the Red Canary product. You’re going to get intelligence insights, intelligence profiles, bulletins, things like that, on a pretty regular basis.

 

G Mark Hardy:

And I get those. Monthly I get to see: here’s Mimikatz. It’s still here and it’s still doing its thing, scraping stuff out of memory and things like that. And, I believe you reported like 60, 000 different events that were all kind of consolidated in there, so a huge sample space, which is in a way, a little bit frightening, but it’s also reassuring that we’ve got that much to look at, which means the trends that you’re seeing really are genuine macro trends. It’s not, hey I’ve got a very small sample space, I’m two guys and a dog, and therefore this is all we see is what’s out our back window.

You’d be able to look at things like that. As we’ve migrated from endpoints to basic cloud, now we’re going to serverless computing, containerization and things such as that. How is that impacting our ability to defend that? Can you actually get the equivalent of an EDR tool into a serverless Lambda function or some container that’s up there in Kubernetes and actually report back? Are these things too ephemeral to be useful as a reporting tool? And if so, how do we protect them?

 

Back to the cloud


Chris Rothe:

I think each technology has to be looked at on its own. And the question to ask is: What does it actually mean to compromise a serverless function? What does that mean? For that one as an example, I think what it means is someone got access to your cloud control plane. They edited that serverless function and they injected their own code or they created their own function or whatever. So, when you think about it from that perspective, you say, well, how would we detect that? Well, you don’t necessarily have to look at the code. You might want to, for context. And you don’t necessarily need to inject an EDR that says, Hey, this serverless function did these things. You can start with just detecting that somebody edited this serverless function and it’s someone who’s not done it before, or it’s an unusual pattern for your organization. So I think the way I look at it is: what does it actually mean to compromise these things?

Whereas traditionally, we think of it in terms of, hey, you’ve got access to an endpoint and then you persisted and you moved laterally and that kind of stuff, that pattern doesn’t necessarily hold when it’s an ephemeral thing. Similar story for a container. A container spins up, if it’s a short-lived container, and the compromise isn’t injected in the source code or injected in the base image or anything like that. Well, assuming it went away, that really didn’t give the adversary any access.

 

G Mark Hardy:

Unless you’re using it as a jump box or something like that to create some persistence somewhere else. But you would see things like that with all the data that you look at and these new and evolving threats.

Ten years ago when I started a fraud prevention company called Cardkill, our tagline was we can preemptively kill stolen credit cards before they get used in fraud. And the reason was we looked at a lot of information and we could detect these really faint signals and break them out from the noise where a single bank would have too much noise to see it. And what would have been successful for that was to be able to get enough data coming in to do it.

But for other reasons that were not part of the show, it turned out that that was a problem I think that some banks just didn’t want to solve. In any case, you’ve got that signal and you’ve got a good signal-to-noise ratio. And as a result, it seems that you’re on top of spotting trends. You can spot the top ways that attackers are working.

And from that though, I’m wondering, do you have any predictions? What do you think our cloud security, both the defenders, as perhaps even the attackers will do in the next few years? And what can we better do to position ourselves to protect the future?

 

Chris Rothe:

If you just think about the cat-and-mouse game, that is, information security adversaries go to the weakest point and the weakest thing where they can make the most impact or steal the most, whatever they’re trying thereafter. Right now we would say that endpoints and identity are still the softer targets, and so for now, that’s sort of a huge focus. Cloud is the most emerging, and it’s funny to say that because the cloud has been around for whatever, 15 years at this point. Certainly there’s been lots of attacks in the cloud, lots of data loss and things like that, but the control plane has not yet been the focus of a lot of adversary attention. We’re seeing that change. The way to think about it from my perspective—more as a technologist than as a security person—is that if an adversary gets into my control plane, they can spin up as much infrastructure as they want to do whatever they want.

So, less so, but I’m certainly worried about our data that’s sitting in our S3 buckets and sitting in our databases and all that kind of stuff. I’m also worried about them spinning up a huge bank of cryptominers in an AWS account that they create that maybe none of our team even sees. And I don’t know about it until my next billing cycle when they say, Hey, you’re running a million GPU instances over here.

Now, obviously the cloud providers have detection that allows them to find that kind of stuff, but that’s the way I tend to think about cloud threats right now, especially cloud infrastructure threats. Once they get in, what can they do in terms of leveraging your bank account, your infrastructure? As opposed to treating it like a traditional network security use case of protecting the data. You gotta do that too but the new part of it is: what can they do if they can stand up as many instances as they want?

 

G Mark Hardy:

That’s a good challenge. And I think that means we’ve got to be monitoring that and have somebody else who knows what they’re doing to look over your shoulder for you, particularly when you’re trying to do something else, like get on with your life. Someone else is doing that as part of their routine.

So we’re getting close to the end of the show. Are there any follow-up thoughts you had, or if someone says, Hey, this is fascinating. I want to learn more about these guys. How do they do so? How would they approach that?

 

Say hi


Chris Rothe:

Sure. redcanary.com would be the best way to get in touch with us. Highly recommend if the Threat Detection Report stuff was of interest. It’s all out on the website. Lots of material around that, lots of video being created, and other things to make it really digestible.

So I think that’s a great callout, come interact with it. We love talking to security people. We’re not going to try to jam our product down your throat and sell it to you. So, reach out and have a conversation with us. We’d love to help with whatever we can.

 

G Mark Hardy:

Well, I love the fact that even as a founder, all these years later, you still got the technical chops to be able to wade into a podcast like that and talk about things, so good on you. And I think it speaks highly of the way that you run the business, but both because my interaction and just as somebody who’s run a bunch of things, I appreciate your leadership style.

And I think your people do too. So for our listeners or watchers, if you’re watching us on YouTube, this has been Chris Rothe, one of the founders of Red Canary. And of course, I’m your host, G Mark Hardy, and we’ve been talking about cloud, cloud security using a managed security provider versus managed detection and response. MDR, I think is really the more complete model that we talked about where it’s kind of deeper into that and the like. We’ve looked at some of the issues with regard to staffing your own people, just trying to find qualified candidates and attract and retain them. And there’s always job offers someplace else. Someone’s going to try to buy them away for another $5,000. And plus the cost of just maintaining a 24×7 watch is quite expensive. And then also we took a look at some of the different tool sets and reports out there, the Threat Detection Report, which we’ll put a link to that in our show notes, but also things such as the Atomic Red team that you’ve done a lot of above-and-beyond contribution to the community, which makes me kind of privileged to have you guys on the show.

 

What we learned by integrating with Google Cloud Platform

 

Shrinking the haystack: The six phases of cloud threat detection

 

Shrinking the haystack: Building a cloud threat detection engine

 

Red Canary’s best of 2024

Subscribe to our blog

 
 
Back to Top