Once upon a time there lived a boy named Benjamin. Benjamin was very smart, and grew up with a passion for Information Security. As an adult he became part of the InfoSec team at “WidgetCo,” whose highly-prized widgets made their network and computing infrastructure a constant target. Benjamin was constantly making recommendations to help the organization defend against a barrage of sophisticated attacks, as widget-enviers desired to steal WidgetCo’s intellectual property.
Part of Benjamin’s role at WidgetCo was making recommendations for the InfoSec spend, so that those approving the budget and signing the checks would understand the value being gained. WidgetCo was attacked so frequently — and by such skilled (one might say “Advanced”) and determined (one might say “Persistent”) malicious actors (one might say “Threats”) — that the organization suffered compromises, despite their best efforts at prevention. And, as their internal resources were limited (by both time and expertise), their typical approach was to bring in an external Incident Response team to help drive their IR efforts.
Benjamin’s Conundrum: “What Value Are We Really Getting From Incident Response Retainers?”
Benjamin saw an issue with their spending. Outsourcing their IR was very expensive, especially considering that no work would actually occur until a breach was discovered. And having a third party on retainer was not only expensive and reactive, but it was also a substantial layout every year. They could be certain the IR firm would pick up the phone in the event of a breach but he was not sure that was worth the expense. While the money was spent proactively, it was used reactively. Granted, at the end of the year, they could use leftover fees to do a pen test or roundtable exercise, but that was only to get some value out of money already spent!
Benjamin really felt that their money was just going down the drain, but at the same time, they couldn’t justify the cost of dedicated internal personnel for threat hunting and IR.
Where does that leave our intrepid hero, besides standing there watching money go down the drain?
The Moral of the Story…
If you’re anything like me, you can probably relate to Benjamin’s dilemma. He knows they need IR capabilities, and have limited resources. He wants to do what is right for his company, which also includes being a diligent financial steward (for spend, not just trying to prevent losses). As a long-time security analyst and forensics investigator, I have had the opportunity to see this from multiple angles, and have some sympathy for the devil (in the details).
So what can Benjamin do? I see four options:
- Internal Incident Response
- Incident Response Consultants
- Incident Response Retainer
- Managed Detection and Response
Let’s take a closer look at each option…
1: Internal Incident Response
Working corporate InfoSec, I was able to build up and lead an internal team for digital forensics, incident response, and eDiscovery, and had a great time with it. It’s very different to navigate corporate politics from inside, as opposed to coming in from outside working for lawyers. I learned a lot and found it very rewarding; it was also very challenging in many ways.
Key lessons on internal IR:
- It’s expensive. Not to beat the dead horse with regard to talent shortages in InfoSec; we all know about that particular “threat landscape.” As a result, finding and retaining the right kind of staff is expensive and time-consuming. In addition, you have to have someone internally to head up that process—someone who knows how to target that talent, organize the team, and build out the processes and infrastructure.
- It’s rewarding. As mentioned above, this can be very rewarding for everyone on the team; there are opportunities to learn and grow, and do amazing things for the business. You get to do the fun, cool, geeky “blue team” stuff that you’ve always dreamed about, while defending your company assets.
- It’s challenging. The dark, soul-sucking side is what we’re talking about here. Wading through corporate politics to do this internally can be rife with landmines, and in some cases, career-ending. In some environments, it is downright impossible because elements of the business are unwilling to commit, or to see the benefit (as opposed to just the cost).
- Being proactive has huge benefits. Being on the inside and having ready access to data can make detection and response more timely and efficient. It also affords the opportunity to engage in “hunting” activity to move the needle even further forward. With a capable team, you can get ahead of the attackers in many cases to help stop attacks and mitigate damage.
- You should always measure success. If you are building out an internal team, undoubtedly you will need (and you should want!) to measure the success of that venture, report such metrics to the business, and show improvement over time. This is important to combat the “cost center” mentality associated with InfoSec in general, and specialized teams specifically. (TIP: To learn actionable ways to report the effectiveness of your security program and tools, check out this upcoming webinar with Red Canary’s SOC leader, Joe Moles.)
It’s entirely possible to start doing threat hunting/IR activities on virtually no budget, and prove the worth as you go. So if you think it’s impossible, don’t give up on the idea just yet!
For more on threat hunting without a budget, watch Frank McClain’s SANS webinar: Threat Hunting for the Masses
2: Hiring Consultants for Incident Response
I’ve walked down the consultant path, and have an understanding of that scenario. When you don’t have the internal resources (due to constraints of personnel, time, or expertise), or the ability to satisfy business/legal requirements, it’s common to turn to consultants. There are some benefits there, as well as challenges; following are just a few of these.
Key lessons on hiring consultants for IR:
- You can quickly reduce constraints. A consultancy can fill in the gaps of personnel, time, and expertise, in addition to fulfilling other requirements. This is probably the top concern for many businesses. Keep in mind that not all consultants are equal, and to get “top tier” you will typically pay a premium.
- Consultants lack familiarity. While they can reduce constraints, they are not as familiar with your environment. They have a general knowledge of what is “normal” but every environment is different. This will require time and input from your staff, to help guide them through the twists and turns.
- Scheduling can be difficult. When you have a situation that requires consultants, you probably need them NOW. There are times that proves difficult to schedule, even if you’re a prior customer. This deficiency may be mitigated by paying a higher premium.
- It’s a reactive effort. Calling in consultants is very much an “after the fact” effort, and as such, it doesn’t get you ahead of your attackers. In essence, it’s really focused on cleanup. Some consultancies sell a “hunting” service, but that is both expensive and starts to bleed over into a retainer scenario. Why pay someone in hourly buckets to do something you can do yourself?
- It’s expensive. Finally, we get to the cost factor. Using consultants is expensive, even on the lower end of the spectrum (in more ways than one). Since they typically have one chance to collect evidence, they will want it all. Memory images and other volatile data collection, forensic disk images of servers and workstations, firewall/proxy/DNS/AD/etc logs, and more get suctioned up into the consultancy machine. This not only takes time, but it’s also invasive, disruptive to the business, and has a direct and significant cost. From there you go into analysis, and possible return visits to both.
When you are not in a position—for whatever reason—to respond to an incident with internal resources, you don’t have much choice except to engage consultants. It is what it is at that point.
3: Keeping an Incident Response Company on Retainer
If you don’t have the internal resources (or you just want to bolster them), and you don’t want to go the ad hoc consultant route, having a company on retainer can be beneficial. This is still using consultants, but with a different dynamic than before. Typically you’ll have a pool of hours to pull from throughout the year, for which you pay up front and/or monthly. The same pros and cons exist as with the ad hoc option, but generally with some overall improvement.
Key lessons on incident response retainers:
- You’ll face an annual renewal. We’ll hit this one first as it’s fairly significant. Not only is the retainer a substantial expense, but there isn’t a “rollover” for the pool of hours/funds from year to year; so as the year is coming to close, you have to use that pool, or lose the remainder.
- Aim to move from reactive toward proactive. Those end-of-year funds can be used for things like security review, tabletop exercises, or threat hunting; all of which help improve the company’s knowledge of your environment. If you’re lucky enough (or pay more) to have at least one person that is assigned to your business as a “go to,” this can help build the knowledge over time (at least while they remain; consultants tend to have a fairly high turnover rate). While this approach is still primarily reactive, it does offer a chance to help you move toward proactive, by using up that annual pool instead of losing it.
- Scheduling is (hopefully) simpler. There may still be some challenges here, depending on variables (such as what “tier” of customer you are). Overall, you shouldn’t have the same scheduling issues as you would with a consultant, since you’re under contract with them (typically with associated SLAs).
- It looks good in audits. In addition to the prior points, if you are in a regulated industry, or under considerable scrutiny, having an IR retainer can help you look better in audits. Not much of a consolation prize for having spent all that money, but at least it’s something.
- Use it or lose it. This one bears repeating. The big issue with keeping an IR company on retainer is that you are paying an annual fee for a pool of hours to use during IR incidents. If you don’t have incidents, you have to spend the money or straight-up lose it. It may seem like you’re getting value from it, but it’s really like betting against yourself – you’re paying money in hopes of getting to use it during an incident; if you don’t need to use it that way, it’s really just money down the drain.
4. An Alternative Approach: Managed Detection and Response
We all know that leadership often views InfoSec as nothing but a cost center and a “House of ‘No’” that slows down the business. As such, those in the trenches have to work hard to improve their company’s security posture in the most cost-effective way—both tactically and strategically.
Successful incident response is built off of two core competencies:
- Timely detection of malicious activity or compromise (Mean Time to Detect)
- Timely and effective response and remediation (Mean Time to Remediation)
Today’s security teams may be able to get more “bang for their buck” by considering a specialized security solution with core services centered around monitoring, detection and threat hunting, and empowered response. Organizations are finding that partnering with a Managed Detection and Response provider drastically reduces attacker dwell time, and depending on your industry and specific IR requirements, could either augment or replace the need for keeping a company on an IR retainer. As opposed to waiting for the breach and having guaranteed access to an IR firm, proactive security teams should consider allocating that budget into a partnership that will detect threats as they enter a network and infect an organization’s endpoints.
What to Do
The key is, you have to strike a balance and find what works for YOUR organization. None of this is “one size fits all” by any means. If none of the above really strike a chord with you, and like Benjamin you’re tired of your organization’s money swirling down the plumbing, it may be time to consider an alternative. You might want to plunge that toilet and take a different approach, to keep those “benjamins” in your pocket instead.