During the course of my career I’ve gone from working at a large Managed Security Service Provider (MSSP) to leading security for a Fortune 150. I’ve been on both sides of MSSP relationships and understand the challenges faced by both parties. If you’re reading this, chances are that you’re in the market for an MSSP and your immediate challenge is determining how to qualify the providers. All too often, buyers assume that it is enough to digest a capabilities matrix then provide the “winning” MSSP with data. This is not the case.
Here we’ll look at five evaluation criteria that can be used to understand the capabilities, strengths, and weaknesses of the MSSP. This is not comprehensive, but will help you understand whether a given provider is a fit for your organization.
1. Use Cases
Every good MSSP has a documented list of use cases based on event type or log source. Depending on the maturity of the provider, you will find that certain log sources are favored over others. You should always ask a few key questions about use cases:
- Provide five use cases related to the log sources I will be providing.
- Provide an example write up of an event from a single log source.
- How often are new use cases created?
All of these assist in understanding coverage, defined as the number and types of threats that the provider is able to detect. This should also provide some indication of the provider’s understanding of and ability to respond to emerging threats. No MSSP is able to address every conceivable threat, so you are looking for strong indicators that they understand the threat landscape and that coverage will improve in step.
2. White glove service
White glove or high-touch service is often applied to sway you into believing the service is not a black hole and that the MSSP’s humans will be in contact more than a few times per year.
Before an MSSP tells you about their high-touch service, consider how you might define it:
- Do I have a dedicated account manager, and what are they doing for me?
- Do I have a dedicated incident response contact?
- Do I have a dedicated support email or phone contact?
- Do I have daily/weekly/monthly virtual meetings with a non-sales person?
Once you have defined what high-touch means to you, ask the MSSP what they provide. If the MSSP meets them on paper, request they prove it. Proving high-touch is simple:
- Customer reference calls
- Sample event escalation
- Sample intelligence report
3. Feedback loops
An MSSP without an actionable feedback loop is useless. An MSSP with an actionable feedback loop is gold. It is also important to note that feedback does not simply flow in one direction (i.e., from customer to provider) and that feedback may apply to several aspects of the service.
State of Health
If the new provider notices that you have logging set at the wrong level for network devices, will they say something? What about Windows event log levels? Does the provider monitor system or appliance health, investigate when events rates aren’t nominal, etc.? Industry is filled with horror stories wherein a breach goes undetected not because of lack of controls or clever attacks but because systems were not functioning as expected.
The job of an MSSP is not to provide you with a complete representation of a detected threat. Rather, it is to alert you when certain criteria are met–to generate alerts. Thus, volume and accuracy matter tremendously.
Ask the provider how they handle both false positives and false negatives. Additionally, ask whether you’re able to provide them with exception feedback (in the event that there is detectable activity but it is justified).
It’s key to ask these types of questions up front, to ensure that the provider is able to be as responsive as your own resources and time constraints require.
4. Assistance during incidents/events
You are hiring an MSSP in part because they have a strength that you do not. You need to make sure that the people on the other end of the phone have an appropriate level of experience triaging, investigating, and responding to threats day-in and day-out.
- How experienced is the SOC staff?
- How did the MSSP respond to a recent “hands-on-keyboard” attacker?
- How long does it take you to get an experienced technical resource on the phone to support during a possible breach?
- What is the average mean time to respond to a customer call/request for help?
You do not want to identify staffing or expertise limitations in the midst of a breach. This will lead to a slow and ineffective response, which often leads to a sharp increase in overall breach cost.
5. Continuous improvement
The primary function of the MSSP should be to detect threats in the environment reliably and in a timely manner. There is often a compliance requirement met along the way. In the end, you need to make sure that the provider is not a black hole into which you send your data. You need them to continually make your security better.
- Request a call with the MSSP’s CISO to understand how they research new threats, tactics, techniques and procedures, and how this research is incorporated into their service.
- Request data on how many times the MSSP was “surprised” by another product or service identifying a threat that the MSSP missed.
One related note: I’ve evaluated many MSSPs over the years and found that large vendors do not always equal better service. Often times you will find, as I have a number of times over the years, that size correlates strongly with compliance requirements, aggressive SLAs and contract assurances, but it does not ensure effectiveness.
These criteria and suggestions should greatly assist you in determining which MSSP best meets the needs of your organization. Again, these are not comprehensive and ignore (until a future post) things like metrics, reporting, incident response retainers, and compliance. Take your time during the evaluation process, determine your needs, ask questions and apply as much rigor to your selection process as you can afford. This should lead you to the MSSP that best meets your stated needs.
The last piece of advice I will give – you do not want to make a mistake, do it right the first time. It is too costly and wastes your valuable time.
Note: Red Canary is not an MSSP. However, we are often asked to advise customers during their search and evaluation process and we are happy to do so.