Five Guidelines for Measuring and Reporting on Your Security Operations Program

Whether you have a well-established security operations program or are building it from the ground up, it’s important for security teams to constantly show value and identify opportunities for improvement. If you can’t answer questions like “How is our security program performing?” and “Where do we need to focus our time and attention?” — start with these five high-level guidelines.

To take a deeper dive, watch the on-demand webinar: What’s the Value of Your Security Program?

1: Measure everything. And it should be a lot more than just logs.

Start by measuring everything you can in your program. As you are setting up your initial framework, it is very unlikely you will have a perfect definition of “who, what, and why.” By measuring and keeping a store of a wide breadth of data, you can go back and review historical trends to answer questions you did not initially consider.

When collecting this information, ensure you keep all your records in one place and one format. This allows you to consistently compare across tools, teams, and solutions. And by keeping a central store, you’ll be able to easily adjust to the lens you are reviewing within. You can use the same numbers to evaluate the effectiveness of a single signature within a tool, as well as the overall effectiveness of the program the tool is part of.

For example, let’s say you’re tracking the time it takes to investigate an event for each IDS signature. You’ll not only be able to identify which signatures are costing your team the most time, but you can also compare the amount of time consumed reviewing IDS alert versus alerts from other tools in your portfolio. These learnings can help you improve your IDS implementation and the process your team uses to investigate alerts.

2: Clearly define your terms.

Are we talking apples to apples? Well, it depends on if two people define an apple as the same thing. This sounds silly, but it can be a significant challenge in the information security space. Certain terms have different definitions depending on who you are talking to. For example, ask three different security professionals to define a false positive. Depending on the context and focus of their role, you will get slightly different answers. These variations can have a significant downstream impact on what a measurement actually means. So make sure you establish a clear definition of the terms you will use and use the definition consistently.

3: Set goals.

The reason for measuring and reporting is to help gauge success and effectiveness. Without some goal or target, capturing metrics really does not do much besides generate busy work. To define success, you need to have a target or goal you are trying to achieve. If you are just starting out, determine what effect you are trying to achieve and make sure that achievement is measurable.

Further, make sure you have buy in from both those you report to as well as those that report to you. Do your goals make sense? Do they clearly articulate the value and focus of your efforts? It goes without saying that you make sure your goals are aligned with your company’s or team’s mission.

As you improve your program, it will be important that you tighten your targets to help drive continuous improvement.

Here are a few sample categories to get you started:

• Mean Time to Detection
• Mean Time to Response
• Time to Patch CVE (Published to Patch)
• Time to Triage/Investigate Alert
• False Positive Rate (especially interesting over time)
• Dwell Time

4: Establish accountability.

Similar to goal setting, make sure what you measure is accountable to something or someone. The accountable party could be your provider, your boss, a different team, or even a specific tool. This will help drive improvement and better target where change is needed. For example if you have metrics across all your alerting capabilities—both the tool and the provider—each should contribute to the goals you have set for alerting accuracy and timeliness. When it comes time to renew contracts or licensing you can very clearly see which are delivering and contributing to meeting your goals and which are not.

5: Trust the numbers, not your gut.

Finally—and this is one that probably everyone has struggled with—trust the numbers. Do not let emotion drive your decisions and focus. Your team might swear that some tool or signature is going to find all the evil. Now that you have all the data, you can clearly see that it is adding little value to your goals and possibly having a net negative impact on the overall mission of the team. This idea can be taken at a number of levels and can help support your discussions when it comes to purchasing, renewal, and staffing. Have the numbers to back up your story; it will bring that much more credibility to the discussion.

Where to Start vs. When to Start

Regardless of how this looks in your environment, the key is to start somewhere. And start yesterday. There are many easy steps that you can take to start this process and cumulatively build upon your success.

Whether you are just starting, have been established for years, or are somewhere in the middle, take the time to have a conversation across the organization. Make the information internally available, regardless of what it shows. If the organization is not meeting its goals, everyone should see it and ask why. Just the same as if there is success—share your results proudly.

Security is everyone’s responsibility, but it is the security team’s responsibility to set the priority and focus. Ensure you set the right direction.