Detection Engineering: Setting Objectives and Scaling for Growth

Finishing the recipe

Armed with our defined objectives and a fancy acronym (“the DFO’s”), we could further refine the process and implement the other aspects of the recipe. If you’re a security leader working to implement your own set of objectives, these steps can help you further refine your efforts, provide valuable feedback, and ingrain the new concepts in your team.

Let’s walk through them one by one…

1. Confirm the objectives.

Without buy-in and ownership, any type of top-down direction can end in disaster. Your team knows their day-to-day better than you and knows what works and doesn’t work. Your job as a leader is to define a strategic plan, multiply the strengths of your team, and help the team realize success along the way.

The first step in the process of defining objectives is to give the team the opportunity to provide feedback. To do this, we sought volunteers to become “champions” of each objective and built sub-teams under each based on the desires and interests of the individuals on the team. To confirm the objective, the team was asked to review the objective, description, and any proposed metrics, then answer a key set of questions:

• “Is this the right objective?”
• “Will focusing on this objective improve the product, provide value to our customers, and benefit the team?”
• “Is this objective aspirational and foundational?”

If any of these were answered in the negative, we had more work to do.

2: Create an aspirational metric.

With the objectives confirmed, we needed to define a simple, single, aspirational metric to assign to each of the DFO’s. Simple counts, ratios, or percentages that can be tracked on a set timeframe (e.g. weekly) and changed when progress is made are preferred as they are easily remembered and represent a clear goal. For example, we can say that 100% of our detections will be Consistent as measured by quality escalations in a week, and the organization knows where we are at any given time.

Make these metrics aspirational and make sure everyone knows that’s what you’re doing. If you pick a metric you already hit week over week, it won’t be as effective in driving progress.

Anyone that has been tasked with creating KPIs, SLAs, or SLOs knows that the process can be full of pitfalls. Metrics for metrics’ sake is a terrible idea—images of “pointy haired managers” come to mind and make me shudder. Aligning a DFO to a single metric avoids a lot of these pitfalls as long as you can make the measurement or observation of the metric consistent.

3: Codify the metrics.

Each team is tasked with coming up with several measurements of success or progress. They must propose their single metric for broader communication and consumption, along with the data and/or automation they will use as part of the measurement.

These metrics can often have sub-metrics that contribute or further define the state of the objective that are useful at the team or initiative level. It’s great to capture these to drive analysis, but the single measure is the most important.

4: Execute a “quick win” project for each objective.

Once you’ve codified your metrics, it’s vital to show how focusing on each objective will help unify the team and demonstrate progress. We accomplish this through the implementation of “quick win” projects for each DFO. Taking a well-defined, time-bounded project with a limited scope from start to finish lets you show the impact on the DFO or, possibly, disprove some assumptions that were made earlier in the process.

For example, our Efficient DFO team decided to look at our top five highest fidelity detectors and implement automation improvements that would allow us to deliver these detections faster and more efficiently. Our First DFO folks decided to tackle tuning of the 14 noisiest detectors and make changes to tighten the detection logic so we don’t miss something bad in a flood of false positive noise. Having the ability to measure the objective before and after each project is a great way to test the correctness of your previous work.

5: Announce, communicate, and track.

By this point, you should have high confidence in your objectives. It’s now time to close the loop by communicating these objectives and your current measurements beyond your team. Depending on your organization, this could mean one step up the ladder or more broadly to the whole company.

Doing so serves a dual purpose:

• It highlights the work your team has been doing and opens up a feedback channel
• It creates an accountability layer beyond your team

Organizations are full of bright people that may be able to look at a problem in a different way and offer ideas that could have a huge impact, but often times we’re too tied up in our day-to-day to think about these harder problems. Showcasing your work at a broader level creates an interrupt where this feedback is more likely to come your way. Holding yourself and your team accountable for the big, aspirational goals you’ve set through this process keeps the objectives fresh and front of mind. It also serves as an internal interrupt, pulling your team out of the daily grind and forcing each team member to consider the projects and initiatives that will eventually make life easier for everyone.

Closing thoughts

Security is a tough challenge for any organization. One of the many things that attracted me to Red Canary was our company’s human-based approach to tackling hard security problems. We don’t hire traditional analysts. We look for people with an analytical mindset who are self-driven, self-critical, and always looking for the next opportunity to make things better for the team and our security allies (aka, customers).

This framework of simple objectives with clear and transparent metrics is one of the ways we are empowering our people to deliver a world-class Red Canary experience. I hope it helps other security leaders who are working to improve the efficacy of their security programs and processes.