Security Operations Lessons: What My Team Learned Building and Maturing a SOC

Building and maturing a Security Operations Center (SOC) is different for every organization. In this guest post, a security engineer at an insurance company in the Midwest shares what he learned as part of a three-person security team charged with implementing a SOC. The following views are his own and not those of his organization or team.

Someone once said security is never complete. The same is true for maturing a Security Operations Center (SOC). Many factors impact how a SOC matures: size, structure, willingness to leverage managed services, capable resources (i.e., people), and budget. Perspective on how things should mature and how they are actually maturing will vary depending on your role.

Three years ago, I joined my company’s three-person security team with the intent of doing identity and access management, risk assessments, and incident response. Within a month or so of joining, we were tasked with implementing a SOC and have been maturing it ever since. As one of the SOC analysts, my role includes everything from initial alert triage to incident response and everything in between.

Here are a few lessons I learned along the way:

1: Stay focused on your goal of improving security.

A SOC’s primary responsibility is helping to ensure the security of the company and the data its clients entrust to it. This has to be first and foremost in your mind when making decisions.

Sure, you might love to reverse engineer malware or create a new exploitation technique, but you have to ask yourself if you should really be working on those tasks. This advice can make a huge difference in your team’s ability to stay focused and help you to deal with (or at least better accept) the frustrating parts of your job.

Frustration is a part of every maturation process; it goes hand in hand with change and growth. Remember how frustrating your teenage years were? Or how about (for some of us out there) the frustrations you felt in your 40s when your body began working against itself? Frustration can be a motivator for improvement, or a demotivator and cause for resentment. How you respond is up to you.

Maturing a SOC takes time and patience. Keep your eye on the prize: improving overall security. Refer back to this lesson when questions come up, which they inevitably will.

2. People make the difference.

People are the strongest asset of a company’s security program. They can also be its biggest security risk. The same is true for the SOC. A poorly performing SOC can introduce more risk than it mitigates. How often have you been reviewing an event and decide that everything looks okay, until it occurs to you that you do not know what “normal” looks like? Training and experience help with this. And experience only comes with…well…experience. You have to handle alerts, create use cases, research, and threat hunt. Not only do you need to understand what those things mean but more importantly, how best to apply them to your environment.

This is how I feel some days reviewing alerts:

A new SOC does not yet have the ability to excel at everything that’s needed unless you (a) have experienced people; (b) can hire them; or (c) work with a managed service. We did none of these. We leveraged what we already had from our SIEM and did what everyone else does: go online to research best practices. That got us started but all of us—including management—knew we had a long way to go.

Which leads to our next lesson…

3: Make time for education and training.

Early on we brought in outside training to jump start our education. This not only helped with our basic understanding, but also helped show where some of our architectural gaps were. As our SOC has matured, we can be more selective about what training we attend. We have a good understanding of our environment and how to secure it. That doesn’t mean we have all our use cases implemented. Time, resources, budget, and tooling all come into play for that.

Once you have the foundational knowledge, you can begin to move into building areas of expertise like threat hunting. A key piece of this is not only knowing what is going on in your environment, but also being able to keep up on the latest tactics and techniques. The hard part is balancing education, reacting to alerts (especially as you add more), and all the other tasks you have to do.

A good penetration test is beneficial not only for jumpstarting a new SOC, but also for showing where your education is lacking. Sometimes it takes the solid proof of an external party stealing the CEO’s credentials or your company’s crown jewels to draw attention to what needs to be done and where more training is needed.

Learn how to start testing your defenses with open source Atomic Red Team tests

4: Set priorities.

So there you are, pounding your head on the desk at the latest person to click on a phishing link. You have ten alerts in the hopper, two requests you need to have completed by the end of the week, and a really cool article on the latest malware staring at you from your browser. Of course, it is at this moment when your supervisor appears at your desk and says, “This is urgent. You need to run these metric reports. We are presenting to senior management tomorrow and need the information.”

Take a deep breath and remember directive one: whatever improves overall security is good. Having senior management better understand your company’s security posture would be beneficial for the company. Repeat that to yourself until you can turn to your supervisor and ask in a calm voice, “So, which of the thirty-eight other urgent tasks does this trump?”

All kidding aside, priority and time management is tough. How do you balance day-to-day requirements like fine tuning alerts with researching new ones, staying abreast of changes to your organization, staying abreast of new techniques, and everyday office tasks? One way is to set priority levels for your tasks, whether it’s “high, medium, low” or “one to ten.” My team has “critical, super critical, urgent critical, super urgent critical” and we’re considering adding “most critical super urgent.” (Okay, maybe not, but it feels like that sometimes.)

The first priority is to know what you need to be working on. As we started down the road of building and maturing our SOC, members of the team kept their own ever-growing list of things to look into. We transitioned to one big shared list. Once a week, we review items and prioritize them. In theory, if we all deem something is “super critical urgent,” then we figure out who has time and either shift items around or delay others. You need to balance tasks as a whole to know what the true priority is and ensure there’s no overlap.

Of course, having a task list doesn’t mean you stick to it. Even the captivating draw of the next alert can’t always keep you on task. Balance the time you spend on tasks with research and improvement. This can be a good way to reduce stress and reinvigorate yourself.

5: Share your knowledge.

The SOC team is a partnership. A good SOC team has people with different backgrounds capable of looking at problems from different angles. Everyone has his or her own strengths and weaknesses. You can be the rock star, the golden monkey, the one everyone turns to for answers. Just remember to share your knowledge, not hoard it, so that you help raise others to your level.

We have a morning SOC meeting. In the beginning, we covered how we responded to alerts and asked others how they would have responded. That worked great; it quickly got all of us up to the same analyst level and instilled individuals with confidence when everyone agreed with their findings. As we have matured, we still present new alerts and ask questions, but we’ve expanded to concentrate on other topics. We’re preparing to onboard a new SOC analyst and are revisiting the routine of presenting one new alert every morning. It will be a good review for us, and a good way for the new person to learn.

Keep in mind there still needs to be ownership. If no one owns something, then no one feels responsible. Take ownership of things like managing your use cases or a tool. Voice when you want to be the one to work on something and be open to taking others’ tasks. Fighting over who gets to work on the new big shiny object is not good for the team. Grow your knowledge and share it with others.

6: Openly and efficiently communicate with management.

Just as important as communication within the SOC is communication with other teams and management. Management is a key part of the SOC. We need them to get approval to work on tasks, get funding, and get buy-in. They are also the people that sometimes bring us tasks that interfere with getting our other tasks done, and the ones we have to reassure that the world is not on fire (or perhaps inform that it is).

Framing discussions with management can be challenging for analysts. While you think you are clearly describing the problem, you might not be. Try reviewing the message with other team members. Know your audience and how to speak to them to ensure that your message comes across.

Imagine the following scenario: Jane the SOC Master stops by her supervisor Joe’s desk and says, “Hey, I think we should block web advertisements.” Joe says, “Good idea, put it on the list,” then turns back to whatever supervisors do. Both are frustrated. Jane is thinking, “I spend two hours a day reacting to noise because someone just happens to browse to a website hosting an ad from a bad IP. Not to mention the time it takes to react when the ad actually downloads or runs something malicious.” Joe the supervisor is thinking, “Why did she stop over and not just add it to the list?”

Now imagine a different approach. Jane stops by her supervisor’s desk and says, “Hey, I sent you an email; take a look at it when you get a chance. I think we could prevent some malware and save a couple hours a day reacting to noise if we block web advertisements. My email includes the number of alerts in the last couple of months and some information to demonstrate the positive impact we could see by making the change.” Joe says, “Good idea, I will take a look at it, but hold off doing anything right now. I just heard from management we might be looking into a solution for this.”

Which conversation went better? Even though Jane wasn’t able to implement preventing web advertisements, the level of frustration is reduced because she can be fairly confident that Joe knows what she is trying to do and the problem she is trying to solve. She also understands why she cannot start on it right away.

Being able to communicate effectively, in both directions, will alleviate a lot of frustration. You are on the front line and have more understanding of the topic than your audience. This doesn’t mean you are always going to get what you ask for. But just because the answer was no today does not mean it will be no tomorrow. Pick your battles. What tends to work in our in environment is to raise a concern, let it sit, and then bring it up again after a period of time has passed. Like good whiskey, some things need to age. It can take a while for ideas to resonate, and sometimes things need to occur before people are ready to accept them.

7: Focus on what makes you safer, not shiny new tools.

Ever been tasked with reviewing a tool or application that you know will not be helpful? Ever had to implement one? All you can do is ask questions and make sure everyone understands to the limits of their ability, what really is going on and what the tools can really do. This also comes with maturity. You learn what to ask, what makes sense in your environment, and when the salesperson is blowing smoke.

At the beginning of our SOC, we went looking for an advanced anti-malware solution. A relatively new one was on the market that claimed they were the silver bullet, that their mathematical model was the only thing you needed to protect your endpoints. At the time, while we understood there really was not going to be one solution to our problem we just didn’t know the right questions to ask or how to test the product on our own. In the end, everything worked out, and we learned a lot as we matured. Now we know good questions to ask and have developed some scripts for testing the efficacy of security solutions.

Related Resource: EDR Buyer’s Guide: 15 Questions to Ask When Evaluating Solutions

Just because a solution doesn’t solve every problem doesn’t mean it is not worth implementing. You just have to know what you are implementing and the problem it is trying to solve. As you mature and become more comfortable with the tools you are using, you will also find new ways to use the tools you have.

Some of the tools available to the SOC can be of value to other teams. We have been asked on more than one occasion to use one of our tools to review a machine to see why something isn’t working or to assist in determining why a credential keeps being locked out. This not only adds value to the tools you own, but also helps you build relationships with the teams around you, which can be great resources when investigating events.

8: Take your time.

In the rush to get a SOC up and running and protecting your environment, it is very easy to go too fast and do things in the wrong order. Knowing your environment is key to protecting it. (Heck, even the consultants say so.)

We implemented a threat feed that was supposed to help us detect malicious activity in our environment. It immediately started sending out alerts. Lots of alerts. If we had taken the time to really review the feed we might have been able to tune the alerts more appropriately or we might have not even implemented it; another insight that comes with maturity.

We also implemented a whitelisting solution. As some of you know this can have a big impact. We took the time to meet with many of the affected users to discuss the challenges and what it would mean to them. We also redesigned some change management process. We did not go to the nth degree, but we were able to get a good structure in place that we could mature on and improve. By discussing what we were doing and why we were doing it, we got buy-in from most people and the transition was much smoother. Lesson learned: Be careful of burning bridges with teams whose help you will need the next time it hits the fan.

Looking Ahead

So where does that leave our organization and our SOC team? We are still young, but maturing. We struggle with common issues, like finding time for threat hunting and more thorough research and education. We are still trying to find communities, online and local, to discuss the issues we are having and how other companies are solving them.

We are also taking the time to go back through and look at some of the processes, procedures, and techniques we have been using. Have we outgrown them? Have we matured enough that those things are now lagging behind? In particular, we’re revisiting how we document our use cases to make them more usable, minable, and reportable. Yes, this means finding the time to do it. And for us, that means delaying threat hunting, which is not ideal. But as with all of security, everything is a balance.

Building a sophisticated blue team is challenging. Learn about Red Canary’s approach to Monitoring & Security Operations >>