Like a goalkeeper, blue teams are tasked with defending organizations against a variety of attack methods. Also like goalkeepers, a seemingly minor oversight can lead to major consequences. In either scenario, the side playing offense only has to find one way in, whereas the side playing defense has to imagine—and defend against—an indefinite number of possibilities.
Planning for the indefinite, as many blue teams know, can be overwhelming. Beyond merely knowing where and what to defend against, you have to know when it’s right to hold off on more information and when it’s right to act. Trying to figure out if your team has considered every factor and every possible outcome can be crippling. Not only do you have to effectively and continuously separate the signal from the noise, you have to do it while simultaneously assigning degrees of urgency to every potential threat. This stress is compounded as events start piling up and analysts start to lose focus, becoming inundated with data in a race to clean up the backlog before the next wave of events, and more data, start rolling in.
These realities can lead to a defeatist mindset among security professionals, inhibiting effectiveness and obscuring solutions. However, while it’s true that one defensive mistake can lead to a goal that causes you to lose the game, it’s equally true that one offensive mistake is all it takes to stop an attack. Similarly, it might seem like you and your team are being overrun with a deluge of information, but with a bit of inventiveness you can start to relieve the burden on individuals and (hopefully) improve the collective outlook. Sometimes all you need is a change of perspective.
In the rest of this blog, we’ll explore four ways that security professionals can leverage creativity and alternative ways of thinking to resolve apparently hopeless problems.
1. Combat monotony with task rotation and automation
Toiling away endlessly on dull tasks is basically the opposite of rewarding. Unfortunately, tedium is almost unavoidable in this line of work, but there are ways of diminishing the repetitive so that you can spend more time and energy on things that are gratifying and proactive.
Assigning primary duties and scheduling a rotation of those duties across your team is a great way to ensure that no single analyst gets hung up with all the monotonous work. From a practical perspective, it diffuses the frustrating work across a group of people, but, on a more psychological level, it enables an optimistic outlook.
Although staff rotation is great for preventing alert fatigue, it can break down under the strain of a heavy load of alerts, which is why we think it’s incredibly important to couple staff rotation with automation. Otherwise, you can end up in a situation where your analysts are only able to track down a handful of events before moving to a different role, effectively solving the toil problem but possibly failing to remediate the actual security problem.
Automation is a great way to reallocate duties so that analysts aren’t bogged down by commonly occurring threats that require minimal analysis. You can start by developing strategies for automating away events that require multiple duplicate actions, those that are well documented and consistent, and those that reliably contain unique and predictable components. Consider configuring your detection platform to do both the initial analysis of whether something is benign or malicious and also to take simple actions like sending alerts or automatically blocking certain files or hashes.
That said, setting aside time to build out an automation program may be difficult for certain teams, especially as events are piling up and you have limited resources and expertise to set aside for tuning. If time and resources are tight, you can consider third-party services as an option for automating tasks. Beyond taking the burden off your already busy analysts and placing it in the hands of people with good automation experience, it puts a fresh set of eyes on the events your analysts are dealing with, and these new perspectives could well open the door to new solutions.
The idea is not to automate all tasks; there still needs to be a human element involved to allow for contextual analysis, tuning, and more executive processes, but automating simply defined tasks will allow the analyst more time for innovating new ways to tackle known issues and more time to focus on more time-consuming problems. In other words, automation can help you decrease work that is mundane and increase work that is fulfilling.
2. Practice playing both sides of the ball
The importance of continual training can’t be overstated.
Humans aren’t perfect, and even the most seasoned analysts will make mistakes every now and again. These errors can lead directly to losses in productivity and morale. As such, it is important to incorporate structured attack scenarios that will give analysts invaluable experience dealing with real-world with attack methods. These scenarios can involve anything from full-fledged attack simulations to free-form practice attacks on virtual machines.
This type of hands-on practice is beneficial for both obvious and obscure reasons. Obviously, practicing attack simulations will make your team more familiar with real-world attacks, and experimentation will allow analysts to discover key components of exploits and attack methods (e.g. required vulnerabilities, utilized commands or resources, required chain of events). In the end, this will improve their ability to quickly identify context associated with malicious behavior. Ideally, this would lead to fewer mistakes. This practice also helps your team understand an adversary’s needs when executing attacks. By understanding adversary needs, you can form a rich idea of how an adversary may use your systems to fulfill their requirements during an incident.
Perhaps less obviously, testing empowers your team to fill previously unknown gaps. It also boosts team confidence and morale, as analysts enrich their professional lives with new skills and diversify the work they engage in. Generally speaking, people like learning new things.
3. Remember that you only have to catch one part of the attack
While it is true that the defenders have to imagine countless eventualities and attackers need only to find one way in, it is also true that, much like a goalie, the blue team often only needs to touch a piece of the payload in order to block the entire attempt. Once your team understands an adversary’s needs, you can move to deter attacks by focusing your time and controls on the most vital adversarial needs.
Now it’s time for a practical example: below we see a series of malicious behaviors that resulted in a payload being downloaded to a remote system. The attacker leveraged LSASS to spawn cmd.exe, create a user account, add the account to the local administrator group, and proceed to execute Regsvr32 to download and execute a binary from an external site.
The binary was downloaded using a compressed file and was executed from the Windows system32 directory. In this case, the binary has a known bad reputation and isn’t commonly seen within this environment. We also have the domain, which also has a known bad reputation, and isn’t commonly accessed by systems in this environment.
A number of these items could be easily monitored and alerted on, and any of them could indicate potential malicious activity. For instance, we could have alerted on LSASS spawning a child process, an executable getting placed in the system32 directory, the use of Regsvr32 to bypass whitelisting, or even the reputation of the domain or IP address from which the file was being downloaded. Any of these would have pointed to suspicious activity and the attack would have been caught. The analyst would simply need to focus on what had been detected, the context of what was found, and follow the chain of events before or after the event. And for preventative controls, we could focus on denying the adversary of two particular needs to render the attack useless—the ability to exploit lsass.exe and the ability to download a script to execute using regsvr32.exe.
Whitelisting of the Windows system32 folder or the temp folder could have also been used to block the file from being downloaded, but this may not have provided the other context as to how the attacker got a foothold within the environment. In this case, a manual approach allowed us to see the entire field, get the big picture, and narrow in on the important aspects. From here we can pat ourselves on the back, automate detection for some of the associated components of this now known activity, and move on to next event.
4. Be transparent and communicate openly
Whether it’s a discussion with members of your team about an issue or communicating with other teams about objectives, changes, or deployments, transparency both holds people accountable and increases trust. It’s too often that useful tools or techniques and even success stories are held closely by teams or individuals. Maybe we are concerned about added scrutiny, or maybe we are worried that our secret resources and playbooks will be overused after they become more widely available. However, it’s difficult to improve when we aren’t challenged, and it’s hard to objectively measure success without outside perspectives.
Furthermore, failures in communication breed mistakes and laudable efforts go unnoticed if they aren’t announced. Open conversations about processes lead to improvement. At the very least, transparency and open lines of communication ensure that everyone understands and is able to convey what they do, how they do it, and how it differs from what others do. In the end, more freely exchanging information—both within and outside the security department—can help the larger organization understand what the blue team does, how it affects business goals, and how the company could adjust their activities to improve their security posture. After all, a stronger security posture for the entire organization will almost inevitably lead to a more positive culture within the security department.
Putting it all together
The idea of burnout in the information security industry is probably exaggerated. Overstatement or not, people do burnout on stressful work, and this has a negative effect on decision making. As such, we do everything we can to keep morale and confidence high, whether it’s through task diversification, automation, or other methods of creative problem solving.
Ultimately, it’s amazing how a change of mindset can refresh a defenders’ perspective on protecting assets despite a never-ending onslaught. And you might be surprised how little changes in day-to-day process can lead to big changes in overall thinking.
It can be nerve-racking to think that surrendering one goal can cause your team to lose the match, but there’s a lot that goes into scoring a goal and a whole team out there to prevent it from happening.
Related Articles
Translating our detection engine: A journey from JRuby to Go
Translating our detection engine: A journey from JRuby to Go
Best practices for securing Azure Active Directory