Security Architect Lessons: What I Learned Managing and Assessing Cyber Risk at a Fortune 200

I worked as the security leader of a global Fortune 200 organization for two years, where I was responsible for cyber security strategy, architecture, and risk reduction during an extended phase of rapid growth and acquisition. I focused on ensuring we had visibility across the most vital layers while working with each entity to mature their security posture and address risks across the businesses.

Inevitably, we faced challenges just like many organizations do today: alert fatigue, skills shortage, and response bandwidth. We eventually pushed past the challenges to assemble an integrated security stack with minimal staff. The lessons I learned during this part of my career taught me a lot about communication, perseverance, and problem-solving — valuable skills for any security professional who is working to address risk at a growing organization.

Here are the top 5 things I learned:

#1: Progress is more important than perfection.

I worked with organizations of all sizes — 100 to 15,000 workstations — and no matter the size or maturity level, there was always one thing on my mind: compromise is inevitable. No matter what policies or technical controls we put in place (enforcement, guidance, training) the human element was always there. It was not a perfect science. Instead of trying to achieve perfection, we focused on identifying critical assets and then determining what level of visibility we needed to protect those assets. We then determined which resources — people, process, or technology — we could bring to bear in each case. Like most organizations, we rarely had 100% of the resources that we would have liked, but we never let that stop us from doing something. Progress over perfection.

Once we began to collect and analyze data, we focused on a different type of progress: We learned from each incident, small or large. And we used what we learned to better prepare the organization by creating or modifying processes, educating associates, and identifying current products to assist with detecting and preventing. Living in a state of “progress over perfection” magnified the continuous approach to securing information in the organization.

#2: No single product can solve all of your problems.

The following was a typical day-to-day scenario:

One of our organizations had a <insert threat> issue and they would call frustrated, defeated, and wanting to know the next steps to prevent and never see these evil things again.

We’d have a conversation to understand what happened, what the root cause was, and how we eradicated the threat. Armed with this information, we would formulate a plan to ensure we had detection and preventative controls in place for next time. We’d then distribute it to all organizations to ensure we had the same mitigating controls in place across our environment.

This process was never fun. Often, we would get a response that went something like this:

“But X and Y product says it will prevent this, it uses math/science/magic/unicorns (sometimes combined—magic scientific unicorns).”

It can be hard for someone who doesn’t do this day in and day out to reconcile that things will be bypassed when all the marketing material within the industry says otherwise. An important lesson to remember is that NO single product can solve all of your problems. Security products are by definition part of much larger systems. And no system is perfect. Products can be bypassed, and where a specific product can’t be easily bypassed, adversaries will simply focus on other parts of the system.

#3: Defense in depth requires a framework.

After observing our organizations for awhile, we realized that we needed a program or a framework that was proactive and provided adaptability to current and future threats. In essence, our security program encompassed five key areas that improved the overall security posture of each organization by providing as much detection as possible and prevention when possible.

We found that the Cyber Intrusion Kill Chain really helped with visually showing where countermeasures could be placed and what parts of the kill chain they could impact. Of course, since the kill chain is a model and all models need to be tailored to your environment, we made our own variant:

We used this to visualize intrusion concepts in a manner that underscored the importance of visibility, layered defenses, and systems thinking. And by mapping existing controls to this model we were able to guide prioritization within the business units.

We also created this pinwheel, visualizing the ongoing process that we planned to implement:

Based on priorities, we began to build layers, each aligned with the killchain.

Wrapping this up in a bundle and providing key performance indicators allowed us to prove the value of the work being performed. In the months that followed we began to see a decrease in commodity threat activity, which freed us up to focus on more subtle and potentially damaging opportunistic threats in the environment.

#4: Visibility — You can’t address what you don’t know.

Our employer acquired roughly 10 to 20 companies a year and almost immediately integrated them into the platform. Sometimes, Information Security had no visibility until “something” happened. Our initial discussions were always interactive as I got to hear about their IT and security stack, operating procedures, and the interesting product they offered to the world.

We initially required the organization to deploy Carbon Black Response for endpoint visibility and began performing vulnerability assessments. Those two components gave us a very comprehensive picture of the organization with which we were now working. Here is a brief overview of the moving parts:

Endpoint Visibility – Carbon Black Response:

We would often identify infected systems on the same day we deployed, or it would be quiet and things would appear over the course of the week. The security team from the newly acquired company was always intrigued that we were detecting things that all of their controls had missed. Sequentially, we would also identify user behavior and application usage, specifically looking for lateral movement and policy compliance.

Vulnerability Assessment:

Our main goal here was understanding the risks we were accepting from the incoming organization. We needed to understand how much exposure this organization had on the perimeter — open ports, vulnerable applications, Citrix gateways, and so on. It’s important to identify these items as they are points of entry for threat actors. This was also the time we would find telnet/ftp/http still in use and we’d have the “clear text” communication conversation and work out a plan to move to more secure methods. Beyond this, we would also discuss patching strategies for operating systems and third-party software.

At the end of the day, the businesses ran autonomously and the best visibility we had was endpoint data provided by Carbon Black and our vulnerability assessment. The two combined gifted us with a full picture of how things operated and looked from the inside out.

• Browse popular resources for making the most of Carbon Black Response

#5: Prevention is ideal, but it only goes so far.

In November 2014, we had a large outbreak of Dyre. Some of the original waves were en masse and took our organization by storm. With Carbon Black we found remnants on the file system and captured the files. We had about five different AV products across the businesses, and we provided many samples to the vendors. Only one vendor came back with definitions immediately for the malicious files that were dropped, including the actual banking trojan. Other vendors ignored or delayed detection for some of the key Dyre components. It took roughly 45 days for a top anti-virus vendor to provide signatures for it.

Prevention only goes so far. It can lead to a false sense of security. Even if you are using machine learning to perform prevention, something will be missed. I am big believer in “prevention is ideal; detection is a must.” Do as much prevention as you can while making sure to carefully evaluate and measure the efficacy of your prevention products. Once you’ve settled on a prevention strategy, focus intently on the resources that you need to detect the things that your prevention tools miss.

Editor’s Note: This blog was originally published in December 2016. It has been updated and expanded based on our ongoing conversations with security architects.