Bootstrapping your AppSec program

Editors’ note: This blog is based on a talk presented at BSides Augusta, RVASec. and HackRedCon 2022.

The world of security is often equated with technicality. We are the robots behind robots. And for building out an application security (AppSec) team, all you need is some nerds who know how to code, right? Not exactly. Red Canary’s Security Engineering team has formulated another ideology that might change your perspective on how to fortify your AppSec team. Let’s take a look.

Firstly, what is AppSec?

Application security, or product security, is a team/program designed to protect all the code you own. That begs the question: what falls under that category of ownership? When we reference “all the code you own,” we’re talking about the code your engineers write, as well as the third-party libraries or open source tools you use. Yes, every piece of code your team touches is your kingdom.

Ok, so what is the AppSec team’s domain with regards to this code? The AppSec team is charged with many things:

• Maintaining a strong relationship with engineering and C-level stakeholders
• Facilitating internal sales to the engineering team
• Promoting smart use of tools and testing
• Understanding that if business doesn’t flow, there’s no product to secure

AppSec teams should be viewed similar to a modern IT program. They have a unique place in the business, affecting numerous parts of the organization. As such, it’s imperative that an AppSec team adds value and security to everything it touches. Keep in mind, value does not always equate to financial gain.

In 2020, the average cost to businesses affected by a data breach in the United States amounted to $5 million (USD). Monetary loss is scary, don’t misunderstand. However, an even more severe loss is one of trust with customers, of reputational damage. You wouldn’t go around donning Balenciaga right now, would you? You want your customers to view you in high regard. And that’s on growth.

How do you build a strong AppSec team?

Let’s focus on hiring. There are two fields of thought when it comes to hiring. The first, and perhaps more antiquated view, is that hiring should be based solely on merit and technical skill. Great, we’re back to robots. Our team is now composed of rockstar security developers who can fix all the bad code. So, what’s the problem?

For starters, these types of individuals are difficult to find and hire. You may have found the diamond in the rough, but they’ll likely be more enticed by a larger company’s golden lamp. Sayonara. Along that vein, good luck finding enough of these rarities to scale properly. When your company makes the jump from 100 to 10,000, you’ll be scrambling to find the needle in the haystack. 

Finally, hiring a hard skill-based team can become a block for engineering. Rather than fostering a team mentality, we see a shift toward competing for the supreme title of “smartest.”

But the goal of an AppSec team is not to face off against one another. That’s what the enemy wants. We’ve got to pull together to defeat the forces of evil!

Well, what’s the alternative here? Stop trying to find the unicorn and seek balance. It’s OK for a candidate to have only a high-level of understanding of application security, so long as they have strong soft skills to make up the difference. These individuals are moldable and willing to learn, providing you an avenue for bridge building, and therefore success as a team. As Herb Brooks says in the movie Miracle, “I’m not looking for the best players, I’m looking for the right ones.”

What else do you need to create a successful AppSec team?

Leaders. The best captains are those who stay with their ship, even as it’s sinking. If your team leader doesn’t genuinely care about the success of the team and the individual, you wind up with anarchy. Many employees seek a mentor in their team leader, a feat that is difficult lest you assure them you are invested in their own growth and progression as much as the success of the team. As a manager, you’ll want to ensure you’re leading from the front. This means getting your hands dirty and proving you’re more than a delegator. 

Other commendable leadership traits include maintaining humility and honesty. Own your faults and pitfalls and ask your team members for help. They’re more than capable, you just have to give them the opportunity to prove it. Above all, as a leader, you should learn to care about your people. It can be difficult, that much we know. But know that compassion goes a long way! Team members want to be seen, to be cared for. If they can see their career progression is important to you, they’ll be more apt to best your expectations.

How to start with threat modeling

The traditional AppSec model is outdated. Buying tools without doing your due diligence is a great way to get stuck in the mud. Our recommendation: threat modeling. 

OWASP defines threat modeling as a tool that, “works to identify, communicate, and understand threats and mitigations within the context of protecting something of value.”

Threat modeling should help you find real areas of concern and exploitable vulnerabilities that are lurking in your environment. Keep in mind, threat modeling is not solely applicable to code! Try applying threat modeling to your hiring process. Where are your weak spots?

Let’s take a look at some principles to follow for threat modeling. It’s imperative that you build out a procedure for threat modeling so others can learn and implement. You should also try to encourage diversity of thought by having an engineer, sales engineer, or architect related to the instance we’re threat modeling. Build bridges with engineering, especially if they’ve never done this before. Lastly, make it fun! No one says “no” to free beer or food.

Once we’ve completed our threat modeling, we’ll have a better understanding of what it is that we need to secure.

For more on threat modeling, check out this blog my colleague Katie Nickels.

What’s your roadmap?

Speaking of security, let’s shift focus to the Secure Software Development Lifecycle or Secure-SDLC. Secure-SDLC is the implementation of a set of controls to secure your software development process. The following list outlines considerations at play when developing your Secure-SDLC process:

• Learn: What problem are we going to solve? What technology will we use? What part of the company do we need to understand? Enumerate!
• Plan: Get all the relevant folks involved and put together a plan of action that includes implementation, metrics, stakeholders, etc.
• Design: Look at how to implement this control in your company. Not all implementation tactics will jive with your organization. Take a holistic approach here.
• Approval/Buy-in: You need stakeholder approval of your progress before executing. Need, not want. Keep that at the forefront.
• Execute: Triple check your plan for soundness. Not once, not twice, thrice.
• Improve: This is your opportunity to fix what may have ended up broken. Continuous improvement is the key to seamless functionality.

Getting over the first control hurdle

Now that we’ve taken a look at our considerations for Secure-SDLC, we’re ready to get started. We have some thoughts regarding the implementation of your first control. Bear in mind that this implementation should be both holistic and based on your company. Every organization is different. 

Our recommendation would be to stay away from starting with Static Application Security Testing (SAST). It’s a great way to wind up with a plethora of false positives and an agitated engineering team. While we’re at it, for our first control, let’s pick one with low false positives. Save yourself the headache. 

Now, we know this will probably be received with groans, but it’s imperative that you write a policy to back up your control with buy-in. Think Secure Coding Standards. And lastly, be thoughtful. If you’re careful and methodical in your approach, you’ll set the tone as such for your entire security program. Oh, and if you hit a few snags along the way, again, food and beer help ice the wounds.

Once you’ve successfully implemented your first control, you’ll want to think about what’s coming down the pike; the next control. This is completely different for each team and each company. Taking a holistic approach and designing a plan based on a threat model can be extremely useful. Consider questions like:

• Do you have a budget and a mass of code issues? If yes, try software composition analysis (SCA).
• Is it just you and are you overwhelmed? If yes, delegate security champions or Slackbot.
• Have you hired some folks? If yes, defer to embedded security engineers who have enhanced visibility, strong inter-team relationships, and can serve as a point of contact (POC).
• Is there some fancy new product coming out? If yes, try threat modeling or conducting a security risk assessment (SRA).

Tried-and-true methodologies we’ve seen leverage these questions to ensure control success. Along the way, make sure you’re reconvening with the relevant persons to discuss the threat model and work to mitigate false positives. Any findings that are discovered should be placed in the lap of the engineers for scrutinization. 

Don’t be a tool

The prospect of acquiring and employing new tools is enticing. Before we dive into the pre-acquisition process, we have to discuss the elephant in the room: budget. How do we get a budget for an AppSec program? Well, sometimes we can’t. If that’s the case, we have to lean heavily on ourselves and the open source tools we implement. 

But if we’re lucky enough, we can squeeze the bank by proving our worth, gradually showing value through careful planning, slow and efficient control implementation, bridge building, and maintenance of metrics. Speaking of metrics, we know they’re hard. Metrics should tell a story centered on continuous improvement, your team’s growth, and benefits for executives and engineers. 

Once we’ve built a strong foundation of value, we can pull out our communication skills to talk to the business and make a plea for our case. If you can show executives why, financially, it’s a good idea for the company, you will get closer to a real budget. Money = spending power = new toys. 

And who doesn’t like getting new toys? But the acquisition process can be grueling and requires the right amount of scrutiny for proper execution. We might get distracted by shiny things, but the rule of thumb is to minimize the number of tools necessary for you to do your job adequately. 

Also, make sure you try before you buy. Test the tools you’re considering and set up a scoring system to encourage being pragmatic about your choosing. Similarly, testing the tool’s APIs before buying is extremely important, lest you end up with a tool that boasts bogus API integrations. Finally, prior to acquisition, you’ll want to build out a plan which includes identifying a centralized location for data and vulnerabilities. 

But the fun testing and re-tooling doesn’t stop here, folks! Once you’re in possession of a new tool, make sure you implement, tune, and improve it before moving on to the next. Remember to take your time. This process is meant to be sluggish. 

Fostering a socially engineered security culture

The final step in building a strong AppSec team is implementing a culture of tenacious security professionals. Part of this initiative is bolstering your education and training programs. Sure, they can be drab, but if you know your audience, you’ll be able to adjust to ensure engagement. So, how do we do this?

• You build bridges with engineers and have real relationships. AppSec is internal sales to the engineering team, and successful sales people find a product that makes both parties happy.
• You find people passionate about teaching
• You understand you are making a safer, more profitable business
• You realize you are making more valuable engineers
• You social engineer a security culture into your company

Social engineering? Yes, you read that right. We’re not talking about the bad kind of social engineering that’s self interested and thrives on manipulation—the kind of social engineering you’d drop a house on. Rather, we’re on the side of Glinda, The Good Witch. Good social engineering constitutes knowing your audience and learning what is beneficial to all parties involved. Emphasize the importance of building a culture based on trust and visibility. 

Once you’ve instituted these steps synopsized above, shift your focus to continuous improvement. Prove your team cares by putting in the work toward betterment:

• Try to refine the most obvious things first
• Set a recurring meeting with an engineering manager to critique controls
• Refine controls by studying your metrics
• Automate, automate, automate!
• Be customer focused (engineering team)
• Be objective

So, what are our takeaways from this extensive word vomit? You have the capacity to make your AppSec team/program great so long as you deploy the right tools and methodologies. Focus on your people, on your approach, on harmony. Oh, and you’re never too busy to improve. As Cady Heron in Mean Girls puts it, the limit does not exist.