Testing adversary technique variations with AtomicTestHarnesses

Our goal is to present enough context that you will understand what’s happened, even if you lack expertise in a particular technique, and also to ensure that there’s a consistent experience across all exported functions. You can expect to see the following object outputs for any function:

• TechniqueID: the MITRE ATT&CK technique ID that corresponds with the function. It is important to be aware of the specific technique that the test corresponds with so that there is no ambiguity.
• TestSuccess: A value of `true` will indicate that the specific technique executed without issue. Most AtomicTestHarnesses functions will include benign template “payloads” that execute the technique-specific functionality and report back to the test code that they executed successfully. This way, you can trust that if `TestSuccess` is `true`, then any subsequent component in your detection validation pipeline can safely assume that the technique actually executed successfully. Having such confidence can dramatically reduce root cause analysis time when diagnosing test versus detection failures.
• TestGuid: This identifier is unique to every test execution. We feel that it’s important to be able to associate a unique identifier with each test run so that it is possible to link the test you ran to subsequent components of a testing/validation pipeline. Every function implements a -TestGuid parameter and you can manually input your own GUID if you want to be consistent about the output.

All remaining object output will pertain to the specific technique. Our goal with object output is that it supplies defenders, detection engineers, and test engineers with all of the relevant information they would require to both confirm event telemetry and validate detection logic. Generally, we strive to capture the “what” and “how” behind each test run. In the case of HTA execution, HTA content can contain embedded Windows Script Host (WSH)—e.g. VBScript, JScript)—script code and execute it. In designing a function that automates this process, we would ideally want to capture the following “what” and “how”:

• What did the HTA execute? (i.e., the WSH script code)
• What was the HTA runner? (i.e., the host process that executed the HTA)
• What relevant operating system artifacts were introduced?
• How did the HTA execute? (i.e., there are many different ways to invoke HTA content. Which variation was used?)

After running one of the AtomicTestHarnesses functions without arguments and after reading through the contextual object output, you may then want to investigate the suite of parameters that are supported to invoke all identified variations of the technique. You can begin to get a sense of the test variations possible by running Get-Help, which will return the following output:

Again, in practice, running the Get-Help command will look like the following:

When you see multiple options like this, it means that the PowerShell function/cmdlet has multiple parameter sets implemented. A parameter set is a set of parameters that are mutually exclusive from other sets of parameters. We try to design each parameter set to comprise all available technique inputs for every identified technique variation. Based on open source and internal research, we identified the following technique variations for HTA execution (note that there will be six listed in line with the implemented parameter sets):

1. HTA execution from a file using mshta.exe as the host runner
2. HTA execution from a file simulating a double click. HTA files are a common phishing vector so we wanted to replicate invoking the default file handler for HTA files
3. HTA execution by downloading from a URI directly using mshta.exe as the host runner. This variation is very commonly used in the wild.
4. Inline execution using protocol handlers using rundll32.exe as the host runner
5. Inline execution using protocol handlers using mshta.exe as the host runner
6. HTA execution from a file embedded within an executable (i.e., polyglot) using mshta.exe as the host runner

Considering all the implemented technique variations, you could spend a while manually experimenting with all the different options, and we certainly encourage you to do so. However, what if you wanted to test all supported variations in order to validate visibility across your detection pipeline? Keeping track of all possible permutations is challenging, and executing everything manually is time consuming and error prone. Fortunately, with each new AtomicTestHarnesses release, we will supply Pester tests that automate and validate the process of executing all implemented variations.

Automatically validating technique execution

Pester is the official testing framework for PowerShell, which made it a natural choice for both testing and validation of AtomicTestHarnesses functionality. What’s great about Pester is its ability to reason over expected behavior, something that Atomic Red Team and Invoke-Atomic were not designed to do.

Considering it may not always be obvious what the expected results for a specific test should be, it is our opinion that the developers of the test harness code should capture and validate expected results. All exposed functionality in AtomicTestHarnesses will come with Pester test code to automate the process of execution and validation. Each test suite is also tagged accordingly using the relevant MITRE ATT&CK technique ID. With our test code, we strive to implement at least one test for every parameter-set permutation. Think of this approach as attempting to maximize code coverage in the traditional unit-testing sense, but, in our case, we’re striving to maximize “detection coverage” by supplying the same level of modular input that an attacker would have when leveraging a technique in an evasive manner.

The above image shows the results of a Pester test for a single function. Given that green text is an indication of success, how do we interpret the results? For starters, a passing test will signify that the technique executed successfully and that the test generated the expected artifacts. With confidence that everything executed as expected, how you interpret the results will be entirely up to you.

If testing endpoint detection and response (EDR) sensor optics, you would likely want to take the output of each test and, through some sort of automated process, validate that your EDR platform observed (note: “observe” as opposed to “detect”) every instance executing. If, for example, an EDR platform does not have optics to support observing every test instance, then it can be inferred that it won’t have the ability to detect suspicious behaviors that run on top of the underlying technique.

For more information on running Pester tests with AtomicTestHarnesses, please refer to the documentation. There’s a lot more to cover about testing philosophy and methodology, but we’ll leave that further deep dive for another day.

AtomicTestHarnesses meets Invoke-Atomic

We have also committed the base parameter sets for each AtomicTestHarnesses as an Atomic test that may be consumed by Invoke-Atomic. To get started, follow the standard procedure to install Invoke-Atomic and download the atomics directory. The following video demonstrates this process as well:

Following our example of using Mshta T1218.005, let’s confirm the new T1218.005 has downloaded by running the following:

Invoke-AtomicTest T1218.005 -ShowDetailsBrief

Test numbers four through nine will use the Invoke-ATHHTMLApplication function, and we can confirm that by reviewing the details of the test number itself with the following command:

Invoke-AtomicTest T1218.005 -ShowDetails -TestNumbers 4

We see under “Dependencies” that Invoke-Atomic is validating whether AtomicTestHarnesses is loaded or not. If AtomicTestHarnesses is not present, running -GetPrereqs for the atomic T1218.005 or the specific test number will finalize the setup by installing the PowerShell Gallery module.

Now we can execute all or a single test. Here we’ll run a single test with Invoke-Atomic:

Invoke-AtomicTest T1218.005 -TestNumbers 4

If you’re running the test on your own, it should look like this:

As testing commences and validation is being completed, we may want to create or modify current tests to extend our testing capabilities. Fortunately, it’s as simple as modifying the input arguments.

By modifying the input arguments on a single test, we are able to exhibit a new behavior.

Now it’s your turn

Now that you know what AtomicTestHarnesses is and how to use it, it’s time to take this tool into your environment and figure out if your detection analytics stand up to the many variations on

• T1134.004: Access Token Manipulation: Parent PID Spoofing
• T1218.001: Signed Binary Proxy Execution: Compiled HTML File
• T1218.005: Signed Binary Proxy Execution: Mshta

For our part, these tests have helped us develop more robust coverage against a wide array of threats while also validating the efficacy of our existing coverage.

This is just the beginning for AtomicTestHarnesses. We’ll be hosting a deep dive into the tool this coming Friday, October 30 at 11 am MT as part of our ongoing Atomic Friday webinar series.

While AtomicTestHarness isn’t currently accepting community-contributed test harness code, we hope to change that once we feel confident that we’ve educated the community about the design and implementation process. In the meantime, check out the repo, run some tests, and fire away with pull requests for any bugs you might find.

In part two of this blog series, we’ll explain in great depth our research methodology for identifying relevant variations on a given technique, and how that work ultimately led us to the creation of AtomicTestHarnesses.