Detecting All the Things with Limited Data

Despite spending millions of dollars annually on information security, organizations invariably realize that their security tools and controls—no matter how costly or advanced—have limitations. Whether it’s endpoint detection and response (EDR) or a firewall, you’re going to find blind spots. The trick is to identify these limitations early and often, and to compensate with controls that reduce the impact of blind spots so your team can continue to reap whatever benefits their tooling offers.

Unfortunately, adversaries are also aware of these blind spots, and they’ll take any opportunity to use them against you. For example, adversaries often encrypt their network traffic to nullify proxy controls or full-packet capture. Or, to bypass antivirus and signature-based scanning, they pack and encrypt payloads.

Scripting Away Visibility

One of the simplest ways to hinder visibility with EDR is to simply place malicious commands within scripts instead of process command lines. This is due to the nature of command shells in all platforms, and its effectiveness depends on which commands you build into a script. With any command shells, certain commands are considered internal commands. This means that the shell will not call an external program to execute code, it all happens within the shell. When these internal commands are executed as command line arguments to a process, EDR products can see them easily. When they are contained within a script or interactive session, not all EDR products can show them. Consider the following command:

Cmd.exe /c copy file1.txt file2.txt

This creates a copy of file1.txt and displays the “copy” data in EDR data as expected. When the same copy is contained within a script, visibility changes. In this case:

Cmd.exe /c C:\copy_file.bat

Some EDR products will not be able to show the copy operation, they will only show the creation of file2.txt. If your EDR product does not display the contents of a script or interactive session, you’d have to rely on capturing the executed script or a memory dump for further analysis.

On the other end of the spectrum, commands external to a shell should be easily visible within EDR data. Imagine the use of xcopy.exe instead of the copy internal command. Xcopy should always be visible as a child process of whatever shell executes it, even in script form. By piecing together the functionality of external commands observed when a script executes, you can surmise the larger purpose of the script without full visibility into its contents. You may not have 100 percent accuracy in guessing the script’s contents, but you’ll be able to determine whether a host needs deeper analysis for an incident.

Even though I cited cmd.exe for this technique, it is not the only command shell capable of these shenanigans. You can do the same for PowerShell using cmdlets, and you can use Bash or other shells with their respective internal commands. In the example screenshot, EDR data could not tell us what was in the malicious script, we had to rely on artifacts created by the script’s execution. No matter which shell or script is used, the principle is the same: keeping an eye on external command usage through child processes will help you build an idea of script contents.

Windows Management Instrumentation and Stored Payloads

Windows Management Instrumentation (WMI) is an administrator’s paradise with its ability to enumerate data about Windows systems, store more data, and enable endpoint management using any tool that can interact with WMI. Even though this technology is used by nearly every Windows administrator on the planet, it remains somewhat opaque since many administrators can’t inspect it as easily as a simple file on the desktop. Adversaries have used this to their advantage by stashing script payloads (like PowerShell commands) into WMI event consumers. Manipulation of WMI to store payloads appears similar to this shot.

They can also bind these consumer payloads to WMI event filters, ensuring execution of the stashed script when specific conditions are met at an endpoint. Some of the conditions include users logging on interactively and uptime reaching a specific time value. An excellent primer on this technique was published by Matt Graeber for Blackhat 2015 here.

This tactic hides exact payloads from EDR technologies by combining the power of scripting with a bit of obscurity. EDR solutions typically do not have visibility into data held within WMI storage. Carbon Black Response, CrowdStrike Falcon, and Endgame are awesome at recognizing file and registry modifications, but WMI modifications may go unseen. Wiley security folks using Sysmon may grin at this, because Sysmon can log the modification of WMI consumers and event filters. Even in these cases, the exact payload within a consumer is not logged, requiring practitioners to dive into the right area of WMI storage to discover what an adversary stored.

There is some good news: adversaries must fill two needs to use this technique. First, they must create WMI consumers and event filters. Second, the payloads stored in WMI must eventually execute. To create the consumers, PowerShell is the most popular method by far. In fact, even the Metasploit WMI persistence exploit takes advantage of PowerShell’s built-in capabilities to modify WMI with their customized payloads. This means that an adversary usually needs to somehow execute PowerShell before they can even use the obscurity of WMI. For execution, security practitioners can look to monitor child processes spawning from the WMI Provider process, wmiprvse.exe. New processes spawned from a WMI consumer payload will appear as child processes of wmiprvse.exe. This strategy is noisy in environments with Microsoft System Center, but a bit of tuning will help you spot suspicious PowerShell or other payloads executed by adversaries.

Reflective Code Injection

Hold on tight, things are about to get a little weird in terms of code execution. In everyday life we’re used to associating one process executing and matching its corresponding binary and code libraries on disk. This sort of execution is easy to observe with EDR data because it follows rules set forth by Windows API functions and EDR products can monitor those law-abiding processes. However, process execution gets much weirder when code injection gets involved. In the simplest forms of injection, one process uses a Windows function to place instructions for execution into another process. As with the other law-abiding processes, EDR products can usually monitor and report code injections following Windows API rules. Then we arrive to a more insidious form of injection: reflective injection.

Reflective injection copies the contents of an entire binary or code library into the memory space of another process. Once this is achieved, the source process instructs the targeted process to execute the copied code. In this case, a couple of the standard injection and code library rules are broken. Since the injected code resides in memory only and not on disk, many EDR products cannot see the injected binary content. In certain cases, the products may not notice an injection occurred at all until the process targeted for injection exhibits odd behavior. We’ve seen this behavior before in an incident where an adversary loaded Mimikatz and a Monero cryptocurrency miner reflectively into an instance of PowerShell to do the malicious work. In this case, evidence of the injection wasn’t apparent in EDR data, but we still saw the end result actions desired by an adversary.

Thankfully, EDR products are getting better at detecting forms of code injection. The technique is challenging to detect because there are numerous ways to change the contents of a process’s memory space, especially when an adversary gains administrator privileges. Constantly monitoring the contents of memory becomes expensive in terms of computing resources, and security products need to find a balance between “good enough” detection that doesn’t break a computer’s performance and “complete visibility” on an unusable system. Newer EDR products that incorporate memory inspection have an advantage in this area: they can inspect code injections based on suspicious memory page permissions or contents.

Windows Registry as Code Storage

Our final technique to discuss is a favorite of “fileless” malware: using the Windows Registry as a storage medium for payloads. Most of the time the Registry is used to store configuration data for Windows and the various programs executing. Much of the Registry requires administrative privileges to change, but there are Registry hives with permissions that allow end-users to configure their own software. Adversaries take advantage of user-writable areas of the Registry to store code—such as executable binaries or scripts for PowerShell or a similar tool. As with using WMI storage, this affords adversaries a storage location that is more difficult to inspect than a simple file on disk. In the example shown, PowerShell read a value from the Windows Registry and executed it, leaving no artifacts but the suspicious command line.

With EDR products, we usually don’t see Windows Registry read operations. In addition, while we typically observe write operations, not all products show the value of the Registry key written. In addition, when inspecting Windows Registry keys for stored code, you’ll likely find adversaries have encoded payloads in base64 or another form as an easy form of obfuscation from casual viewers. An excellent resource showing this is Mari Degrazia’s blog post demonstrating this technique. The good news is that adversaries must still fill a couple needs to use this technique: they must retrieve the code and they must execute it. We sometimes see both needs fulfilled at once using PowerShell such as in this incident where a PowerShell cryptocurrency miner was stored in the Registry. In other instances, we observe the use of VBScript code retrieving the Registry value and calling another process to execute it. In both cases, the retrieved code acts in a manner similar to our first scripting example. We may not see the full payload, but the behavior in process data will allow us to piece enough of the incident together to determine whether more analysis is needed.

Combating Limited Visibility

Limited visibility can be disappointing, especially when you’ve invested so much into your controls. This is something the detection engineers at Red Canary have learned to work around, and we devise strategies for success despite these (and sometimes more) blind spots. The overall strategy that works for us is to pursue an intelligence-driven model of detection that focuses around detecting patterns of behavior. Instead of seeking to develop specific signatures for every hacking tool in existence, we seek to detect patterns of behavior that indicate malicious persistence, privilege escalation, defense evasion, and other phases of a larger incident. This strategy helps us focus on the end goal to holistically describe a series of events that occurred on customer systems even if there are a few gaps in data to show the complete story.

Intelligence-driven detection sounds like a good idea, but how do you get to that point? Once we choose a category of adversary behavior, we look for examples of malicious techniques. For example, we look for intelligence reports, tutorials, and blog posts to understand how adversaries use things like scripts, WMI consumers, and Registry-stored payloads. From there we craft searches looking for suspicious creation of these mechanisms where possible, accepting that we only need to know that they exist and may not know what they contain. Then, we research more to discover what process data represents when the payloads are executed. Even if EDR products miss the execution of some commands or storage of a malicious payload, an adversary will make himself visible when executing a payload to achieve a larger objective.

Here are a couple of examples of behavior patterns we’ve used to identify larger adversary objectives even when the EDR data didn’t have the full picture:

For credential theft, we’ve crafted detectors identifying unexpected programs loading samlib.dll and vaultcli.dll. These code libraries are legitimate parts of the Windows operating system, but loading into the same process together at the same time is often an indication of Mimikatz usage. True to form, adversaries have adapted to bypass this signature and there are now builds of Mimikatz that don’t load unnecessary libraries. Other patterns we’ve observed for credential theft involve using Sysinternals ProcDump or a similar tool to dump sensitive portions of memory containing credentials. Even if these tools are used in one of the blind spots mentioned earlier, we’d see these patterns represented in process data for detection.

For privilege escalation, we’ve crafted detectors to look for suspicious process chains. One such chain is when the Windows Event Viewer (eventvwr.exe) spawns a suspicious child process. This technique is often used to bypass User Account Control in Windows, and it is easily included in scripts or stored payloads. Even if we don’t see the original payload, we can still recognize this behavior, fit it into a larger incident, and develop hypotheses on where to investigate for more potentially malicious artifacts.

Focus on The Goal

Security products are imperfect, and they definitely have blind spots. It’s not the end of the world, and it’s not always a good reason to throw out a product that works for you. The best summary I can give is to focus your detection controls around a larger goal. Seek to identify behavior patterns that indicate a larger objective such as persistence, privilege escalation, or credential theft. Learn how adversaries achieve these objectives through researching threat intelligence reports and blog posts. By keeping a relentless focus on this goal, you can compensate “good enough” detection with the ability to recognize how pieces of an attack fit into a larger objective, ultimately allowing you to detect attacks even when you can’t see all of their constituent parts.