Black Hat: Detecting the unknown and disclosing a new attack technique

In a Black Hat briefing on Thursday afternoon, Red Canary director of applied research Casey Smith disclosed a new attack technique that leverages dbgsrv.exe (dbgsrv)—a signed and trusted Windows binary—to establish remote TCP connections, conduct process hollowing, and bypass application whitelists. As far as we know, no one had publicly discussed or demonstrated this threat technique previously. Of course, the threat does not rely on a vulnerability or exploit but rather on dbgsrv’s legitimate remote debugging capabilities.

Despite having no advanced knowledge of the dbgsrv technique, Endgame senior threat researcher Ross Wolf managed to discover it while threat hunting with Event Query Language (EQL) in a dataset containing simulated threats, Atomic Red Team tests, and normal user behavior.

The premise

In fact, this was precisely the premise of Casey and Ross’s Black Hat briefing, “Fantastic Red Team Attacks and How to Find Them!” Casey’s goal was to target a developer system with a unique attack technique and gain control over that system via a reverse shell; Ross’s task was to identify the full scope of the compromise, including this new attack technique.

As such, Casey set up a virtual environment running Sysmon to record endpoint telemetry, loaded that environment with routine user behavior that you’d expect to find on a developer’s machine, and conducted a variety of simulated attacks in the environment, including the yet-unknown dbgsrv technique referenced above. He then packed up those Sysmon logs in a JSON file and shipped them off to Ross.

Ross analyzed the endpoint logs from that system using EQL—an open-source architecture for event collection that supports hunting and real-time detection. Beyond knowing nothing of the dbgsrv technique, Ross didn’t have any advanced knowledge of the other attacks that Casey had conducted in the test environment. Throughout the bulk of their briefing, Ross walked through the EQL queries he used to examine some 350,000 events from Casey’s test machine, differentiating routine user and computer behavior from Atomic Red Team tests and actual malice.

Finding unknown evil

Tools like Atomic Red Team can help security teams generate—and subsequently hunt for—telemetry associated with known attacks, but how do you hunt for unknown threats? As Ross rightly pointed out in their briefing, you can’t simply search for indicators or tooling artifacts. Finding the unknown requires searching for behaviors that you observe over time. That means you have to be collecting telemetry on the endpoint where the action is actually happening.

Ross safely assumed that the vast majority of the 350,702 unique events recorded in Casey’s test environment would be benign. Given the enormity of this dataset, he needed some way to pare it down. What follows is an abridged version of Ross’s hunt, which he explained in detail at Black Hat.

Threat Hunting in EQL

Having worked with Atomic Red Team (the open-source detection testing platform that Casey helped develop and maintain), Ross had an inkling that Casey’s attack techniques would likely rely on Windows utilities—or living off the land binaries (LOLbins). To this point, Ross started his search by running an EQL query for any binaries that had been renamed and executed. This turned up nothing. Then he asked if there were any callbacks that were established from binaries that are commonly used in LOLbins, which, after reduction, returned a handful of results.

Nearly half of the events immediately seemed to be Atomic Red Team tests and chain reactions emanating from the Atomic Red Team Github repository. Simple research revealed that a couple of the other events, like the “AmazonAssistant” one in the image below, were legitimate.

Another of the events here involved explorer.exe spawning PowerShell directly and without arguments, which, in turn, made a network connection to Microsoft. A quick search in EQL for PowerShell instances spawning from Explorer and making network connection returned nearly 50 results.

However, nearly all of these results eventually pointed back to Atomic Red Team tests.

Another event involved PowerShell invoking Mimikatz, which was part of an Atomic Red Team chain reaction (made to emulate APT 32) that Ross had already examined.

The final LOLbin that Ross observed making a network connection involved installutil.exe (a Microsoft installer tool) loading a compiled HTML (CHM) file. You can see this in the image above. Ross identified this as malicious, and Casey confirmed that the CHM file contained a .NET assembly with Meterpreter shellcode loaded into it.

However, this was not the novel technique that Ross was looking for.

Ross then searched for unique PowerShell commands that, unlike the previously examined PowerShell, contained arguments.

This search returned three results, all of this related to Atomic Red Team tests.

Ross’s next search looked for files that were created by non-system users—but later executed as system. He was essentially looking for potential privilege escalation events, but this also returned nothing.

From there he went back to another tried and true hunting method: looking for network connections coming from processes that don’t typically make network connections. So, he searched for the 15 most recent processes that made their very first network connection.

The 15 returned results included 11 events where Microsoft utilities were communicating normally with expected destinations or where Ross was seeing routine developer activity.

And then there were four

One of the remaining four events was the installutil.exe activity that Ross had already confirmed as malicious. Another contained MSIexec installing two software packages, one relating to Amazon Assistant (not pictured below), which, again, had already checked out, and another downloading a debugging tool, which seemed benign enough.

As you can see in the above image, there was the dbgsrv debugging tool and, maybe most suspiciously, notepad.exe making an external network connection. Further examination showed that Notepad and the debug tool had both made a network connection back to the same destination. At this point, Ross noted that dbgsrv had stopped looking so benign. In fact, further examination showed that the tool was reaching out to Azure and returning a different IP address.

Here Casey confirmed that Ross had located the new threat technique.

What is dbgsrve.exe?

Casey explained that dbgsrv enables the creation of a remote connection—essentially a reverse TCP connection—using a trusted, signed binary. It’s functionally equivalent to a process hollowing or a whitelist evasion tool.

More specifically in this instance, Casey had configured an attacker controlled machine—this was the destination for the Notepad network connection—to receive a callback from dbgsrv, as if it was conducting legitimate remote debugging. Casey changed the parent process so that, when he executed dbgsrv, it came from explorer.exe. Once he had a connection between the two machines via dbgsrv, he was able to take his Meterpreter shellcode and inject it into the Notepad process. Ultimately, this gave Casey remote access to the developer system.