Slaying Evil Around the Clock with Red Canary’s Cyber Incident Response Team

Red Canary’s Cyber Incident Response Team (CIRT) is comprised of two groups: detection engineers and incident handlers. Our blog posts often focus on threats we detect, but it’s rare to get a glimpse of our incident handlers in action. This article will walk through a recent threat in a customer’s environment, from the initial discovery to the incident handling team’s response and remediation with the customer.

Part One: the Detection

One relatively quiet evening, Red Canary’s detection engineering team was investigating events, looking for signs of suspicious or malicious activity. The evening shift is used to seeing events that include system admin updates and migrations because that’s when organizations typically set automated and scripted tasks to run.

But on this particular evening, one of the analysts noticed some unusual activity coming from PowerShell that appeared to be related to cryptomining activity.

“PowerShell was reading a registry key and made an external network connection to a Monero mining domain—a clue that’s frequently associated with suspicious activity. My curiosity was piqued. I wanted to understand how it got there and what it was doing.” – Tony Lambert, Detection Analyst

Tony quickly determined that something was seriously amiss. The Kaseya Virtual Systems Administrator (VSA) tool had exhibited behaviors that were highly suspect and often associated with malware. Others on the team corroborated the odd behavior by searching for indicators to identify other instances within the environment.

“It was on 70 hosts, which was an indication that the event queue was about to light up,” explained Matt Saunders, Detection Analyst. “We quickly put out the word to the incident handling team and warned them that a slew of high malicious detections were being written up.”

The screenshot below shows the identification of the malicious behavior. It started with the Kaseya agent spawning the command line and PowerShell, instructing it to make an outbound network connection and download a .ps1 file.

From there, an outbound network connection was made and Windows Registry power settings were written, ending with values consisting of a to e and x.

The attack progressed with the creation of an XML file that utilized PowerShell to create a scheduled task, as shown below. We frequently see attackers use the Scheduled Task technique to establish persistence within the environment. In order to evade detection, the task was named ReportScriptErrors.

The following screenshot shows the Windows Service Host (svchost.exe) that spawned PowerShell to run the Base64 encoded payload from the powersettings registry key previously created. As a result, an outbound connection to a domain associated with a Monero mining activity was made.

It quickly became apparent that the root cause of this cryptomining behavior was likely the Kaseya VSA. The software had a previous vulnerability and the customer had not yet implemented a patch in their environment. Because Kaseya was a trusted network tool, the malware quickly spread through the customer network.

“Once Tony found it, the team ramped up to deal with it. Everyone stayed up late to respond to the incident. It’s always cool to see the energy on the team when we focus on tackling an issue together.” —Julie Brown, Detection Analyst

As the detection engineering team tackled the onslaught of events, the incident handling team dug in and coordinated response with the customer.

Part 2: Handling the Incident

Joe Casazza, a Red Canary incident handler, led the remediation effort. In order to verify that the malware could no longer spread, he needed to work with the customer to remove aspects of persistence that the malware had created and plan a massive environment-wide restore. This was a challenging and time-consuming task, as the malware was located on more than 70 devices across the environment.

“Since this was a known attack vector, we first needed to find source material for the actor and the identified threat. This was easy because we had the telemetry about the actor’s actions and how it was carried out.

Next, we needed to verify the findings in order to make sure we got everything, including the registry keys writes, the PowerShell execution, file writes, and scheduled tasks. The mining was done via PowerShell making network connections, so that needed to be shut down.” – Joe Casazza, Incident Handler

Joe pulled in additional expertise to expedite the remediation. He ran communications with the customer while other handlers jumped in to write scripts, isolate endpoints, and execute other tasks required to remediate the environment.

The incident handlers continued to work with the customer on the incident, running through different scenarios and updating the response plan. As the night turned into early hours of the morning, a script was developed and run against all the devices in environment. It removed all elements of persistence so that the malware would not be able to return autonomously. After working with the customer to validate that workstations and server endpoints were restored and the network was successfully connected, the incident handlers called it a night.

The next morning, employees at the customer’s organization returned to work, without any indication of the night’s previous events. The security director sent the following email to Red Canary’s CEO Brian Beyer:

“I just wanted to share how impressed we were with the way Joe handled the issue. I know there were other people on the team, helping with scripts and such, but Joe was the ‘voice’ on the line all night. The 5 hours last night provided massive value.” – Executive Cybersecurity Director, Red Canary Customer

Key Takeaways & Best Practices

Even the most trusted software has vulnerabilities that can be exploited. As is common in many of the breaches we see, this incident could have been avoided by having a regular patching program in place. Help safeguard your network by deploying a patch management program and following other IT fundamentals.

If you’re using Kaseya VSA, make sure to check out these updated Kaseya Detection and Removal Procedures to prevent similar cryptomining incidents.

Like this article? Read about other threat detections involving cryptocurrency mining.