How an IT Service Provider and Red Canary Stopped a Malware Outbreak

Most IT service providers can relate to the following scenario: It’s an idle Thursday. You just got to the office and your phone rings. One of your customers is panicking because their entire network has been compromised by rapidly spreading malware. This can be a disastrous situation—not only for the affected company, but also for the IT service provider if they’re caught unprepared.

Luckily, the IT service provider in the story I’m about to tell was prepared. The Sr. Security Engineer reached out to Red Canary to help provide the deep visibility and rapid response his customer needed. In this article, we’ll walk through:

• A behind-the-scenes view of the incident
• How Red Canary partnered with the IT service provider to quickly respond and eradicate the threat
• Best practices organizations can adopt in order to avoid a breach

Watch our on-demand incident response webinar to learn how to cut mean time to remediation and reduce the impact of incidents.

The Call for Help

As an incident handler at Red Canary, my job is straightforward: help make security better for our customers. When the IT service provider called us for help, I jumped on a conference call with their senior security analyst, as well as Joe Moles (Customer Security Operations) and Michael Haag (Threat R&D). The goal: put an incident response plan in place to eradicate the malware that was spreading through the customer’s network.

Jumping Into Action

Within a few minutes, we’d put together a straightforward action plan:

1. Gain visibility
2. Detect the threat
3. Execute a response

Let’s take a deeper dive into each step…

Step 1: Gain Visibility

Visibility lays the groundwork for any successful incident response plan. If you can’t identify an adversary in your environment, you can’t detect them and thus cannot perform appropriate remediation. Bottom line: visibility is crucial.

Red Canary utilizes endpoint sensors from industry-leading Endpoint and Detection Response (EDR) tools in order to provide visibility on endpoint workstations and servers. I worked with the IT service provider to deploy sensors for Carbon Black Response across their customer’s entire environment. This would not only provide the visibility we needed to detect the malware, but also deliver the tooling to quickly respond.

Within minutes of deployment, endpoint telemetry began reporting into the Red Canary platform. Once our team saw the telemetry, it became very clear that the affected customer was dealing with a rapidly spreading variant of malware, referred to as a “network worm.” The most challenging part of dealing with a worm is that as soon as a cleaned machine rejoins the network, it becomes a new target for the malware to spread again.

Step 2: Identify the Threat

We all know how the traditional security product game works. A customer deploys a new product that detects “All the things.” However, the customer can’t respond to ANY of the things because of the overwhelming number of false positives mixed in with the actual threats. The result is alert fatigue that causes real threats to get lost in the noise.

Part of our job at Red Canary is to filter through that noise, only notifying customers about threats that truly matter. We try to do so in a way that is meaningful and helpful, with analysts adding notes and observations to our Threat Detections to explain what the customer is looking at.

Within 90 minutes of initial contact with Red Canary, the IT service provider began receiving Red Canary Threat Detections. Below is an example of the observed activity.

Right away, we could see an oddly named binary executable called helpwmi.exe spawning out of the c:\windows\syswow64 directory. Immediately afterward a network connection was made to a malicious IP address. The activity continued with multiple randomly named unsigned binary executables. These indicators have all the signs of a malware compromise.

Step 3: Eradicate the Threat

Within two hours, the customer’s security staff was armed with IP addresses to block due to the command and control activity that our team observed. In addition, 20 known malicious binary hashes throughout the infected customer environment could be banned with confidence because of the malicious behavioral actions associated with them.

One of the key features of Carbon Black Response is having the ability to isolate an endpoint on a network. The Red Canary portal allows this to be done right from the detection timeline.

The IT service provider used this tooling to isolate key infected endpoints and prevent the malware from spreading to more endpoints. Within 48 hours of initial compromise, all of the customer’s impacted endpoints were back under full control and the threat had been eliminated.

Mission Accomplished

The IT service provider sent this nice thank-you note to our team:

“After your help during a recent outbreak, our CEO was elated. Standing this up took far less time and effort than expected and has already yielded excellent and very actionable results. It has also helped us convince our customer that an accurate asset inventory is indispensable.

Thanks again, you have saved us and our customer a LOT of headaches, time, and money.” —Sr. Security Engineer, IT Service Provider

Lessons Learned

In situations of incident response, it’s critical to be prepared. Here are several guidelines every organization can follow in order to help resolve incidents and prevent them from occurring.

1: Act quickly.

When dealing with a highly motivated adversary, you need to have the infrastructure and processes in place to gain visibility into what is occurring so you can quickly and effectively respond. Adversaries can completely take over an unsuspecting network in a matter of hours.

2: Keep it simple.

Incident response doesn’t have to be overly complicated. Put together a simple plan that will get your company back up and running as efficiently as possible. It is important to stick to a plan during a compromise in order to prevent making rash or emotional decisions during the initial stages of the attack.

3: Manage inventory.

You can’t protect what you don’t know about. Take time to get over the hurdle of asset management so that when the time comes to protect all your endpoints across your organization, you know exactly where they are and how to safeguard them from further compromise. We continue to invest in better asset visibility in the Red Canary platform because this is so important.

4: Control administrative access.

Users who have administrator-level access and don’t need it are severely detrimental to an organization’s overall security posture. When an adversary is trying to compromise a network, the first goal is to get administrator level access. If all end users have administrator access by default, it makes it easier for the adversary to completely take over an entire network. Instead, use the the principle of least-privilege, and create access levels that correspond with what users need to accomplish their jobs.

5. Work with Red Canary.

Just kidding. Well, kind of. In situations like these, it’s important to have partners and expertise you can trust. We’re here to help when you’re ready!

Editor’s Note: This blog post was originally published in March 2018. It has been updated based on our ongoing work with security teams looking to improve their incident response processes.