It was a Friday afternoon when the alert came in. One of Red Canary’s customers had experienced a breach. The compromise occurred on an unsecured endpoint—an isolation development box that was used for testing.
The customer had deployed Red Canary across its most critical endpoints: domain controllers, front-facing web server, executive endpoints, databases, and other endpoints where a compromise could have a serious impact. In this case, the customer had not deployed Red Canary on the compromised machine because it was not connected to privileged information. After receiving an alert from a third-party, the customer quickly discovered that they were dealing with a full-blown compromise with an attacker who had root access.
The customer’s Security Analyst pushed a Carbon Black Response sensor to the box and reached out to Red Canary for support.
The Hunt Begins: Threat Investigation #5207
Joe Casazza, a Red Canary incident handler, reached out to Joe Moles, Red Canary VP of Customer Security Operations, to help conduct a threat investigation on the fly. Moles had already seen the host name appear and inquired where the new system had come from.
Part of the value of having the Red Canary solution deployed is that it tracks all endpoint activity and performs ongoing behavioral analysis to identify threats. But because the sensor had not been deployed on the machine prior to the compromise, establishing root cause and finding the attack vector would be very difficult. The detection engineering team would not have the historical endpoint telemetry to understand where the process spawned and how the attack originated. However, they could do an ad-hoc analysis and provide insights and recommendations on how to best handle the threat.
The Red Canary team quickly jumped into action:
Step 1: Laying the Groundwork
The team prepared for the investigation by testing and troubleshooting to make sure the newly deployed sensor was providing the required data. The customer understood the difficulties posed by not having historical data and knew a root cause analysis was most likely not possible. More than anything, they wanted Red Canary’s advice on what to do next and to help ensure they didn’t miss anything.
Step 2: Baselining the Environment
Casazza and Moles started working to better understand the endpoint and the scope of the breach. Moles was seeing a high volume of activity spawning from the ssh daemon (server side process for inbound ssh connections) with suspicious events that indicated post-intrusion recon, intel gathering, and post-exploitation.
For the remainder of this article, you will see screen captures from the actual detection report that was sent to the customer.
Moles began looking for services running, active net connections, and credentials stored in memory, and scanned outward connections to find “juicy” targets. His first objective was to validate which users and activities were known good so he could weed out false positives.
Step 3: Finding the Attacker
To trim down the noise, Moles worked with the Security Analyst to identify all activity associated with the customer’s triage and response efforts. Moles was then able to isolate unknown activity while removing all activity from the stream that didn’t have to do with the compromised root account.
“Usually Red Canary has a head start because we’re already familiar with the systems in customers’ environments, but in this instance you’re trying to find the actor and the actor is trying to find you.” —Joe Moles, VP of Customer Security Operations
Step 4: Analyzing Behavior and Intent
The Red Canary detection engineering team observed data gathering, file writes, and execution of randomly named binaries. It was difficult to determine the attacker’s actual intent based on the beacon mechanisms. Was this activity associated with exfiltration of data, delivery of other malware, or a distraction from their real mission? Because the machine was not connected to sensitive data, the customer was able to use the ongoing investigation as a learning experience.
Then things got interesting.
Step 5: “We have an active outbound connection to China”
Someone in China had established a remote connection to the box with back-and-forth communication. The new outbound connection indicated reverse shells, data exfiltration, and similar malicious activity.
Casazza had been using Red Canary’s Surveyor tool to look at the device and see if any of the activity and data being compiled was related to passwords. As Casazza was talking to the customer’s Security Analyst, a randomly spawned binary started and made an outbound connection, indicating the attack was still in progress.
Rather than continuing to gather data, the team agreed to shut down the connection immediately. Casazza showed the Security Analyst how to isolate the host through the Red Canary portal. With the host isolated and the connection shut down, the threat was contained…for the moment.
Step 6: Creating an Action Report
On Saturday morning, Casazza sent a formal report, outlining the previous evening’s activity. The customer’s Director of Security indicated they were still seeing activity on the endpoint. The device was still spawning binaries and attempting to make network connections. Clearly an automated persistent mechanism was in place.
Step 7: Linux Incident Response
Casazza recommended the Director collect all data on the device and assess any credentials that might have been compromised. The security leader wasn’t familiar with the specifics of forensics imaging on this Linux box, so Casazza brought in another member of the Red Canary incident handling team with Linux expertise to walk the customer through the process of performing backups, imaging, forensics, and using Carbon Black Response to remotely shut down the device.
As a result of Red Canary’s support, the customer was able to secure the endpoint and ultimately their environment. Following the incident, the customer passed along this message:
“I’ve been so impressed that you guys would take the time to support a smaller customer like us. It is nice to know you have someone at your back.” —IT Security Analyst
When asked about the incident, Casazza commented, “I don’t think this was going above and beyond. This is what we do. If a customer is in trouble, we help them out.”
Key Takeaways from the Threat Investigation
Situations like these are not uncommon in development environments. Red Canary recommends standardizing controls across dev, test, and production environments as a first step in any security program. Minimize or prevent development systems from having access to the internet and make sure development systems have the same level of security visibility as production systems.
And lastly, find partners that will support you during an emergency. Red Canary partners with organizations ranging from small and mid-sized firms to the Fortune 500, to help defend their endpoints against the worst types of threats. If you are looking for a strategic partnership, let Red Canary show you how we are different.
READ THIS NEXT: Want Better Security? Start With These 5 Proven IT Fundamentals