Damage from Malicious Admins and Credential Access

Good security sometimes requires us to get back to basics on a number of things, including how we use and secure administrative credentials. Admin accounts enable us to configure all sorts of technologies, from software installations and Windows network controls to WordPress servers. If you can administer it, odds are good that there’s a special account for it. Because these accounts have a high level of privilege, they need to be kept safe to prevent unauthorized activity changes.

In the mix of trying to secure their administrative accounts, organizations can get caught up on the idea of a system administrator exceeding their authorization to shut down systems, add backdoors, and ransom access upon getting fired. After my post on Lateral Movement with WinRM, a security engineer wrote an article on mitigating “bad admin” problems where administrators sought to exceed their own privileges in a network. These scenarios show up a few times per year in the news and are often referred to as “rogue admins.”

But what about bonafide bad guys that somehow obtain administrative accounts? This is essentially an unauthorized user that has gained privileged access into one or more systems across your enterprise. The MITRE ATT&CK Matrix categorizes this sort of compromise as Credential Access, and it’s just as important as the threat of a bad admin. The good news is, detection and mitigation of Credential Access will cover the threat of bad admins at the same time.

Let’s take a look at how Red Canary observes the compromise and misuse of administrator credentials. Then, we’ll share some tips on how to detect and prevent admin account abuse and credential theft.

The Purloined Credentials: A Tale from the Cyber Incident Response Team

A Red Canary customer recently participated in a rather loud and effective red team engagement. Our security analysts began detecting odd activity one night with the execution of two suspicious binaries, one of which behaved like a web proxy.

Our analysts noted the first binary, svchost.exe, executing from an odd location. This process usually doesn’t live in the “inetsrv” folder, because that folder holds binaries related to Windows IIS web servers. In addition, the binary didn’t possess any of the signing or other metadata properties we usually see with Windows processes. Next, it spawned another suspicious binary that masqueraded as a Windows process, this time the WMI Provider Host (wmiprvse.exe).

Just like the first, this binary didn’t have the usual signing or metadata information, but this one took another step: it began loading code into memory typical of network or web activity. Our hypothesis for this binary was that it served as a point for an adversary to pivot further into the customer’s network.

The real fun began the next morning. Using incoming endpoint data, we noticed numerous endpoints in the customer’s environment executing a suspicious binary. The execution was followed by some account enumeration activities using the “net group” command, and output was saved to a file. On the first few endpoints, account enumeration was limited to looking for Domain Administrator accounts. This was a clear red flag because this group is highly privileged in Windows networks and allows administrators to reconfigure and export data related to Active Directory domains (which the customer had).

Pretty soon, the adversaries upped the ante by enumerating another group, Enterprise Administrators. Windows admins should be sweating bullets now, because that group gives a higher level of privileges that can manage and export data from numerous Active Directory domains if your domain is part of a larger organization.

What About That Suspicious Binary?

Around the same time the enumeration activity began, we observed a binary named mimnew64.exe. Endpoint data indicated that binary hadn’t been seen in the customer’s environment before the day of execution, but on that day it was observed on numerous endpoints. Given the context, we hypothesized that the binary was a recompiled version of Mimikatz, based upon several observations.

Dynamic-Link Library (DLL) code loaded by the binary was consistent with behavior we’d expect from Mimikatz, and it made a cross-process memory read to the Local Security Authority Subsystem Service (lsass.exe). We see legitimate apps make this memory read all the time, but it always stokes our curiosity when a binary we haven’t seen before does it. In addition, the binary executed as the local SYSTEM account because it had been spawned from the Windows Service Host process (services.exe). This combination of context provided a perfect storm where Mimikatz could be used to dump password hashes from workstations.

Armed with this hypothesis about the suspicious binary, we had a new larger hypothesis as well: the adversaries were attempting to dump credentials for all the workstations they could touch until they encountered a workstation a Domain Administrator had logged into. Pretty soon this hypothesis was tested by events reported from the customer’s Active Directory domain controller…

On the domain controller we first noticed that a “whoami” command had been issued via PSExec. While this itself isn’t malicious, it’s a good first indicator that something is amiss during incidents. Adversaries often verify which account they’re using to see if they need to escalate privileges.

At some point, the adversaries realized they were on a domain controller and switched tactics into data theft mode. They spawned an instance of “vssadmin.exe”, a tool to manage Windows Volume Shadow Copies.

As our analyst noted in the detection, shadow copies are used to copy and backup files in use by Windows (like a credential database). However, use outside of backup utilities is suspicious. In this case, the adversaries created a shadow copy and began working with that shadow copy. First, it appears they copied the domain controller’s SYSTEM and SAM registry hives into a network share for file sharing or hosting.

Next, they extracted a file named “ntds.dit”. Again, the Windows admins out there should be sweating: the bad guys just obtained a copy of the Active Directory database containing usernames and password hashes! The extracted files were copied to a network share for exfiltration.

How Could This Happen?

Nobody would willingly give administrative credentials to bad guys, right? Clearly there was a breakdown somewhere, because someone was able to perform actions only a Domain Administrator should: an Active Directory export. In this case, it appeared that the admin account was compromised because it had been used to log into workstations on the network rather than just the domain controllers. Once the adversaries ran their suspected Mimikatz on a workstation and found password hashes belonging to a Domain Admin, they could dump those hashes and use them to further authenticate as that account.

How to Detect Credential Access

This threat detection brings forth the question: How can we detect this type of activity? By definition, administrator accounts can move freely about enterprise systems without many limitations. It’s a bad situation when adversaries gain access to these credentials, because they can move as freely as your admins.

The solution is less than sexy: implement audit controls and know your environment. Many newer systems incorporate audit systems that let you know when someone accesses a privileged account through notifications in event logs or other means. This is possible in Windows Active Directory environments by enabling auditing options with Group Policy Objects. Something to consider is that administrators can sometimes disable audit controls. In cases where this is possible, it is a good idea to audit when these controls change.

Once audit controls are in place on whatever systems you use, you can use those controls to find out when administrative accounts exhibit odd behavior. Look for outliers in your data. Is an account logging into a system it never logs into? Is it logging in at odd hours when the person owning that account is asleep? Is it executing suspicious commands? Answering each of these questions requires you to know your environment and how the real life people operate.

While partners like Red Canary can help to detect unauthorized credential access and bad admin activity, context must be associated with the activity in order to differentiate between authorized admins and adversaries. At the end of the day, an organization’s security team knows more about the specifics of their environment and can identify this activity more effectively than anyone.

5 Tips for Preventing Credential Theft

1: Start with good hygiene.

Most of the time, credential theft can be prevented with good security hygiene. Don’t give passwords to unauthorized parties, don’t store credentials in areas or forms where unauthorized people can easily access them. Depending on the systems in your network, this hygiene may need to extend further. If your credentials need to be transferred over a network connection (such as to a web server) make sure the network connections use encryption. Use HTTPS instead of HTTP. Use SSH instead of Telnet.

2: Understand where your credentials are stored or cached.

Windows systems cache user credentials by default when logging in. This means that a Domain Administrator’s credentials could be cached on workstations if they are used to troubleshoot or fix the system. This sort of caching can be dangerous, and Microsoft has added functionality to newer versions of Windows to let you specify users that do not cache credentials. In addition, Windows 10 added a feature called Credential Guard to help protect against these threats.

3: Minimize use of administrator jump-boxes.

As with lateral movement, minimizing the number of computers used to login with administrator credentials will help you focus your efforts to harden the systems.

4: Implement multi-factor authentication.

This requires top-down management support but it greatly decreases the damage stolen credentials can cause. These solutions ensure that just a stolen password or hash cannot be much use without an additional token or other mechanism to prove their identity.

5: Limit potential damage.

Outside of these hygiene measures, the next best thing you can do is structure your network and its systems in ways that limit the damage done by malicious admins. If possible, generic Administrator credentials on computers should be different for each system. In Windows, Microsoft has provided LAPS, which helps achieve this goal. This goal is harder to achieve in Apple and Linux environments, but it’s still possible with some work. Lastly, consider establishing processes that help you rapidly disable or de-provision compromised accounts. Both of these goals will help decrease the potential for damage during an attack or if one of your legitimate system administrators leave.

Credential theft can be a huge issue for IT organizations, but with some time and planning you can put safeguards in place to help secure your network and keep powerful admin accounts in the hands of people you trust.