The adversary's gift: When one technique opens a Pandora's box

One of the greatest rewards of establishing detection coverage is catching an adversary in the act, which then shines a light on new techniques and behaviors, enabling the further expansion of coverage. Nearly every confirmed detection offers an opportunity to expand coverage. It’s as if adversaries give us the gift of all their secrets the moment one of them is revealed. This highlights the relevance of the intruder’s dilemma that we, as defenders, have the privilege to exploit.

A gift within a gift

Working the queue, Thomas Gardner received one such gift from an adversary based on a detection that caught an XMRig coinminer being dropped on Linux to a hidden directory within the /dev/shm directory. This directory is rarely used legitimately to execute code and is known to be used by adversaries as a non-persistent, temporary file store. Not long after discovering attacker abuse of the /dev/shm directory, Tony Lambert established initial coverage around the execution of code within hidden temporary directories. He also referenced these behavioral indicators in an accompanying post about hunting for Linux-based indicators.

The downloading and dropping of a cryptominer was only caught by our /dev/shm detector, and an indicator hit for a known XMRig hash. A table requires at least three legs, and detecting adversary activity based on only two detectors—a win in itself—also establishes an opportunity to build a third leg by investigating opportunities to expand coverage based on observed behaviors. This is why it is so important to have surrounding telemetry in spite of it not being initially tied to the original detection.

Unwrapping

While investigating the /dev/shm event, Thomas noted the following surrounding artifacts and behaviors:

1. The xhide tool was used to masquerade command-line execution, giving the coinminer the appearance of running as a legitimate system daemon process: /usr/sbin/systemd -k start
2. The artifact of the xhide tool is that the masqueraded process it spawns is a child process of the xhide process. So if one could reliably detect xhide usage, then one could reliably correlate xhide to the malicious code it executed.
3. chmod was used to mark a hidden file within the user’s .config directory as executable.
4. The impersonated systemd process executed from a temporary directory and established an external network connection.
5. A known cryptominer process executed from a hidden directory.

All of these observations served as initial hypotheses for establishing new detector coverage. With the telemetry we have, we know the actual file path and hash of the code that executes in spite of command-line masquerading. So while xhide could be used to evade naive, strictly command-line-based detection logic, it generates enough of its own signal to be detectable on its own. This is a common dilemma that adversaries face: their evasive tooling may be good at avoiding the artifacts it is designed to evade, but, in the process, it almost always generates its own unique artifacts from which further detection coverage can be established.

Regifting

Red Canary detection engineers arrived at the following conclusions upon testing the established detection hypotheses:

• While it is unlikely that a legitimate systemd process would execute from a temporary directory, it is even less likely that it would establish an external network connection.
• Cryptominers pose a gray area when it comes to detection because they are not sufficient in isolation to establish intent, considering they may be used legitimately. We still track cryptominers with the knowledge that it is unlikely that cryptominers will be used legitimately from within hidden directories.

There was one additional detection idea that was hypothetically promising but failed to stand up to reality. At first, marking code as executable with chmod in hidden directories seemed sufficiently uncommon, and so we developed a bit of targeted detection logic to alert on this. However, as we tried to implement this logic at scale, it turned out that developers frequently use chmod to signify executables in hidden directories. In the end, this was not a feasible detection analytic.

A thank you note

So thanks to Thomas’s observations and the validation of his hypotheses by our detection engineers, we were able to quickly expand our coverage for Linux-based threats. Not only do we benefit from increased coverage for the specific threat, but we also arbitrarily increase coverage for unknown threats that employ overlapping tradecraft. So while technically, increasing coverage for a specific threat in the wild is reactive in nature, that same expansion of coverage is proactive in catching new, distinct threats. There is only so much tradecraft that is reasonable, and, being human, adversaries are always likely to overlap tactics, techniques, and procedures.

Continuing to iterate on this detection engineering process and philosophy can yield exponential growth in net new detection ideas, particularly among security teams that are just getting started. Shortly after the detection referenced in this blog, the same adversary changed much of their observable behavior. There was no more /dev/shm, and they’d taken to renaming xhide to evade brittle detection. However, they were still storing part of their payload in a .config directory, and, thanks to our earlier investigation, we were able to catch the behaviors despite the changes. Further, our analysis of the later detections (where the adversaries changed up their tradecraft and attempted to conceal payloads) allowed us to fine tune our existing xhide coverage and explore more detector ideas, like hidden folder masquerading. In the end, one detection analytic led to six or seven across just two confirmed threat detections.