Bypassing Application Whitelisting: How IT Teams Can Detect It

Let me start by saying that if you’re looking for a deep dive into bypassing application whitelisting, this probably isn’t the right place to start. The intent of this article is to provide an overview of what bypassing application whitelisting means and how it looks from the view of an endpoint. The challenge security teams face is that even after going through the arduous process of determining which applications to whitelist in the first place, merely whitelisting an application is not enough. Knowing what patterns of behaviors the application should be exhibiting is also imperative.

As a security analyst at Red Canary, I often see creative subversions of whitelisted applications that wouldn’t warrant a second glance if they were only looked at superficially. This post will walk through a very basic overview of application whitelisting, why it’s a good thing, and—most importantly—how to look for signs that it’s being bypassed. Stay tuned for our next threat detection post, where we’ll dig deeper into a real-world example of a built-in tool being used to get around a technical control to illustrate how the irregularity looks when monitoring endpoint behavior.

What Is Application Whitelisting?

Let’s start with a definition of whitelisting, if nothing else so it’s here for easy reference. NIST provides a good, OS agnostic definition:

An application whitelist is a list of applications and application components (libraries, configuration files, etc.) that are authorized to be present or active on a host according to a well-defined baseline.

Whitelisting is actually very effective, because unlike blacklisting where you need to know everything that’s bad and block it, you choose to define what you know is allowed. This greatly lessens the attack surface of your environment. There are a gazillion programs on the Internet that someone can go grab (intentionally or accidentally), some of which may have unintended functionality, backdoors, or just bad code. By only allowing the applications that you’ve at least marginally vetted, you will prevent the gamut of things that antivirus doesn’t have a signature for yet or that aren’t necessary for the organization to optimally function.

Recognizing that whitelisting is a good strategy, the major OS vendors out there are aware that security teams need a way to lock down their environments, and many have started providing built-in methods of whitelisting in some form or another. Some well-known options available are Microsoft’s AppLocker and DeviceGuard and Apple’s Gatekeeper. Essentially, these technologies provide a central control method to define what programs (and their related dependencies) are allowed to execute in your environment, and then nothing else is allowed to execute. They generally do this by some combination of filepath, filename, digital signature, or hash.

Why Application Whitelisting Can Be Difficult

Sounds easy enough, right? Well, a key phrase in the above definition is a “well-defined baseline.” If you’ve ever tried to look at every application on even just your own computer, you probably didn’t realize how many different applications (and multiple versions of that application) are actually running rampant throughout your organization. For example, a single instance of explorer.exe can load over a hundred different dll’s. Computers are far from static, and a constant onslaught of updates and new applications are always being released. Let’s just say it’s not exactly a configuration picnic, especially for security teams who may not be experts in…well…everything.

With this in mind, it’s easy to see how some organizations struggle just to get to the point of having an accurate whitelist, not to mention the constant challenge of maintaining it. As you can guess, the level of effort rises the more dynamic an environment is. But let’s assume that you have a pretty decent handle on what should be running. Does that mean you’re good? Unfortunately, no.

Bypassing Application Whitelisting: Why It Happens and How to Spot It

Even the most locked down environment still requires a lot of moving pieces in order for normal operations to actually occur. After all, security is great, but if employees can’t get their work done, it’s not really aiding the overall mission. So let’s assume you’ve locked down your environment to allow only the necessary applications, thus hopefully ensuring that no one is executing that new piece of malware on any of your systems (or that toolbar that Sally has to have to shop while at work). Then you run across the phrase “bypassing application whitelisting.” While a relatively straightforward concept, it may not be that intuitive without some explanation.

A lot of programs are built-in or necessary for computers to function. Many of these programs are relatively complex—sometimes by design, sometimes because everybody’s trying to push a product out on time and “hey, it works” often wins out. The downside is that with some time and effort, a curious person can find a way to get these programs to do something outside their normal intent or find some built-in functionality that could have alternate uses. In other words, it’s more about intent. It’s not that the software isn’t supposed to do something; it’s that the unintended consequence of a feature can provide it with the ability to do something that allows an entity to perform unauthorized actions. You can’t block these programs outright as many are critical to the OS actually working.

When a Whitelist Bypass Can Happen

A simple example is your computer’s ability to initiate network connections. There’s a lot of networking happening behind the scenes, not just your browser. You also want to be able to talk to the printer down the hall, or have your password changed on the domain and not just your local workstation. However, because these API calls exist, nothing prevents someone from using them to download their own script or sending data out. (Well, not nothing, but I digress.)

Though a lot of what is out there is geared towards Microsoft, it’s important to note that a whitelist bypass can happen with any application, not just OS files. I suspect a lot of times researchers focus on built-in programs versus third-party applications because OS-related programs are almost “guaranteed” to be on every system by default. Third-party applications are going to vary much more widely across industry and technology segment, therefore potentially having a more limited attack surface. Spending a lot of time figuring out the inner workings of a scientific program only found in three labs across the whole U.S. has a much more limited impact than focusing on something that exists in almost every household. (Unless, of course, an attacker really wants into those labs; then it might be worth it.)

Looking Beyond Whitelisting: Monitoring Endpoint Activity to Detect Irregular Behavior

So does this mean whitelisting is not useful? Not at all. It just means that no one solution is ever perfect. Preventing everything all the time is impossible, especially when the line you draw on a program’s usage is often not due to its inherent functionality, but the intent and actions taken with that functionality. Think of something as simple as SCP, which is used to copy files between systems. Many in the IT field would argue that they can’t do their jobs without it. However, it can also be used to send files outside the network, or to copy sensitive information to a host on the network that is not as protected. It gets even more fun with something like a compiler; without it, developers cannot create new software. However, because it exists on the system, someone can use it to compile a program tailored specifically to their needs, without needing to send an actual executable over the wire. This is where actively investigating what is normal in a given environment becomes key.

Learn about Red Canary’s approach to managed detection and response.

Key Takeaways for Defenders:

Application whitelisting is a sound practice and allowing only trusted programs to run can be a very effective way of preventing unwanted software from executing. The problem is that people are going to use things that you trust in order to do things you don’t want. There are ways to detect abnormal behavior, but that means not only figuring out what is in your environment but figuring out what it normally does. Some examples to think about: What applications normally make network connections? Does a process normally launch child processes? If so, is it only a select few?

Unfortunately, endpoint data is generally overly voluminous and not very intuitive—but the flip side is that with volume can come a lot of insight. Just looking at a couple common executables or dll’s by running a query that shows you all the arguments passed to them, and trying to weed out things that happen frequently can start to help you dial in when something doesn’t look right. It’s a moving target and it’s never going to be perfect, but hey—that’s why humans are still needed.