I recently had the opportunity to talk with a number of security decision makers at a security event in Chicago. As much as I enjoy discussing the impact Red Canary has on our customers’ security postures, it’s even more enjoyable for me to simply talk shop with other security folks. What kind of problems are they facing? What solutions are they considering? The initiatives I heard about were what you would expect: “Looking to buy EDR/UBA/Analytics/Next-Gen.” However, when we dug into their current pain points, many of these teams were dealing with challenges like flat networks, no central patch management, and (my personal favorite) everyone having domain admin rights. It goes without saying that these are problems.
The CISOs and Security Management I talked with all seemed to legitimately want to do the right thing to secure their environments. Unfortunately, a lot of cybersecurity vendors have convinced the industry that security is a box you can just go buy. I saw it firsthand at the event. Take your pick of “Math,” “AI,” and “Machine Learning”—all the buzzwords were out in full force, being promoted as the solution to all the evilz. There’s nothing wrong with these products, and I’m sure many of them offer insight into one’s organization. The problem is, no product can take the place of basic IT fundamentals.
In the wake of the WannaCry and Petya/NotPetya outbreaks, security vendors came out in droves explaining how their product could have prevented it. But you know the most effective prevention techniques to prevent WannaCry? A good patch management process, effective network segmentation, and solid IT practices. The patch for the vulnerability WannaCry exploited was two months old at the time of the attack.
Read more: 5 Practical Techniques and Countermeasures to Prevent Ransomware
These are problems we solved years ago; you could go so far as to say these are foundational security controls. The 5 IT fundamentals I’ll discuss here are not exceptionally groundbreaking or exciting, but they are critical when evaluating the strength of your security posture.
1: Know What You Have
This will not come as a surprise to you, but you cannot defend what you do not know about. This is a seemingly simple concept, but how many times do you hear about an old workstation being pulled out of a closet and placed on the production network after not being patched for two years? Or an endpoint that has been compromised for months, but there was no visibility because patch management or antivirus agents stopped reporting in?
Take inventory from two fronts: physical and software.
Physical Inventory
You should know the physical location of every endpoint your organization owns. Or, if they are transient, know the user assigned to the equipment. This can help you to quickly locate a device in the event of a compromise. For larger organizations, barcoding systems can help keep track of endpoints (specifically user workstations) as they move around the company.
Once you know what systems you have and where they are, you will want to monitor for any future unauthorized systems attempting to talk on your network. You can control access to network resources with solutions such as NAC or 802.1x, or implement a tool such as arpwatch to monitor new devices that show up uninvited.
Software Inventory
Software inventory is a bit more ephemeral, as software can be deployed and removed rapidly throughout an environment. Keeping up with a known inventory of software is important for several reasons. One is knowing your impact when a zero-day vulnerability undergoes active exploitation in the wild. Another is knowing when unauthorized software is present on your network.
Related Reading: Why identifying unwanted software is critical
Generally, approved software, and the enforcement of it, is one part policy and one part technology. Establishing a written policy of what software is approved for company use and who is allowed to install and uninstall it is the first step.
There are various ways to determine software installed on an endpoint. Powershell and Wmic are two quick ways on Windows systems. At Red Canary, we help our customers baseline the software on their endpoints by using Carbon Black Response and our own open source tool Surveyor.
The key for both software and hardware inventory is to establish a system of record for your systems and approved business software running on them, and keep it up to date. Anything outside of these baseline documents should be treated as hostile until proven otherwise.
2: Clean Up Your House
Sometimes simplicity makes for poor security. In the past, I have been called into several smaller organizations that have one simple network boundary. Workstations sitting on the same network with domain controllers and file shares are problematic for several reasons. They often do not leave room for basic network securities such as firewalls between workstations (arguably the most compromise-prone endpoints in any environment) and critical servers and infrastructure. An attacker landing on one of these endpoints can quickly walk the entirety of the network, making the compromise of high value systems much simpler and faster.
So what should defenders do?
Network segmentation!This is a great way to chop your network into defensible sections, with the opportunity for protections between every segment. Create network segments for your end user endpoints (by department if possible), your different classes of servers, and other critical infrastructure. At the end of the day, a VLAN costs you nothing, so go nuts!
For anyone looking to really dive in on this, there are countless books written about network security architecture and building defensible networks. For everyone else looking for the Sparknotes version: one of the most basic principles is limiting your attack surface. That is, limit the ports and protocols each system on your network is serving, and restrict those services to only the endpoints and networks required to perform the business function of the system.
For instance, if you have an internet facing web server, only the required protocol (hint: this should only be HTTPS) should be available to other systems on the internet. If you’re smiling because you have a border firewall, first go and remove the “allow all” at the bottom of your firewall rules that someone added in lieu of troubleshooting.
Joking aside, protections are needed in three main places:
- At your network border
- Between your network segments
- Between your endpoints within a network segment
The first two are the most common, but many organizations do not employ firewall restrictions between their endpoints. The good news is, there is no need to purchase an expensive host-based firewall solution. The built-in firewalls for Microsoft (Windows Firewall), Application Firewall (OSX) and Unix systems (iptables/firewalld) are extremely effective on their own.
As a recent example, restricting SMB traffic East-West between workstations with a host-based firewall would greatly reduce the impact of WannaCry, Petya/NotPetya, and other malware wielding the ETERNALBLUE exploit.
Related Reading: A quick way to mitigate ETERNALBLUE
There certainly are a few good examples where end-user workstations need to communicate directly with each other, such as broadcast traffic, peer to peer applications, and some security tools. However, all collaborate file sharing should be served from a central server with a good offline backup solution, not from your workstations. In short, allow access between your systems judiciously, and log all connection failures for evidence of potential malicious behavior (or broken legitimate business systems; firewalls are unforgiving).
3: Close the Holes
So often in breaches, attackers are able to gain unauthorized access and privilege to systems, causing untold financial and reputational damages to organizations. And they aren’t doing it by wielding zero-day exploits that have never been seen before; they are simply exploiting long-known vulnerabilities in outdated software.
Patching systems and software is an unsexy, much maligned task that is often put on the backburner for any number of reasons. Operations complains about uptime, end users complain about killed queries and unsaved documents. Overall, no one gets excited when a new patch is released.
I’ll say something crazy: You should be patching within 7 days from the release of a patch. Or at the very least, trying to patch in that timeframe. (Read between the lines the importance of software inventory.)
Every organization has different tolerances for downtime across infrastructure, specified change windows, and change management processes, etc. The truth is that you need documented patch management procedures that account for the needs of the business, but are trimmed down to the smallest amount of time your systems sit unpatched on your network. The most effective way to ensure you are covering these bases is to treat security patches like software development. Here’s how.
How to Effectively Manage Security Patches:
1: Deploy to Dev — The first round of patches should be applied to your development environment, or systems with the highest tolerance for failure. You will want systems in this class to mirror high-uptime production systems (in platform version and software if possible) to work out any issues ahead of time. Including pilot workstations and servers from across the organization ensures that you have good coverage over the majority of your systems and applications.
2: Deploy to Qual — If no issues are encountered in your pilot deployment within 24-48 hours, roll out to a broader series of systems. These systems will include less critical, but still in-production systems.
3: Deploy to Prod — After a defined period with no encountered issues, blast the patches to the remainder of your environment. Congratulations, you’re patched (for now)!
Pointers to Make Patch Management Less Painful:
- Establish a feedback loop — Give the system owners of your pilot systems good communication mechanisms to alert if something explodes. It is critical to get good feedback early in the deployment process so you can adapt before taking down production systems.
- Have a solid, tested backout plan — There should be mechanisms in place to quickly uninstall a patch in case of system failure. This being said, do not go full knee-jerk if a problem is reported; find the specific conflict, identify which systems are likely to be impacted, and deploy around them
- Have strong, tested backup restoration procedures — You should periodically test your backups as a matter of course. Having a backup is a good feeling, but knowing confidently that it can be restored to a fully working system is priceless. In the event that a patch completely destroys an application, you will have a solid tool to restore everything to normal in short order.
4: Become Big Brother
Knowing the systems talking on your network is an important first step, but to quickly identify and respond to threats, you must have deep, system-level visibility into the activity on said systems. Here at Red Canary, we have spent years using Endpoint Detection and Response (EDR) tools to gain visibility into system activity for the purpose of application and process analysis and incident response. We’ve written extensively about the benefits of leveraging EDR tools in organizations of all sizes, which I will not rehash here. The one takeaway regarding EDR is that having a single tool to collect detailed process and network telemetry across every endpoint in your environment is incredibly valuable for proactively identifying threats.
But what if bringing in a commercial EDR tool isn’t in your current budget? Fear not. Building good visibility into your security program is possible leveraging built-in system capabilities and open source tooling.
In general, most platforms have native logging functionality that can be tuned to record whatever level of data you want—ranging from almost nothing, all the way up to an overwhelming deluge of data. While the initial instinct may be to LOG ALL THE THINGS, this is not an effective strategy. Most security teams could never review this amount of data without extensive tooling and significant tuning.
So how do we decide what should be logged and what to leave out? Suggestion: Start with a framework. There are several established guides that can provide a baseline logging policy to help you configure audit log settings for each of your platforms. Compare your use cases, ensure you are collecting logs to support them, and adjust accordingly.
A few frameworks to explore:
For Windows-based systems, the Sysmon utility is an incredibly powerful EDR sensor released for free by the Sysinternals team at Microsoft. This agent can collect all of the telemetry data you would expect from an EDR tool, logging specified event data into a custom Microsoft event log. Out of the box, Sysmon’s reporting of system activity is a bit basic. However, several security professionals have tuned Sysmon configuration files to provide a solid baseline of relevant data for security operations and threat hunting; two that I like are by SwiftOnSecurity and Michael Haag.
Collecting this telemetry data on each endpoint is a good start, but not conducive to efficient analysis. Thankfully, Microsoft introduced Windows Event Forwarding several years ago to facilitate the compilation of system logs from many systems in a single place. All you need is an extra server (or several, depending on the size of the environment) with adequate storage to act as a collection point. All logging settings can be defined via group policy, ensuring your logging settings are applied across your entire domain.
Most Unix-based systems and network devices have the ability to forward system audit information via syslog configuration. These logs can be aggregated with an rsysolg server, or similar tool.
Once you have a good handle on capturing and analyzing system logs, consider adding the following:
- Middleware (Database logs, Web Server logs, etc)
- Application logs (specifically web applications sitting in your DMZ)
Collecting audit logs is an important first step, but is a waste of time and resources if logs are not actively analyzed on a regular basis. Work toward having at least one dedicated analyst reviewing log data and triaging alerts as their primary task. For smaller teams, try to dedicate as much daily time as possible to reviewing this data. To make life simpler, consider deploying a Security Information and Event Management (SIEM) solution to collect, parse, and correlate your disparate log sources. There are several open source options [SOF-ELK, ELSA] that can be deployed with minimal expense. However, keep in mind that tuning and maintaining a robust SIEM solution will typically take at least one full-time engineer to support.
5: Limit Your Admins: Restrict the Keys to the Kingdom
In the course of running a security program, you will often feel that the end users on your network are out to get you. They click every malicious link, download cat screensavers bundled with malware, and when presented with an email promising a free gift card, they will happily type their credentials into whatever logon box the “proxy” prompts them with. Giving these users administrative privileges is like handing them a loaded gun, one which eventually they will use on you (or themselves).
The principle of least privilege, or the idea that a user’s account is provisioned with only the essential rights and permissions required to perform their job duties, is one of the more commonly ignored core security principles. It is somewhat common sense that a customer service representative does not need rights to administer your Domain Controller (though I have unfortunately seen this in several environments). Why, then, do so many user accounts on so many networks have full, unfettered administrative rights that are not in any way required for their scope of work?
The answer is often a mix of:
- Lack of process – It is not unheard of for individuals to be responsible for installing their own departmental software, especially in organizations with limited support resources. It can also be difficult to change this culture because “This is how we’ve always done it.”
- Ease of administration – It is often easier to grant a user administrative rights than it is to troubleshoot application issues. Some poorly written applications may require additional configuration after installation to run without administrative rights, and an overwhelmed support technician is likely to simply grant the rights to make the problem go away.
- Politics – In many organizations, members of the upper management hierarchy may feel entitled to install software and configure their systems as they see fit. These battles are often left unfought so as to not ruffle any feathers.
So why is this a big deal? Imagine that a user receives a phishing email with a link (which they click of course) allowing a bad actor remote access to their workstation. If this user has standard, non-administrative rights, the actor is limited to these same constraints, and would need to escalate privilege prior to leveraging many of their tools. However, if this user is logged in with an administrative account, the actor immediately has access to disable security controls, load software, harvest credentials, etc. If the user has administrative rights across multiple systems on your network, the actor is able to very quickly move through the network unhindered.
Aside from the increased risk of compromise, allowing users administrative rights gives them the ability to circumvent security controls (i.e., disabling antivirus because it consumes too many system resources) and approved software policy.
How do you tackle the problem?
- Understand the scope of the problem — Collect the user membership of the Administrators group (Windows) and the sudoers, wheel, admin, and adm groups (Unix/Mac OSX). There are ways to do this in an automated fashion which will make this process less painful.
- Validate the need — Some users legitimately require administrative rights, but “I need to download and install software” isn’t one of them. Expect this to be painful, as many users will fight for administrative rights without fully understanding what they actually do.
- Develop an approval process — There will be legitimate business needs for administrative rights. Ensure that you give users a route to authorized rights
- Revoke where you can — Begin removing rights from users without a legitimate business need. Note that these users will likely require additional support for the functions previously handled with their administrative rights, so have these processes in place to make the transition less onerous.
- Monitor for changes — Once you have cleaned up administrative rights, watch membership changes to any administrative group diligently. Any unapproved additions could be a sign of malicious activity, or the least, policy violation.
- Recertify regularly – Approved administrators should have their rights recertified annually. A regular approval process cleans up unnecessary rights as individuals change job roles and responsibilities
Other administrative rights best practices:
- Separate Admin Account – All administrators should regularly log in and perform daily functions (email, web browsing, etc) with a standard, non-administrative account. For all administrative functions, the user can elevate those processes specifically.
- No Shared Accounts – Administrators should have an administrative account assigned to them specifically, rather than using a generic built-in “Administrator” account. This allows for accountability and non-repudiation for all actions performed. If the local administrator account is leveraged, consider implementing a solution such as Microsoft’s LAPS (https://www.microsoft.com/en-us/download/details.aspx?id=46899) to provide secure, audited access to this account.
Conclusion
As I said before, there is nothing groundbreaking in here. These concepts are foundational. And every organization should be able to check these boxes. At Red Canary, we see firsthand the impact proper IT and security hygiene has on an organization’s risk. Talk with your team about these concepts, assess where you need to make improvements, and make that your next project, not evaluating the next potential silver bullet. Our hope is that we all can champion “Better Security through Better IT.”
More Security Advice & Best Practices
- How to Use Windows API Knowledge to Be a Better Defender
- Five Guidelines for Measuring and Reporting on Your Security Operations Program
- Security Architect Lessons: What I Learned Managing and Assessing Cyber Risk at a Fortune 200
- Using Tabletop Simulations to Improve Your Information Security Program
- Passive DNS Monitoring – Why It’s Important for Your IR Team
Related Articles
How AI will affect the malware ecosystem and what it means for defenders
How AI will affect the malware ecosystem and what it means for defenders
Why Taylor Swift fans should work in cybersecurity