Mastering Kubernetes security: Safeguarding your container kingdom

In this blog post, we’ll outline a Kubernetes security model and analyze the threat landscape to provide security analysts with a better understanding of their organization’s Kubernetes environment, enabling them to enhance its security.

What is Kubernetes?

Kubernetes is an open source container orchestration framework that features a variety of components. We’ll be discussing the components listed in the simplified diagram below.

• Cluster: The entire Kubernetes environment
• Master node: The node hosting management components
• Worker node: Nodes responsible for running pods
• Pods: Groups of one or more containers, defined by the administrator
• API server: The core component that handles all API requests and serves as the control center
• ETCD: A key-value database storing the cluster’s state
• Kubelet: An agent running on all Kubernetes nodes, responsible for updating the API server
• Kubectl: Command-line utility to interact with the API server

The Kubernetes threat landscape

Threats targeting Kubernetes can be broadly classified into four categories: Code, Container, Cluster, and Cloud. This classification is known as the 4Cs model of cloud-native security. We will not be discussing the Cloud category in-depth due to its scope.

It is also important to note the differences between managed and self-managed Kubernetes environments. These two environments have distinct architectures, leading to differences in their telemetry sources.

In managed Kubernetes environments, the vendor provides administrators with a fully configured cluster and assumes the responsibility of maintaining, scaling, and applying security patches to both the underlying hardware and the Kubernetes cluster. This means administrators only interact with Kubernetes objects and are shielded from the aforementioned node operations, creating a serverless experience where they lack direct access to the kernel. Consequently, there is no direct access to host telemetry, necessitating an alternative method for collecting container telemetry, often achieved through injector or sidecar containers.

In contrast, self-managed environments do not face this limitation, since traditional EDR agents retain the capability to collect container telemetry as well as traditional endpoint telemetry such as processes, file modifications, and network connections.

Code: Vulnerable applications

Adversaries often focus on externally exposed vulnerable applications, which can become their primary attack vector. When these applications are hosted on pods, vulnerabilities could potentially enable unauthorized access to the underlying pod. Mapping out the details for this category of the 4Cs model can be challenging because the path of exploitation largely depends on the specific characteristics of the application.

Fortunately though, many of the endpoint-based detectors remain relevant. For instance, consider a pod running a vulnerable Apache web application. The process chain to read a sensitive file may resemble runc -> apache -> cat /etc/shadow. Except for runc, the same process chain would occur if the web application was hosted on the node.

Container: Insecure containers

For both the Container and Cluster categories, it is crucial to maintain a distinction between pods and containers, despite their interdependence. A pod is a Kubernetes object encompassing one or more containers, essentially acting as an abstraction layer above the containers. Therefore, before delving into pod security, we should first analyze container security.

Much like the Code section, providing a comprehensive list of all considerations in container security can be challenging. Containers are essentially a set of processes that adversaries can abuse in a wide variety of ways. Here are several patterns of abuse worth looking out for:

Malicious images

An adversary can deploy a pod with a malicious container. In this scenario, the processes within the container execute activities such as running a malicious script or conducting reconnaissance. This is often observed with cryptominers and backdoors. AquaSec reported on one such instance of this with the XMRig miner.

Suspicious registry locations

Malicious containers are frequently stored in untrustworthy registry locations. Similar to phishing domains, adversaries may employ typosquatting attacks on container registries. The whitepaper “Exploring the Uncharted Space of Container Registry Typosquatting” presents a proof-of-concept exploitation conducted by researchers. The paper reveals the widespread occurrence of this behavior, the challenges associated with its detection, and the minimal cost involved in executing such an attack.

On-the-fly containers

Adversaries can abuse on-the-fly containers to circumvent static and signature-based security measures. By dynamically creating containers in real-time, adversaries can make it challenging for traditional security tools to detect and respond. This contrasts with pod creation methods for privilege escalation (discussed below in the Cluster section) in two aspects: it doesn’t require execution within a Kubernetes environment, and because the image is built locally on the host, there’s no need to connect to a remote registry.

Image layers

Developers often embed sensitive information, such as passwords, API keys, or other credentials, directly into container image layers for configuration purposes. This practice, while convenient, introduces significant security risks. Adversaries gaining access to these embedded details could potentially escalate their attacks. This InfoWorld article highlights how JFrog Xray’s new Secrets Detection feature caught AWS credentials leaked through Dockerfile environment variables.

Note that it is important to adjust the level of scrutiny when reviewing telemetry from containers. While activities like extensive port scanning, accessing sensitive files, or downloading cloud tools are often encountered at the host level, they become less typical and more unusual within a container. Containers are intended to be short-lived processes that serve a specific purpose and are not generally intended for administrative tasks.

Cluster: Misconfigurations

For this layer of the 4Cs model, we will delve into Kubernetes-level attacks. This builds upon Code and Container sections and encompasses Kubernetes-specific objects such as pods, service accounts, role-based access control (RBAC), and misconfigurations in API settings.

RBAC misconfigurations

Every pod is assigned an associated service account which is automatically mounted within the pod during its creation process. This configuration grants applications inside the pod access to the Kubernetes API server. While it is needed for some applications, administrators should exercise caution to avoid unintentionally granting excessive permissions to service accounts, which could potentially provide adversaries with significant access if they compromise a pod.

Imagine a scenario in which the default service account has been associated with the cluster-admin role, providing unrestricted access to the entire cluster. This can happen if you create a clusterrolebinding between the cluster-admin cluster role and the “default” service account.