Skip Navigation
Get a Demo
 
 
 
 
 
 
 
 
 
Resources Blog Security operations

Mastering Kubernetes security: Safeguarding your container kingdom

In the era of containers, Kubernetes has emerged as the de facto framework for container orchestration. But this widespread adoption has also introduced several new attack vectors.

Madhav Nakar
Originally published . Last modified .

In this blog post, we’ll outline a Kubernetes security model and analyze the threat landscape to provide security analysts with a better understanding of their organization’s Kubernetes environment, enabling them to enhance its security.

What is Kubernetes?

Kubernetes is an open source container orchestration framework that features a variety of components. We’ll be discussing the components listed in the simplified diagram below.

Diagram of the components of a Kubernetes cluster

  • Cluster: The entire Kubernetes environment
  • Master node: The node hosting management components
  • Worker node: Nodes responsible for running pods
  • Pods: Groups of one or more containers, defined by the administrator
  • API server: The core component that handles all API requests and serves as the control center
  • ETCD: A key-value database storing the cluster’s state
  • Kubelet: An agent running on all Kubernetes nodes, responsible for updating the API server
  • Kubectl: Command-line utility to interact with the API server

The Kubernetes threat landscape

Threats targeting Kubernetes can be broadly classified into four categories: Code, Container, Cluster, and Cloud. This classification is known as the 4Cs model of cloud-native security. We will not be discussing the Cloud category in-depth due to its scope.

It is also important to note the differences between managed and self-managed Kubernetes environments. These two environments have distinct architectures, leading to differences in their telemetry sources.

In managed Kubernetes environments, the vendor provides administrators with a fully configured cluster and assumes the responsibility of maintaining, scaling, and applying security patches to both the underlying hardware and the Kubernetes cluster. This means administrators only interact with Kubernetes objects and are shielded from the aforementioned node operations, creating a serverless experience where they lack direct access to the kernel. Consequently, there is no direct access to host telemetry, necessitating an alternative method for collecting container telemetry, often achieved through injector or sidecar containers.

In contrast, self-managed environments do not face this limitation, since traditional EDR agents retain the capability to collect container telemetry as well as traditional endpoint telemetry such as processes, file modifications, and network connections.

Code: Vulnerable applications

Diagram of a malicious user communicating with a pod in a container's worker node

Adversaries often focus on externally exposed vulnerable applications, which can become their primary attack vector. When these applications are hosted on pods, vulnerabilities could potentially enable unauthorized access to the underlying pod. Mapping out the details for this category of the 4Cs model can be challenging because the path of exploitation largely depends on the specific characteristics of the application.

Fortunately though, many of the endpoint-based detectors remain relevant. For instance, consider a pod running a vulnerable Apache web application. The process chain to read a sensitive file may resemble runc -> apache -> cat /etc/shadow. Except for runc, the same process chain would occur if the web application was hosted on the node.

Container: Insecure containers

For both the Container and Cluster categories, it is crucial to maintain a distinction between pods and containers, despite their interdependence. A pod is a Kubernetes object encompassing one or more containers, essentially acting as an abstraction layer above the containers. Therefore, before delving into pod security, we should first analyze container security.

Much like the Code section, providing a comprehensive list of all considerations in container security can be challenging. Containers are essentially a set of processes that adversaries can abuse in a wide variety of ways. Here are several patterns of abuse worth looking out for:

Malicious images

An adversary can deploy a pod with a malicious container. In this scenario, the processes within the container execute activities such as running a malicious script or conducting reconnaissance. This is often observed with cryptominers and backdoors. AquaSec reported on one such instance of this with the XMRig miner.

Suspicious registry locations

Malicious containers are frequently stored in untrustworthy registry locations. Similar to phishing domains, adversaries may employ typosquatting attacks on container registries. The whitepaper “Exploring the Uncharted Space of Container Registry Typosquatting” presents a proof-of-concept exploitation conducted by researchers. The paper reveals the widespread occurrence of this behavior, the challenges associated with its detection, and the minimal cost involved in executing such an attack.

On-the-fly containers

Adversaries can abuse on-the-fly containers to circumvent static and signature-based security measures. By dynamically creating containers in real-time, adversaries can make it challenging for traditional security tools to detect and respond. This contrasts with pod creation methods for privilege escalation (discussed below in the Cluster section) in two aspects: it doesn’t require execution within a Kubernetes environment, and because the image is built locally on the host, there’s no need to connect to a remote registry.

Image layers

Developers often embed sensitive information, such as passwords, API keys, or other credentials, directly into container image layers for configuration purposes. This practice, while convenient, introduces significant security risks. Adversaries gaining access to these embedded details could potentially escalate their attacks. This InfoWorld article highlights how JFrog Xray’s new Secrets Detection feature caught AWS credentials leaked through Dockerfile environment variables.

Note that it is important to adjust the level of scrutiny when reviewing telemetry from containers. While activities like extensive port scanning, accessing sensitive files, or downloading cloud tools are often encountered at the host level, they become less typical and more unusual within a container. Containers are intended to be short-lived processes that serve a specific purpose and are not generally intended for administrative tasks.

Cluster: Misconfigurations

For this layer of the 4Cs model, we will delve into Kubernetes-level attacks. This builds upon Code and Container sections and encompasses Kubernetes-specific objects such as pods, service accounts, role-based access control (RBAC), and misconfigurations in API settings.

RBAC misconfigurations

Every pod is assigned an associated service account which is automatically mounted within the pod during its creation process. This configuration grants applications inside the pod access to the Kubernetes API server. While it is needed for some applications, administrators should exercise caution to avoid unintentionally granting excessive permissions to service accounts, which could potentially provide adversaries with significant access if they compromise a pod.

Imagine a scenario in which the default service account has been associated with the cluster-admin role, providing unrestricted access to the entire cluster. This can happen if you create a clusterrolebinding between the cluster-admin cluster role and the “default” service account.

 kubectl -n k8s create clusterrolebinding BAD-RBAC  --serviceaccount=k8s:default --clusterrole=cluster-admin

Upon obtaining initial access to the pod—for instance, through exploiting an application—an adversary can perform various actions, including the creation of additional pods.

kubectl auth can-i create pods --as system:serviceaccount:k8s:default

Depending on the type of excessive permission, an adversary can conduct reconnaissance, deploy malicious containers or escalate privileges by creating insecure pods (as we’ll see in the next section).

While it largely depends on the environment, allowing the default service account to create new pods is often not required. This is only one example of a service account bound to a role with excessive permissions. Other examples include permissions to view secrets, list deployments, or other actions that can expose an organization’s development setup.

Remediations:

  • Disable the automountServiceAccountToken flag by setting it to false, which can be done in either the pod YAML file or serviceaccount YAML file. This action serves to block access to the service account from the pod.
  • Develop standardized RBAC objects to reduce the necessity for administrators to create them, as this could lead to increased security vulnerabilities due to potential errors.
  • Create network policies (firewalls for pods) to limit unwanted network communication.

 

Pod specifications: securityContext

One of the more prominent ways to escalate privileges in a Kubernetes environment is to create pods. The security context field in a pod specification file defines the security permissions of the pod and the container.

Consider the following pod:

Screenshot of a Kubernetes pod with overly permissive securityContext

 

A few security concerns to note:

  • The securityContext grants the pod excessive permissions:
    • privileged (line 10): maps the user id 0 inside the container to user id 0 of the host. This gives the container unfettered access to the host.
    • readOnlyRootFilesystem (line 11): setting this to false allows a user to write files inside the container. This can allow an adversary to drop additional files post initial access violating the immutability of containers.
    • capabilities (line 12): Gives the container root-equivalent access via the CAP_SYS_ADMIN permission. This serves the purpose of finely delegating permissions to containers without granting full root access. For instance, granting a container the NET_ADMIN permissions for network configuration. These should be monitored carefully to prevent unintended access.
  • Line 19: Mounting the /etc hostpath gives the container access to sensitive files such as /etc/shadow.
  • It is important to note that Kubernetes namespaces are not security boundaries and a privileged pod carries its permissions across the cluster. In other words, Kubernetes namespaces serve as a means for developers to structure the cluster, rather than functioning primarily to confine permissions and privileges.

There are several permutations and other flags that can cause a pod to be insecure. This blog from Bishop Fox does an excellent job of highlighting them.

Remediations:

  • Configure an admission controller to review pod creation requests. This can require an intimate knowledge of the environment because preventing all of the above settings may not be feasible.
  • Use a tool like kubesec to conduct a static security analysis of pod YAML files.

 

Exposed kubelet API

Trend Micro conducted an assessment of exposed Kubernetes APIs and found that the kubelet API was often exposed. This is especially dangerous because Kubernetes audit logs do not log that activity.

In the following output, unrestricted and anonymous access has been granted to the kubelet API on one of the worker nodes.

Screenshot of exposed kubelet settingsChanging kubelet settings to expose it publicly

 

This change allows an adversary from a remote host to query cat /etc/shadow on one of the running pods.

Screenshot of malicious query from remote host

Building upon this, adversaries have the capacity to download binaries and scripts. The threat actor TeamTNT, for instance, followed this procedure to take advantage of /runningpods and /run (referred as endpoints) to start mining for the Monero cryptocurrency on nearly 50,000 IPs.

Other Kubernetes attacks

This section highlights a few more noteworthy attacks that I wanted to mention due to their significance. While I could delve deeper into these, I’ll keep it concise, as detailed information is readily available in excellent open source resources.

Interacting with the Metadata server

  • Cloud service providers offer a Metadata API that provides information about the running virtual instance. This can be exploited by adversaries who may use it to directly retrieve credentials by querying the API.
  • This technique was utilized by ScarletEel as discovered by Sysdig.

Anonymous Kube-Apiserver access

  • Admins often use this setting for configuration testing. If not reverted to false, it can lead to unauthorized access to the cluster as system:anonymous, posing a significant issue, especially if there are permissions associated with the user system:anonymous or user group system:unauthenticated.

Misconfigured Kubectl Proxy commands

  • Kubectl Proxy establishes a proxy server between the user’s machine and the Kubernetes API server, facilitating local access to the cluster’s APIs. When configured incorrectly, an adversary can execute Kubernetes commands with the same level of privilege as the user who established the proxy.
  • This blog from AquaSec goes into more detail about the above two attacks.

Exposed sensitive services

  • The exposure of ETCD and Kubernetes Dashboard services may lead to information leakage about the cluster. ETCD serves as the state of truth for the whole cluster and is not encrypted by default. The Kubernetes Dashboard is a web-based UI that allows administrators to manage the cluster.

 

Conclusion

We hope that this deep dive into cluster architecture will help you better understand and protect the containers in your environment.

 

Engineering a MDR solution for Microsoft Azure

 

The benefits of GenAI by SOC function

 

Manage your SOC like a product

 

The RSA Conference talks we’re looking forward to most

Subscribe to our blog

 
 
Back to Top