Etcd is more than just a storage component in Kubernetes; it’s a critical part of the architecture. It’s a key-value database that stores all the cluster’s information. The data in etcd is stored using a serialization format called Protobuf, developed by Google for efficient data exchange between systems. Kubernetes uses Protobuf to serialize different kinds of data, such as pods and roles, each requiring different parameters and definitions.

The research describes a tool called auger, which can serialize and deserialize data stored in etcd into more readable formats like YAML and JSON. NCC Group has developed a wrapper for auger called kubetcd to showcase how a compromised etcd can be critical.

However, exploiting etcd has limitations. You’d need root access to the host running etcd and have the necessary certificates for authentication. Moreover, this technique mainly applies to self-managed Kubernetes environments, not managed ones offered by cloud providers.

Direct access to etcd could be used to tamper with Kubernetes resources, like changing the startup date of a pod or creating inconsistencies that make pods difficult to manage.

Direct access to etcd, the key-value store for Kubernetes, could allow an attacker to make unauthorized modifications to the cluster state, which could lead to various security issues. Here are some examples of how this could be exploited:

Changing Pod Timestamps:

• Attackers with access to etcd could alter the creation timestamps of pods. This could be used to disguise malicious pods as long-running, trusted processes.
• Example:
  bash kubetcd create pod nginx -t nginx --time 2000-01-31T00:00:00Z
  This command sets the timestamp of an nginx pod to January 31, 2000, making it appear as if it has been running for over 20 years.

Gaining Persistence:

• By changing the path where a pod’s data is stored in etcd, an attacker could prevent the pod from being deleted by the regular Kubernetes API commands.
• Example:
  bash kubetcd create pod maliciouspod -t nginx -p randomentry
  This command creates a pod and stores its data under a different path, making it difficult for Kubernetes to manage or delete it.

Creating Semi-hidden Pods:

• Attackers could manipulate the namespace entries in etcd to run pods in a namespace that does not match their manifest. This can cause confusion and make pods difficult to manage.
• Example:
  bash kubetcd create pod hiddenpod -t nginx -n invisible --fake-ns
  This command creates a pod that appears to run in the default namespace but is actually associated with the invisible namespace in etcd. This pod will not be listed under the default namespace, making it semi-hidden.

Bypassing Admission Controllers:

• Admission Controllers enforce security policies in Kubernetes. By injecting resources directly into etcd, an attacker can bypass these controls and deploy privileged pods that could compromise the cluster.
• Example:
  bash kubetcd create pod privilegedpod -t nginx -n restricted-ns -P
  This command injects a privileged pod into a namespace that is supposed to be restricted by Pod Security Admission (PSA) policies.

Tampering with Cluster Roles and Role Bindings:

• Attackers can modify roles and role bindings directly in etcd to escalate privileges.
• Example:
  bash kubetcd modify rolebinding admin-binding --clusterrole=cluster-admin --user=attacker
  This hypothetical command changes a role binding to give the attacker user cluster-admin privileges.

These examples show the potential dangers if an attacker gains direct access to etcd. They can make changes that are not subject to the usual Kubernetes API checks and balances, leading to unauthorized control over the cluster’s resources. This is why securing etcd access is critical in a Kubernetes environment.

Mitigation

To mitigate the risks associated with etcd and prevent the kinds of tampering mentioned earlier, several steps and best practices should be implemented:

Access Control:

• Restrict access to etcd by implementing strong authentication and authorization mechanisms. Use TLS client certificates to secure communication with etcd.
• Regularly rotate etcd access credentials and audit access logs to detect unauthorized access attempts.

Network Policies:

• Limit network access to etcd servers, ensuring that only specific, authorized components can communicate with etcd.
• Implement firewall rules or Kubernetes network policies to control the traffic towards etcd servers.

Etcd Encryption:

• Enable encryption at rest for etcd to protect sensitive data. Even if attackers gain physical access to the etcd storage, they should not be able to read the data without the encryption keys.

Regular Backups:

• Regularly back up the etcd data store. In case of a breach, this allows the cluster to be restored to a known good state.
• Ensure backup integrity by verifying and testing backups periodically.

Monitoring and Auditing:

• Implement monitoring to detect abnormal activities, such as unexpected changes in etcd.
• Use tools like etcd’s built-in audit capabilities, Falco, or other intrusion detection systems to alert on suspicious behavior.

Least Privilege Principle:

• Apply the principle of least privilege to etcd access. Ensure that only the necessary components have the minimum required access level to perform their functions.

Patch Management:

• Regularly update etcd to the latest version to mitigate vulnerabilities caused by software defects.

Admission Controllers:

• Use Admission Controllers like OPA Gatekeeper or Kyverno to define and enforce policies that can help prevent the creation of unauthorized resources within Kubernetes.

Security Contexts and Policies:

• Apply Security Contexts and Pod Security Policies (or their successors, like Pod Security Admission) to enforce security-related settings in pods and prevent privilege escalation.

Disaster Recovery Plan:

• Have a disaster recovery plan in case etcd is compromised. This plan should include steps to isolate affected systems, revoke compromised credentials, and restore from backups.

Education and Training:

• Train the operations team to understand the security risks associated with etcd and Kubernetes, and how to apply best practices for securing the cluster.

By implementing these mitigations, organizations can significantly reduce the risk of unauthorized access and tampering with etcd, thus securing their Kubernetes clusters. Mitigating the risks associated with etcd ensures the integrity and reliability of Kubernetes clusters. By implementing industry best practices for security and maintaining a proactive stance on potential vulnerabilities, organizations can confidently deploy and manage their containerized workloads, keeping them secure in an ever-evolving threat landscape.

The post Is Your etcd an Open Door for Cyber Attacks? How to Secure Your Kubernetes Clusters & Nodes appeared first on Information Security Newspaper | Hacking News.

The flaw, identified as CVE-2023-3676 and with a CVSS score of 8.8, affects Kubernetes’ processing of YAML files. These files are used by the container orchestration system for a variety of purposes, including configuration, management, and the handling of secrets, among other things.
YAML files are used for almost every aspect of Kubernetes’ operation, including configuration, automatic deployment, scalability, administration of containerized application pods, and a great many others.

It was discovered in 2022 that the constructor of SnakeYAML included the CVE-2022-1471 vulnerability, which allowed for the execution of remote code in applications that were susceptible.

Many vulnerabilities were found by Kubernetes itself, such as CVE-2021-25749 (workloads might execute as ContainerAdministrator even though the runAsNonRoot option was set to true), CVE-2017-1002101, and CVE-2021-25741 (race situations and symlinks in combination with the subPath subproperty in a YAML file).

When users establish a pod, they also have the option of creating a shared directory between the pod and the host. This directory is referred to as “volumes.”

In order to make the volumes accessible, the YAML file has to have the volume parameter added to it, in addition to the mountPath (which specifies the location of the container) and the hostPath (which specifies the location of the host).

Mounting the shared directory at a specific place may also be accomplished with the help of the subPath subproperty. This YAML file is then read by kubelet, which verifies every parameter in the YAML file and uses the isLinkPath inner function to make sure that no symlinks are formed on the subPath argument. A PowerShell command to determine the path type is generated by taking the subPath subproperty from the YAML file and using it as a parameter in the command. After that, the exec. Command function call receives this PowerShell command and processes it.

An further research discovered that the “exec.Command” variable may be used in conjunction with user input that has not been sanitized, which results in a command injection vulnerability.

Disabling the usage of Volume.Subpath, utilizing the open source Open Policy Agent (OPA) agent to set rules to prohibit particular YAML files, and applying role-based access control (RBAC) to restrict the number of users who may conduct activities on a cluster are some of the recommended remedies.

“CVE-2023-3676 demands minimal privileges and, as a result, establishes a low threshold for attackers. All an attacker needs is access to a node and the ability to apply privileges. “High impact coupled with ease of exploitation usually means that there is a higher chance of seeing this attack (and similar attacks) on organizations,” writes Akamai. “High impact coupled with ease of exploitation also means that there is a higher chance of seeing similar attacks.”

The post How to hack Kubernetes pods and executing malicious code remotely appeared first on Information Security Newspaper | Hacking News.

Additionally, the attacker tried to remove several already-existing deployments from namespaces like “api-proxy,” “worker-deployment,” “kube-secure-fhgxtsjh,” and “kube-secure-fhgxt.” In order to maximize the amount of CPU that was available and lower the likelihood of being found if the server was full, we presume that the attacker was stopping older campaigns or rival campaigns.

When the attacker employed RBAC to acquire persistence, that was the most fascinating component of this attack. An additional ClusterRole with almost administrative powers was established by the attacker. Because it was a cluster role, it was not restricted to any one namespace in particular. After that, the adversary established a ‘ServiceAccount’ with the name ‘kube-controller’ under the ‘kube-system’ namespace. In the last step of their attack, the malicious actor built a ‘ClusterRoleBinding,’ which bound the ClusterRole to the ServiceAccount in order to provide a persistent state that was both robust and covert.

At this stage, the attacker has generated persistence that enables for continued exploitation of the cluster. This is true even in the event that access for anonymous users is blocked. In addition, attaching the ‘cluster-admin’ role to a new or questionable user may trigger alerts; nevertheless, the adversary devised a cunning technique to blend in beautifully with the API audit logs. This was accomplished by creating a creative approach to blend in. In the end, the attacker would be able to stay under the radar without triggering any warnings by configuring the ClusterRoleBinding to access “system:controller:kube-controller.” This binding seems to be completely acceptable.

After that, the malicious actor constructed a DaemonSet in order to distribute containers across all nodes using a single API call. The container image known as ‘kuberntesio/kube-controller:1.0.1’ was included in the DaemonSet formation request object. This image was stored on the public registry known as Docker Hub.

The container image with the name ‘kuberntesio/kube-controller’ is an example of typosquatting that is attempting to spoof the real ‘kubernetesio’ account. In spite of the fact that it only has a few dozen container images, it has gathered millions of pulls. The image also imitates the well-known ‘kube-controller-manager’ container image, which is an essential part of the control plane. It is a container image that operates inside a Pod on each master node and is accountable for determining when a node has failed and taking appropriate action in response to that failure. It is a frequently used K8s component that should exist on the cluster and has the potential to fool people into believing it is a valid deployment rather than a cryptominer. This is because it should exist on the cluster. Because it is intended to function in an unceasing manner, nobody would question its existence.

This new K8s attack is  known as RBAC Buster. Its purpose is to exploit K8s API servers in order to construct a ClusterRoleBinding and acquire complete access to the cluster with persistence after the misconfiguration is fixed.

The post RBAC Buster – A new K8s attack technique to hack in Kubernetes cluster appeared first on Information Security Newspaper | Hacking News.

A cryptojacking operation using Monero was also discovered; this operation is aware of the Dero effort and is actively competing with it. The Monero campaign mines XMR on the host by elevating their privileges via the usage of DaemonSets and mounting the host as the root user.

Attackers specifically targeted Kubernetes clusters running on non-standard ports by scanning for and locating exposed vulnerable Kubernetes clusters that had the authentication setting —anonymous-auth=true. This setting enables anonymous access to the Kubernetes API and was the target of the attackers’ attention. It is possible for a user with adequate access to mistakenly expose a secure Kubernetes API on the host where kubectl is operating by performing the “Kubectl proxy” command. This is a less apparent approach to expose the secure Kubernetes cluster without authentication. The Kubernetes control plane application programming interface does not provide anonymous access out of the box in Kubernetes. Nevertheless, since the choice to make secure-by-default the default was delayed, and there are a variety of ways in which Kubernetes might be inadvertently exposed, there is still a legacy of exposed systems on the internet.

After the first engagement with the Kubernetes API, the attacker will next install a Kubernetes DaemonSet with the name “proxy-api.” On every node in the Kubernetes cluster, the DaemonSet installs a pod that contains malicious code. This makes it easier for attackers to operate a cryptojacking operation by simultaneously using the resources of all of the nodes in the network. The mining efforts that are performed by the pods are donated back to a community pool. This pool then divides the reward (in the form of Dero currency) among all of its contributors in an equitable manner via their own digital wallets.

Once the vulnerable Kubernetes cluster had been compromised, the attackers did not make any attempts to pivot either by moving laterally to attack additional resources or by scanning the internet for discovery. This is a pattern that is common among many cryptojacking campaigns that have been observed in the wild.

In addition, the attackers did not try to remove or interfere with the functioning of the cluster. Instead, they used a DaemonSet to mine Dero. The name of the DaemonSet was disguised as “proxy-api,” and the name of the miner was “pause,” both of which are phrases that are often seen in Kubernetes logs.

These targeted behaviors seem to define the goal of this campaign, which is that the attackers are only seeking to mine for Dero. This is the conclusion that can be drawn from the actions that have been taken. As a result, we have reason to believe that a cryptojacking actor driven by financial gain is the one responsible for this initiative.

Attackers have taken advantage of the fact that Kubernetes has become the most popular container orchestrator in the world to focus their attention on misconfigurations, design flaws, and zero-day vulnerabilities inside Kubernetes and Docker.

The post New cryptojacking malware can hack in Kubernetes clusters using this easy trick appeared first on Information Security Newspaper | Hacking News.

CVE-2023-22482 (CVSS score of 9.0) affects versions 1.8.2 through 2.6.0-rc4, v2.5.7, v2.4.18, and v2.3.13, and has been fixed in versions 2.6.0-rc5, v2.5.8, v2.4.20, and v2.3.14.

Beginning with v1.8.2, all versions of Argo CD are subject to an inappropriate authorisation problem that causes the API to accept certain erroneous tokens.

In signed tokens, OIDC providers add an aud (audience) claim. The value of that claim identifies the token’s intended audience(s) (i.e. the service or services which are meant to accept the token). Argo CD verifies that the token was signed by the selected OIDC provider. However, since Argo CD does not confirm the audience claim, it will accept tokens that are not meant for Argo CD.”

Users of Argo CD are urged to upgrade to a patched version of the platform as soon as possible since there are no workarounds for CVE-2023-22482.

The vulnerability, identified as CVE-2023-22736 (CVSS score of 8.5), is an authorisation bypass that enables an attacker to deploy Applications outside of the set authorized namespaces. This vulnerability was addressed in Argo CD versions 2.5.8 and 2.6.0-rc5.

There is a fix for this issue that does not need updating 

“Running just one replica of the Application controller will prevent this flaw from being exploited. Making ensuring that all AppProjects’ sourceNamespaces are confined inside the limits of the set Application namespaces will also prevent this flaw from being exploited.

After two vulnerabilities were discovered that might enable an attacker to access Argo CD and deploy Applications beyond the defined authorized namespaces, users of the Argo continuous deployment (CD) tool for Kubernetes are being asked to roll through fixes.

The post 2 Critical vulnerabilities in Argo CD allow complete take over of your Kubernetes appeared first on Information Security Newspaper | Hacking News.

1. Analysis of Vulnerability and Exposure Using Clair

Clair is a vulnerability static scanning tool that is free source and designed for containers. The application supports a variety of deployment strategies and excels in situations requiring a high level of scalability and availability. Clair is compatible with REST APIs and offers scan reports in HTML format. The CVEs database that was developed as part of the Clair project is used by the Amazon Elastic Container Registry (Amazon ECR), which then produces a list of discoveries.

2. Using Trivy to find security holes

Trivy is a scanning tool that is available as open source and is an ideal option for teams who are interested in doing static application security testing as well as rapid scans. In addition to detecting configuration problems and providing complete vulnerability identification for operating systems, infrastructure as code files, and language packages, it is also used to identify vulnerabilities in code files. Additionally, Trivy is able to automatically update its vulnerability database.

After running a Trivy scan, you will be presented with a list of vulnerabilities, along with an indication of the severity of each vulnerability and a CVE number, if one is available.

3. Using kubeaudit to do audits on Kubernetes

Kubeaudit is an open source tool that was developed by Shopify. It is used to audit Kubernetes installations against standard security measures including the following:

run in a non-root capacity
Use a root filesystem that can only be read.
Get rid of frightening capabilities and don’t introduce any more.
don’t run privileged

4. Using Kubescape for Security Testing

Kubescape is a program that can assess whether or not Kubernetes is installed in line with practically all key compliance standards, such as the NSA-CISA and MITRE ATT&CK®, in addition to DevSecOps best practices. Kubescape is an open source tool. Integration with Continuous Integration (CI) technologies is also possible with Kubescape.

5. CIS Benchmarking Utilizing the Kube-Bench Platform

Kube-bench is an open source program that determines whether or not Kubernetes is implemented in the most effective manner in accordance with the CIS Kubernetes Benchmark. This benchmark includes a collection of Kubernetes security best practices. As a result, kube-bench is most effective when it is necessary to scan just for the sake of CIS benchmarking.

Within a pod, you are able to execute the kube-bench program. The GitHub repository includes job-cloud provider>.yaml files that are cloud-specific, and kube-bench will automatically select which test set to run depending on the version of Kubernetes that is currently running on the computer.

6 .Penetration Testing Utilizing the Kube-Hunter

Kube-Hunter is an open – sourced penetration testing tool that was written in Python. It allows you to design custom modules that can be performed from local workstations, inside the cluster, and remotely in both active and passive mode. Kube-Hunter was developed by Google.

When running in active mode, kube-hunter will search for and then further exploit any vulnerabilities it finds. Additionally, you have the option of running kube-hunter on a staging environment in the form of a malicious pod, which may simulate a true attack in its own right.

Each of the three tools discussed previously, kubeaudit, Kubescape, and kube-hunter, generates a comprehensive report that includes information on misconfigurations, vulnerabilities, checks that were skipped, insecure behaviors, and more.

7. Real-Time Protection Utilizing Sysdig Falco

An open source runtime security solution known as Sysdig Falco is deployed across Kubernetes clusters for the purpose of continuous risk and threat detection. The technology functions as a surveillance camera that monitors for unusual activity, changes in configuration, intrusions, and stolen data in real time. For the time being, Sysdig Falco is the only open source solution for Kubernetes runtime security that has been authorized by the CNCF.

Falco, which allows you to write your own plugins and contains all of the major features found in its commercial tools, consumes the raw stream of system call information utilizing drives such as a kernel module or an eBPF probe. Additionally, Falco gives you the ability to write your own plugins. Falco has the capability to operate as a daemon on Kubernetes Nodes. In this mode, it is able to monitor and send real-time security warnings in JSON format through stdout, HTTP hooks, and syslog.

8. Docker Image Analysis Using Static Scanning

A Docker image may be created by following the instructions in a file known as a Dockerfile, which is an ordinary text document that includes a set of instructions. In addition, containers are the instances of Docker images that are now operating, and Kubernetes supports the Docker runtime.

Before attempting to publish your Docker images to a Kubernetes runtime environment, it is strongly advised that you do a security vulnerability assessment on them. Shifting to the left like this can help you avoid assaults on your supply chain and increase your security.

Docker Engine has at this point included Synk’s scanning capabilities in order to find vulnerabilities in the images. The list of CVEs and recommendations is what is produced as a result of doing the scan.

As you can see, the ecosystem of free source technologies that may be used to increase Kubernetes security is rather robust. You can execute automated scans, CI integrations, runtime security, penetration testing, and audit checks using one of the many tools that are accessible to you on GitHub. Utilize this article as a reference guide while you put your Kubernetes security program into action to ensure the safety of your apps, network, and data.

The post Top 8 Free Tools for security testing and audit of your Kubernetes cluster in 2022 appeared first on Information Security Newspaper | Hacking News.

Flux is an open and extensible CD solution to keep Kubernetes clusters in sync with configuration sources, and is used by firms across all industries, including Maersk, SAP, Volvo, and Grafana Labs, among many others. In its most recent version (Flux2), multi-tenant support was introduced, among other features.

The vulnerability was described as a remote code execution (RCE) error that exists due to improper validation of kubeconfig files, which define commands that will be executed to generate on-demand authentication tokens: “Flux2 can reconcile the state of a remote cluster when a kubeconfig file exists with the correct access rights,” points a report posted on GitHub.

Paulo Gomes, a software engineer who collaborates at the Cloud Native Computing Foundation (CNCF), which originated GitOps and provides support for Flux and Kubernetes, mentions: “The tool can synchronize the declared state defined in a Git repository with the cluster in which it is installed, which is the most commonly used approach, or it can target a remote group.”

Gomes adds that the access required to target remote clusters depends largely on the intended scope. This is completely flexible and is based on the fact that Kubernetes RBAC has a wide range of granularity. This behavior allows a malicious user with write access to a Flux source or direct access to the target cluster to create a specially crafted kubeconfig file to execute arbitrary code in the controller container.

When analyzed according to version 2 of the Common Vulnerability Scoring System (CVSS), this vulnerability was considered of medium severity and received a score of 6.8/10, because in single-tenant deployments, the error is less dangerous and the attackers obtain almost the same privileges required for exploitation.

However, the flaw receives a score of 9.9/10 according to CVSS v3.1, as this release includes a metric around ‘scope’ changes, which means that the flaw can affect resources beyond the security scope managed by the developers of the vulnerable component.

The flaw has already been addressed by the creators of the tool, so users of affected deployments are advised to upgrade as soon as possible.

To learn more about information security risks, malware variants, vulnerabilities and information technologies, feel free to access the International Institute of Cyber Security (IICS) websites.

The post Critical vulnerability in Flux2, a Kubernetes continuous delivery tool, enables hacking between neighboring deployments appeared first on Information Security Newspaper | Hacking News.

Below are brief reports of the detected failures, in addition to their respective tracking keys and scores assigned according to the Common Vulnerability Scoring System (CVSS).

CVE-2022-24768/CVE-2022-1025: Inadequate access restrictions would allow remote authenticated users with access to an application’s source Helm or git repository or sync</code> and <code>override access to obtain administrative privileges.

This vulnerability received a CVSS score of 8.6/10 and resides in the following Argo CD versions: 2.1.0 – 2.1.13, 2.3.0 – 2.3.1, 2.2.0 – 2.2.7, 2.0.0 – 2.0.5, 1.7.0 – 1.7.14, 1.8.0 – 1.8.7, 1.6.0 – 1.6.2, 1.5.0 – 1.5.8, 1.4.0 – 1.4.3, 1.3.0 – 1.3.6, 1.2.0 – 1.2.5, 1.1.0 – 1.1.2, 1.0.0 – 1.0.2, 0.7.0 – 0.7.2, 0.6.0 – 0.6.2, 0.5.0 – 0.5.4.

CVE-2022-24730: An input validation error when processing directory traversal streams on the endpoint /api/v1/repositories/{repo_url}/appdetails would allow remote users to send specially crafted HTTP requests and read arbitrary files on the system.

The vulnerability received a CVSS score of 5.7/10 and resides in the following Argo CD versions: 1.5.0 – 1.5.8, 1.6.0 – 1.6.2, 1.7.0 – 1.7.14, 1.8.0 – 1.8.7, 2.2.0 – 2.2.5, 2.1.0 – 2.1.10 and 2.0.0 – 2.0.5.

CVE-2022-24731: On the other hand, an input validation error when processing path traversal streams in the Helm chart would allow remote administrators to send HTTP requests specially designed to read arbitrary files on the system.

The flaw received a CVSS score of 4.6/10 and resides in the following versions of Argo CD: 1.5.0 – 1.5.8, 1.6.0 – 1.6.2, 1.7.0 – 1.7.14, 1.8.0 – 1.8.7, 2.2.0 – 2.2.5, 2.1.0 – 2.1.10 and 2.0.0 – 2.0.5.

While these flaws can be exploited remotely by authenticated threat actors, so far no active exploitation attempts related to these flaws have been detected. Still, users of affected deployments are encouraged to apply the available patches as soon as possible.

To learn more about information security risks, malware variants, vulnerabilities and information technologies, feel free to access the International Institute of Cyber Security (IICS) websites.

The post Privilege escalation and path traversal vulnerabilities affect Argo CD, the GitOps continuous delivery tool for Kubernetes appeared first on Information Security Newspaper | Hacking News.

On this occasion, pentesting experts from the International Institute of Cyber Security (IICS) will show you how to perform multiple hacking tasks on Kubernetes, including cluster disruption, certificate deletion, and node connection, all without downtime for the services being run.

Before continuing, as usual we remind you that this article was prepared for informational purposes only and should not be taken with a call to action. IICS is not responsible for the misuse of the information contained herein.

To get started, let’s remember that the main Kubernetes control panel consists of a few components:

• etcd: Used as a database
• kube-apiserver: API and heart of the cluster
• kube-controller-manager: For the deployment of operations on Kubernetes resources
• kube-scheduler: Main planner
• kubelets: To launch containers directly on hosts

Each of these components is protected by a set of TLS, client and server certificates, whose purpose is to authenticate and authorize components between them, mention pentesting experts. These resources are not stored anywhere in the Kubernetes database, except in certain cases, but are presented as normal files:

# tree /etc/kubernetes/pki/ /etc/kubernetes/pki/ ├── apiserver.crt ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key ├── apiserver.key ├── apiserver-kubelet-client.crt ├── apiserver-kubelet-client.key ├── ca.crt ├── ca.key ├── CTNCA.pem ├── etcd │ ├── ca.crt │ ├── ca.key │ ├── healthcheck-client.crt │ ├── healthcheck-client.key │ ├── peer.crt │ ├── peer.key │ ├── server.crt │ └── server.key ├── front-proxy-ca.crt ├── front-proxy-ca.key ├── front-proxy-client.crt ├── front-proxy-client.key ├── sa.key └── sa.pub

The components themselves are described and run on the masters as static pods from the /etc/kubernetes/manifests/ directory.

The most important thing is to know how to make a functional cluster of all this; let’s imagine that the Kubernetes components mentioned above somehow interact with each other. The basic diagram looks like this:

To communicate, they need TLS certificates, which, in principle, can be taken to a separate level of abstraction and completely rely on their distribution tool, be it kubeadm, kubespray or something else, pentesting experts mention. In this example, we’ll look at kubeadm because it’s the most common Kubernetes deployment tool and is often used as part of other solutions.

Let’s say we already have a cluster in place. Let’s start with the fun part:

rm -rf /etc/kubernetes/ 

In masters, this directory contains:

• A set of certificates and CAs for etcd (c/etc/kubernetes/pki/etcd)
• A set of certificates and CAs for Kubernetes (c/etc/kubernetes/pki)
• Kubeconfig for cluster-admin, kube-controller-manager, kube-Scheduler and kubelet (each also has a base64-coded CA certificate for our /etc/kubernetes/*.conf cluster)
• A set of static manifests for etcd, kube-apiserver, kube-Scheduler and kube-controller-manager (c/etc/kubernetes/manifests)

Suppose we lost everything at once.    

Fixing the control plane

To avoid confusion, the researchers recommend making sure that all of our control plane pods are also stopped:

crictl rm `crictl ps -aq`

It should be remembered that kubeadm, by default, does not overwrite existing certificates and kubeconfigs, so to reissue them; you must first manually delete them.

Let’s start by restoring etcd, because if we had a quorum (3 or more master nodes), the etcd cluster would not start without most of them present.

kubeadm init phase certs etcd-ca

The above command will generate a new CA for our etcd cluster. Since all other certificates must be signed by it, we will copy it along with the private key to the rest of the master nodes:

/etc/kubernetes/pki/etcd/ca.{key,crt}

Now, let’s regenerate the rest of the etcd certificates and static manifests on all nodes in the control plane:

kubeadm init phase certs etcd-healthcheck-client
kubeadm init phase certs etcd-peer
kubeadm init phase certs etcd-server
kubeadm init phase etcd local 

At this stage, we should already have an etcd cluster working, mention the experts in pentesting:

# crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
ac82b4ed5d83a       0369cf4303ffd       2 seconds ago       Running             etcd                0                   bc8b4d568751b

Now let’s do the same, but for Kubernetes, on one of the master nodes, run:

kubeadm init phase certs all
kubeadm init phase kubeconfig all
kubeadm init phase control-plane all
cp -f /etc/kubernetes/admin.conf ~/.kube/config 

The above commands will generate all SSL certificates for our Kubernetes cluster, as well as statistics for the manifests and kubeconfigs for Kubernetes services. If you are using kubeadm to join kubeletes, you will also need to update the cluster information configuration in the kube-public namespace that still contains the hash of your old CA.

kubeadm init phase bootstrap-token  

Since all certificates from other instances must also be signed by a CA, copy them to the other nodes in the control plane and repeat the above commands on each of them, pentesting experts recommend.

/etc/kubernetes/pki/{ca,front-proxy-ca}.{key,crt}
/etc/kubernetes/pki/sa.{key,pub}  

As an alternative to manual copying certificates, you can now use the Kubernetes interface, for example, the following command:

kubeadm init phase upload-certs --upload-certs

Certificates will then be encrypted and uploaded to Kubernetes for 2 hours, so you can rejoin the masters as follows:

kubeadm join phase control-plane-prepare all kubernetes-apiserver:6443 --control-plane --token cs0etm.ua7fbmwuf1jz946l     --discovery-token-ca-cert-hash sha256:555f6ececd4721fed0269d27a5c7f1c6d7ef4614157a18e56ed9a1fd031a3ab8 --certificate-key 385655ee0ab98d2441ba8038b4e8d03184df1806733eac131511891d1096be73
kubeadm join phase control-plane-join all

It’s worth noting that the Kubernetes API has another configuration that stores the CA certificate for the front proxy client, used to authenticate apiserver requests to webhooks and other aggregation layer services. Fortunately, kube-apiserver updates it automatically. However, you may want to manually clean it from the old certificates:

kubectl get cm -n kube-system extension-apiserver-authentication -o yaml

In any case, at this stage we already have a fully functioning control plane.

Fixing the workers

This command will list all the nodes in the cluster, although they will now all be in the NotReady state:

kubectl get node

This happens because they are still using the old certificates and waiting for server requests signed by the old CA. To work around this issue, we’ll use kubeadm and create a logging node in the cluster.

When both teachers have access to the CA and can connect locally:

systemctl stop kubelet
rm -rf /var/lib/kubelet/pki/ /etc/kubernetes/kubelet.conf
kubeadm init phase kubeconfig kubelet
kubeadm init phase kubelet-start 

Then, for the union workers, we will generate a new token:

kubeadm token create --print-join-command

And in each of them we execute:

systemctl stop kubelet
rm -rf /var/lib/kubelet/pki/ /etc/kubernetes/pki/ /etc/kubernetes/kubelet.conf 
kubeadm join phase kubelet-start kubernetes-apiserver:6443  --token cs0etm.ua7fbmwuf1jz946l     --discovery-token-ca-cert-hash sha256:555f6ececd4721fed0269d27a5c7f1c6d7ef4614157a18e56ed9a1fd031a3ab8

Attention: /etc/kubernetes/pki/ it is not necessary to delete the directory in the masters, since it contains all the necessary certificates, say the experts in pentesting.

The previous procedure will reconnect all your cublets to the cluster without affecting the containers that are already running on them. However, if you have many nodes in the cluster and you are doing this at the same time, you might have a situation where Controller-Manager starts recreating containers with NotReady nodes and tries to start them on the active nodes in the cluster.

To avoid this, we can temporarily stop the controller-administrator, on the masters:

rm /etc/kubernetes/manifests/kube-controller-manager.yaml
crictl rmp `crictl ps --name kube-controller-manager -q` 

The last command is only necessary to make sure that the controller-administrator is not actually running. Once all the nodes in the cluster are connected, we can generate a static manifest for the back of the controller-administrator.

To do this, in all teachers they will have to execute:

kubeadm init phase control-plane controller-manager

Note that you need to do this at the stage where you have already generated the join token; otherwise, the connection operation will crash when trying to read the cluster-info token.

If the kubelet is configured to receive a certificate signed by your CA (optional TLSBootstrap server: true), you will also need to reconfirm the csr of your kubelets:

kubectl get csr
kubectl certificate approve <csr> 

Fixing ServiceAccounts

There is one more thing. Since we lost /etc/kubernetes/pki/sa.key, pentesting experts mention that this is the same key that signed the jwt tokens for all our ServiceAccounts, so we need to recreate the tokens for each of them.

This can be done by simply removing the token field from all secrets as kubernetes.io/service-account-token:

kubectl get secret --all-namespaces | awk '/kubernetes.io\/service-account-token/ { print "kubectl patch secret -n " $1 " " $2 " -p {\\\"data\\\":{\\\"token\\\":null}}"}' | sh –x 

After that, kube-controller-manager will automatically generate new tokens signed with a new key, pentesting experts mention. Unfortunately, not all microservices can re-read the token on the fly, and you’ll most likely need to manually restart the containers where they’re used:

kubectl get pod --field-selector 'spec.serviceAccountName!=default' --no-headers --all-namespaces | awk '{print "kubectl delete pod -n " $1 " " $2 " --wait=false --grace-period=0"}' 

For example, this command will generate a list of commands to delete all modules using a non-standard service account. I recommend starting with the kube system namespace. You have kube-proxy and the CNI plugin installed, which are vital for configuring your microservices to communicate.

To learn more about information security risks, malware variants, vulnerabilities and information technologies, feel free to access the International Institute of Cyber Security (IICS) websites.

The post How to perform Kubernetes pentesting and secure it? appeared first on Information Security Newspaper | Hacking News.

The report, prepared by Palo Alto Networks, notes that Siloscape was detected in early March, and was named for it because the malware’s primary goal is to escape Windows containers through a server silo. The malware uses the Tor proxy and an onion domain to establish a connection to a C&C server, to which the data and commands intercepted by the hackers are sent.

Siloscape operates under the CloudMalware.exe label, targeting Windows containers to launch attacks using known vulnerabilities that have not received updates, gaining initial access to servers, websites or databases.

Once access is gained, hackers will attempt to perform a remote code execution attack on a container’s underlying node using various hacking techniques, including CExecScv spoofing.exe: “The malware mimics CExecScv.exe privileges to invoke a newly created symlink to exit the container, linking its X drive into the container as the host’s C drive.” , the report states.

In a successful attack the threat actors will try to create malicious containers, stealing data from applications executed in malicious clusters and even loading tools for cryptocurrency mining, generating huge profits for the operators of the attack, who can go completely unnoticed.

Malware developers focus especially on obfuscation, even to the point where functions and module names are only deprecated at runtime, complicating reverse engineering. Palo Alto researchers were able to gain access to C2 and identify at least 23 active victims, as well as 313 total victims, likely insured in campaigns over the past year.

Microsoft recommends that containers be implemented if this process is used as a form of security boundary rather than relying on standard Windows containers. On the other hand, the researchers added that Kubernetes clusters should be configured correctly and should not allow node privileges alone to be sufficient to create new deployments.

To learn more about information security risks, malware variants, vulnerabilities and information technologies, feel free to access the International Institute of Cyber Security (IICS) websites.

The post Siloscape: the new malware to compromise Windows containers and Kubernetes clusters appeared first on Information Security Newspaper | Hacking News.

Kubernetes was originally developed by Google and now under the control of the Cloud Native Computing Foundation, and is an open source system for process automation such as deploying, scaling, and managing workloads, services, and containerized applications on host groups.

The vulnerability, considered of medium severity and identified as CVE-2020-8554, was reported by Etienne Champetier of the security firm Anevia and can be exploited remotely by unidentified threat actors with basic permissions without the need for user interaction.

Because this is a glitch in Kubernetes design, all currently used versions are affected: “If malicious hackers can create or edit services and pods, they may be able to intercept traffic from other pods in the container,” the report says. Fortunately it is not all bad news, as the report also notes that the vulnerability should affect only a small number of Kubernetes implementations, this because external IP services are not widely used.

The Kubernetes report includes a workaround to mitigate the risk of exploitation: Users should restrict access to vulnerable features by using an admission webhook container to limit the use of external IPs. The source code for implementing this workaround is available on kubernetes’ official platforms. It is also possible to restrict external IPs using the Open Policy Agent Gatekeeper policy controller.

To detect attacks that attempt to exploit this vulnerability, you must manually audit the use of external IPs within multi-tenant clusters using vulnerable features.

The post Critical vulnerability affects all versions of Kubernetes appeared first on Information Security Newspaper | Hacking News.