How to Design the Most Secure Kubernetes Architecture

Kubernetes environments come in many shapes, forms, and sizes. Some are inherently more secure than others.

In other words, Kubernetes architecture (the architectural strategy that you select when designing your Kubernetes environment) can have important implications for overall security. A multi-cluster environment may be more secure in some respects than one that runs everything in a single cluster (although multiple clusters also increase complexity, which is a con from a security standpoint). Likewise, there are security implications for third-party tools, like monitoring agents and service meshes, that you may choose to build into your Kubernetes architecture.

So, which types of architectural strategies lead to the strongest security posture for Kubernetes? This article offers an answer to that question by talking through the high-level design choices that admins typically have to make when planning a Kubernetes environment, as well as by discussing what each element means from a security perspective.

Managed Kubernetes Service vs. Self-Managed Kubernetes

Probably the first question that most teams ask themselves today when planning a Kubernetes installation is whether to use a managed Kubernetes service – such as Amazon AKS, Azure Kubernetes Service, or another Kubernetes platform that runs in a public cloud – or to deploy and manage Kubernetes themselves on infrastructure that they control.

A managed Kubernetes service is almost always easier to set up and maintain because the Kubernetes provider handles at least some of the provisioning and maintenance tasks required to keep clusters running. (Note that the extent of the management features that vendors deliver can vary significantly from one “managed” Kubernetes service to another, but that’s tangential to our discussion here.)

Managed Kubernetes may also be more secure in two key respects:

• The host infrastructure is professionally managed and monitored for security threats. With managed Kubernetes, in other words, you don’t need to worry as much about security issues at the OS level on your Kubernetes nodes.
• Your managed Kubernetes provider may provide pre-configured settings or tooling that (in most cases, at least) adheres to best practices from a security perspective.

On the other hand, because managed Kubernetes architecture involves the use of tools and (in most cases) infrastructure that is owned by a vendor, it has the inherent security drawback of restricting the amount of control and privacy that users can achieve. You can’t use a managed Kubernetes service if you want to “air gap” your clusters from the Internet, for instance. And although certain types of managed Kubernetes platforms (like Platform9) are compatible with on-prem infrastructure, most require the use of infrastructure hosted in the public cloud, which is more likely to be exposed to security threats.

The bottom line: if you’re new to Kubernetes and not well versed in Kubernetes security principles, a managed Kubernetes service will probably result in a more secure environment than one that you set up yourself. But if you require an advanced security architecture that includes features like air-gapping, you’ll have to set up and manage your environment on your own.

Single vs. Multi-Cluster Kubernetes Architecture

Another key architectural decision you’ll have to make as you plan a Kubernetes installation is whether to set up just one cluster or multiple clusters.

Traditionally, almost all teams used just a single cluster. But multi-cluster environments are becoming more popular, according to the CNCF, in part because they make it possible to achieve a high degree of isolation between workloads. You can deploy different workloads in different clusters, which significantly reduces the risk that a security issue in one workload will “spill over” to affect others. Learn more about securing Kubernetes clusters.

That said, teams must weigh this security benefit against the major downside of multi-cluster Kubernetes: added complexity. More clusters mean more audit logs to track, more RBAC policies to create, apply, and monitor, more networks to configure and isolate, and so on.

If you have strong automations in place for managing and monitoring your clusters, this added complexity may not pose much of an issue. In that case, go with a multi-cluster architecture. However, if you’re going to struggle to keep track of everything with that type of setup, stick to a single cluster, where it will be easier for you to handle tasks like detecting security events and ensuring that RBAC policies remain up to date.

Single vs. Multiple Namespaces

Even if you don’t run multiple clusters, you can achieve fairly strong workload isolation by creating multiple namespaces. Namespaces are essentially virtual clusters that are hosted within the same physical cluster.

Typically, you’ll want to create a different namespace for each application you are running in Kubernetes. You should also create different namespaces for dev/testing and production.

Remember, however, that the isolation provided by namespaces is limited. Any permissions that you assign via ClusterRoles will be applied to all namespaces. In this sense, having multiple namespaces is only beneficial if you create RBAC policies that allow you to isolate users and accounts within each namespace. Do this by assigning permissions using Roles instead of ClusterRoles wherever possible.

Service Meshes

Service meshes, which manage service discovery and connectivity for resources running inside a Kubernetes cluster, are a critical tool for any Kubernetes environment that includes more than just a handful of services.

Overall, service meshes are a plus on the Kubernetes security front. In addition to helping to manage connectivity, most service meshes also provide monitoring and alerting features that can help you to detect security threats. Because Kubernetes itself doesn’t record network-related security incidents (Kubernetes audit events don’t cover the network, just the API), service meshes close a critical security visibility gap.

The downside of service meshes is that they increase the overall attack surface and complexity of Kubernetes. An attacker who manages to breach your service mesh can use it as a beachhead to breach your entire cluster (or clusters).

That said, as long as you deploy service meshes securely, you can mitigate the associated risks. If you deploy your service mesh software via sidecar containers that run alongside your application pods, be sure to isolate the sidecars and lock down their access using RBAC and network policies.

External Monitoring and Security Software

In addition to service meshes, it’s common for teams to deploy third-party performance and security monitoring software to help them manage their clusters.

These tools also increase your attack surface. But as long as you deploy them securely, the benefits far outweigh the risks.

You can optimize the security of external tools by following practices such as these:

• As noted above, enforce least privilege for agents that run as sidecar containers.
• When possible, use webhooks to stream data from Kubernetes to external tools. That way, you avoid having to run the tools directly within your cluster, which poses a greater security risk than running them in isolation.
• Be sure that you secure any data that external tools generate or store persistently. Information like that found in Kubernetes audit logs could potentially help attackers gain access to your cluster, so you need to be careful about how you manage that data.

Designing the Most Secure Kubernetes Architecture

Ultimately, there is no one-size-fits-all approach to designing a secure Kubernetes architecture. Architectural decisions will typically reflect the scale of your environment, the types of workloads you are running in it, and how many different users or teams are sharing the environment. Smaller environments are likely to be simpler in design and require fewer external tools, which translates to a stronger security posture by default.

Nonetheless, complex environments – those that involve multiple clusters and a variety of third-party integrations – can be just as secure as architecturally simple configurations.

Thus, the key to designing a secure Kubernetes environment is not to avoid a certain type of configuration or ensure that you do or don’t include a given tool. It’s simply ensuring that, every time you add to the architectural complexity of Kubernetes, you identify and address the security implications of your decision.