Security is among the primary considerations in any application development. Security standards and practices should be integrated into all aspects of software development, from infrastructure and databases to maintenance and management.

Kubernetes supports plenty of security configurations that can be used to secure clusters and underlying pods. In this article, we will see what the Kubernetes security best practices are and how to integrate them into DevOps.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

Cloud-native security

Let’s begin with a look at the cloud-native security architecture when identifying the base of Kubernetes security. The 4Cs in Cloud-native security refers to the four security layers:

• Cloud
• Clusters
• Containers
• Code

Each layer is built on top of the previous layer providing maximum security to a Kubernetes cluster:

• Cloud. This layer represents the underlying physical infrastructure. You may deploy your Kubernetes application either on-premise hardware or on a third-party cloud provider, but you need to follow the relevant security best practices to secure all the infrastructure.
• Cluster. A Kubernetes cluster should be securely configured with cluster services like Kubernetes API and all the applications within the cluster. Securing applications is even more important when dealing with microservices since a minor vulnerability in a single service can affect the whole cluster.
• Containers. Build your containers using the smallest possible image by enforcing strict user privileges and removing all unnecessary libraries and features. Additionally, all containers must be scanned regularly to identify any vulnerabilities.
• Code. The application code can be the largest attack surface due to production bugs and failure to follow security practices during the development, deployment, and management. Simple actions like removing unused ports, encrypting data transfers, regular testing, and scanning can mitigate many security issues in the application code.

Kubernetes security best practices

Let’s have a look at some best practices and recommendations for creating a secure Kubernetes environment.

Cloud security best practices (Infrastructure)

• Control panel access. Only a selected set of users from an authorized location (whitelisted IP) should be allowed to interact with the Kubernetes control panel. All public access should be disabled.
• Node access. Access to Kubernetes nodes should be controlled by only accepting connections from Kubernetes services like NodePort or LoadBalancer restricted to specific ports and protocols. Public access to nodes should also be discouraged if possible.
• Cloud Provider API. When interacting with cloud provider APIs through Kubernetes, only assign roles and policies that limit the privileges to the core functions required by Kubernetes, such as provisioning new nodes.
• Kubernetes etcd. The access to datastore should be limited to the Kubernetes control panel and be configured with TLS. Besides, the datastore should be encrypted for additional security.

Cluster security best practices

• Role-based access control (RBAC). Access for all services should be limited to authorized users and roles with specific IP addresses.
• Pod security policies. Enforce standardized pod security policies across the cluster environment.
• Network security policies. Enforce network policies for all network-related resources to ensure data security.
• Data encryption. By default, encrypt all possible data storage from etcd to volumes where sensitive data are stored.

Container security best practices

• Vulnerability scanning. Scan and identify any security issues related to the container images and address them before building the container.
• Image signing. Enforce image signing so that the cluster can only deploy signed trusted images.
• User control. All privileged users should be removed, and only users with limited privileges should be included in the container (user vs root).
• Dependency management. Any unnecessary features or software dependencies should be removed from the base image itself.

Code security best practices

• Encrypted communication. Integrate encryption to all the network traffic to minimize any data leaks.
• Application access. Expose a limited number of application endpoints and ports and enforce authentication to reduce the attack surface.
• Dependency management. As with the container base image, strip out all unnecessary third-party frameworks and libraries to reduce the risk of third-party vulnerabilities affecting the application.
• Development methodology. Security best practices should be integrated into the SDLC as part of the DevSecOps process with regular vulnerability scanning and code reviews. This would help developers to identify and fix any security issues before the product is released for production.

Integrating Kubernetes security to a DevOps pipeline

There are two major considerations that any organization will face when implementing Kubernetes security:

• Self-configuration. Kubernetes consists of several features to enforce security best practices. Importantly, none of them are configured by default, so the cluster administrator must manually configure them in the cluster.
• Lack of knowledge. It requires an in-depth understanding of the Kubernetes environment and the application to properly configure all the security features, policies, and best practices.

The best option to overcome the above issues would be integrating Kubernetes security best practices in a DevOps pipeline. It enables developers to both:

• Get a broader view of the entire application architecture
• Gradually incorporate security practices into a Kubernetes environment with incremental changes

We can boil down the Kubernetes DevOps pipeline into three distinct sections to better implement the security practices.

Let’s have a look at each.

Application & container security at the image build stage

There are a number of application security best practices, such as:

• Data encryptions
• User privileges
• Token management
• Threat assessments
• Code reviews

These should be organically integrated into the development, from the initial stage of application development. Since the application is the core component of the container, an insecure application can expose the whole production Kubernetes cluster to attacks—even if all the other aspects are secure.

When building the container image, use an updated base image without any security vulnerabilities and strip out any unnecessary components or features of it.

For example, suppose we can use a smaller base image such as alpine Linux instead of Ubuntu without affecting the functionality or performance of the application. In that case, we should always go with the smaller base image. It both:

• Reduces the attack surface
• Creates a much more lightweight container

Then we should carefully go through all application dependencies to identify vulnerabilities using a tool like synk and address them.

Another concern is the ports exposed via the container. We should only expose the necessary ports and require authentication for all requests. In the end, we can build a container image that is ready to be deployed in Kubernetes.

Another best practice is to conduct a penetration test of the application in a staging environment and sign the image before pushing it to the production cluster.

Security at Kubernetes infrastructure deployments

You must have in-depth knowledge of Kubernetes to properly configure the cluster when deploying a Kubernetes cluster. As a rule of thumb, we should try to avoid using any default configurations. We’ll also need to:

• Scope user permissions in the cluster
• Set up proper network connectivity with ingress and load balancers to control network traffic within the cluster
• Create network segmentation to provide maximum physical security

Additionally, proper networking will eliminate the need for nodes to be exposed directly to the internet.

Having configured pod policies configured, we can ensure that only pods which comply with the required policy will be created, and no rogue pods will be created. Network policies help you to control:

• The traffic flow to pods
• How a pod can communicate with other network objects, pods, etc.

Another important consideration is cloud security. Even though we configure a secure Kubernetes cluster, our applications can be exposed to attacks if the underlying infrastructure or cloud provider is insecure. Therefore, administrators should secure the underlying infrastructure with proper access controls and authentication mechanisms before deploying infrastructure so that only authorized users can modify infrastructure.

Finally, the administrators need to enforce images registries so that only signed images from authorized container registries can be deployed in the cluster.

It is also essential to configure continuous monitoring and vulnerability scanning for both containers and clusters. It will act as an early warning system to identify issues and fix them. A secure container image created in the build stage combined with a secure cluster environment provides a solid foundation for Kubernetes security.

Managing security at runtime

This final section is about managing security at runtime.

Users need to maintain security practices even after the deployment, or else the application and the cluster will become vulnerable over time. Furthermore, you’ll need to keep track of all the security layers from the code to the cloud layer at runtime.

The production bugs of an application that are discovered at the production can also introduce vulnerabilities. So, a proper process should be in place to quickly deploy a fix for the bug. Besides, clusters can quickly become complex structures with multiple applications and services running in them. Thus, to maintain the security of these complex clusters, you’ll need to have:

• Strict advanced network control policies
• Multi-tenant support
• Properly configured firewalls
• Encryption standards

In case of a vulnerability, the affected pod(s) should be quickly isolated and removed while new replacement pods with patched vulnerabilities are created

Lastly, you’ll need to rely on a proper health monitoring and log management solution to identify all these issues. All in all, these security practices help users to ensure Kubernetes security at runtime.

Kubernetes security has no stopping point

Kubernetes security is a vast subject. In short, the process of securing an application starts from inception and continues until the EOL of the application.

Securing Kubernetes goes hand in hand with securing the application since application vulnerabilities can affect the cluster. As I’ve outlined here, the best approach is to include security practices in the DevOps process, considering all aspects of the Kubernetes cluster and application to maintain the security.

Related reading

• BMC DevOps Blog
• Kubernetes Best Practices for Enhanced Cluster Efficiency
• Kubernetes Multi-Clusters: How & Why To Use Them
• Kubernetes vs Docker Swarm: Comparing Container Orchestration Tools
• SecOps vs DevSecOps: What’s The Difference?
• The State of Kubernetes: A Report Roundup

There are different types of applications, from single large applications to microservices-based applications that cater to different needs. When it comes to the states of those applications, there are two states:

• Stateless applications can be run independently in isolation without any knowledge of past transactions.
• Stateful applications have full knowledge of the past information (state).

Most applications we use are stateful applications, and their state data may consist of user preferences, recent activity, database transactions, credentials, settings, etc.

Kubernetes provides StatefulSets when creating a stateful application in a Kubernetes cluster. Managing states within a containerized environment has become even more significant with the popularity of deploying database clusters in Kubernetes.

In this article, we will focus on how to deploy a PostgreSQL database on a Kubernetes cluster using StatefulSets.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What is Kubernetes StatefulSets?

StatefulSet is a Kubernetes workload API object that can be used to manage stateful applications.

In a normal deployment, the user does not care how the pods are scheduled as long as it doesn’t have a negative impact on the deployed application. However, there is the need to properly identify pods to preserve the state in stateful applications with persistent storage.

StatefulSet provides this functionality by creating pods with a persistent identifier that will pertain to its value across rescheduling. This way, a pod will get correctly mapped to the storage volumes even if it is recreated, and the application state will be preserved.

StatefulSets use cases

There are several use cases for StatefulSets.

Ordered deploying & scaling

When an application relies on multiple containers, the ordered approach to scaling ensures that dependent containers are created in an orderly manner at deployments and scaling scenarios.

Ordered automated rolling updates

Updating applications or microservices that are dependent should also be updated in an orderly manner. Additionally, an update should not affect the functionality.

Therefore, users can decide the order in which the applications or microservices should be updated by using a StatefulSet.

Mapping persistent storage

When considering databases, persistent storage is the most critical part as applications need to store data. With a StatefulSet, users can:

1. Define which pods correspond to each persistent storage
2. Create resilient application deployments

Using unique network identifiers to create persistent network connectivity

With unique identifiers, network users can manage and route traffic for specific pods without worrying about IP changes at rescheduling. This provides greater control over the network communications between pods by providing the ability to configure persistent routing, policies, and security configs for desired pods.

Even with these benefits, StatefulSets do not provide a solution for all requirements. For instance, StatefulSets are not interchangeable with deployments or ReplicaSets—these are instead geared to stateless configurations.

Drawbacks of StatefulSets

StatefulSets also come with a set of limitations that users should be aware of before deploying the application.

• The storage for a StatefulSet must be provisioned either by a PersistentVolume Provisioner based on the storage class or pre-provisioned.
• Scaling or deleting pods will not affect the underlying persistent storage in order to ensure data safety. The provisioned volumes will remain within Kubernetes.
• The user needs to create a headless service manually to ensure network identity in StatefulSets.
• SatefulSets does not guarantee the termination of current pods when the StatefulSet is deleted. So, best practice is that the user implement an SOP to scale the StatefulSet to zero pods before deleting.
• Rolling updates with the default pod management policy may cause issues when deploying if a pod is broken (due to an application config error, bad binary, etc.). In such instances, users need to:
  • Manually revert to a previous deployment template.
  • Delete the broken Pods before attempting to rerun the new updates.

Setting up a StatefulSet in a Kubernetes cluster

Now that we have a basic understanding of a StatefulSet, let’s look at a sample StatefulSet deployment.

StatefulSets are ideal for database deployments. In this example, we will create a PostgreSQL deployment as a StatefulSet with a persistent storage volume.

postgresql.yaml

# PostgreSQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-db
spec:
  serviceName: postgresql-db-service
  selector:
    matchLabels:
      app: postgresql-db
  replicas: 2
  template:
    metadata:
      labels:
        app: postgresql-db
    spec:
      containers:
        - name: postgresql-db
          image: postgres:latest
          volumeMounts:
            - name: postgresql-db-disk
              mountPath: /data
          env:
            - name: POSTGRES_PASSWORD
              value: testpassword
            - name: PGDATA
              value: /data/pgdata
  # Volume Claim
  volumeClaimTemplates:
    - metadata:
        name: postgresql-db-disk
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 25Gi

In the above YAML file, we have defined a simple StatefulSet to deploy a PostgreSQL database. We are creating a StatefulSet called postgresql-db with two pods (replicas: 2).

Additionally, we are creating a Persistent Volume using the volumeClaimTemplate and using it in the StatefulSet to store the PostgreSQL data. The default Persistent Volume provisioner will provision the volume, and we can deploy this by running the following command.

kubectl apply -f postgresql.yaml

Result:

Now we have successfully created a PostgreSQL StatefulSet yet need a service to expose it outside of the Kubernetes cluster. That can be done by creating a service that points to the StatefulSet.

postgresql-service.yaml

# PostgreSQL StatefulSet Service
apiVersion: v1
kind: Service
metadata:
  name: postgres-db-lb
spec:
  selector:
    app: postgresql-db
  type: LoadBalancer
  ports:
    - port: 5432
      targetPort: 5432

This will create a load balancer, and the service will expose our PostgreSQL database using the selector “app: postgresql-db.” You can create the service by running the below command.

kubectl apply -f postgresql-service.yaml

Result:

Now let’s see if both the StatefulSet and the service are successfully created in Kubernetes by running the following command.

kubectl get all

Result:

The result indicates that two pods are created and running with a load balancer service exposing the StatefulSet via IP 10.111.253.4 using port 5432.

Testing the connectivity

Having deployed PostgreSQL, we need to verify that we can access it without any issues. We will use the pgAdmin4 client to initialize the connection:

1. Download the pgAdmin client in your environment.
2. Connect and try to initialize a connection.

With the above deployment, we will use the external IP of the postgres-db-lb service (10.111.253.4) with port 5432. Since we only defined a password in our environment variables for the PostgreSQL StatefulSet, the configuration will have the default username “postgres” with the password we defined.

If all the details are correct, the connection will be initiated when you click on “Save,” and the user will be able to see the connected database.

Using ConfigMap in StatefulSet

In our earlier example, we defined the environment variables with the StatefulSet YAML.

However, the best practice would be to separate the environment variables using ConfigMaps and call the ConfigMap from the StatefulSet deployment. This makes it easier to manage and maintain each component of the deployment.

So, let’s create a ConfigMap and modify the StatefulSet YAML as shown below.

postgresql-configmap.yaml

# PostgreSQL StatefulSet ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-db-config
  labels:
    app: postgresql-db
data:
  POSTGRES_DB: testdb
  POSTGRES_USER: testdbuser
  POSTGRES_PASSWORD: testdbuserpassword
  PGDATA: /data/pgdata

In the above ConfigMap, we have extended our environment variable to specify a PostgreSQL database, user, password, and data store. Now let’s create the configMap and view the configurations using this command:

kubectl apply -f .postgresql-configmap.yaml

Result:

We can get the information of the created ConfigMap using the describe function:

kubectl describe configmap postgres-db-config

Result:

The next step is to modify the StatefulSet to call the data from the ConfigMap. That can be done by using the envFrom field to point to the above ConfigMap. This will also enable us to create a StatefulSet using the data in the ConfigMap.

postgresql.yaml

# PostgreSQL StatefulSet - ConfigMap
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-db
spec:
  serviceName: postgresql-db-service
  selector:
    matchLabels:
      app: postgresql-db
  replicas: 2
  template:
    metadata:
      labels:
        app: postgresql-db
    spec:
      containers:
        - name: postgresql-db
          image: postgres:latest
          volumeMounts:
            - name: postgresql-db-disk
              mountPath: /data
          # Config from ConfigMap
          envFrom:
          - configMapRef:
              name: postgres-db-config
  # Volume Claim
  volumeClaimTemplates:
    - metadata:
        name: postgresql-db-disk
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 25Gi

With our Persistent Volumes, deletions in the underlying database will be preserved even in case of Pod errors. As a StatefulSet, the state of Pods will also be preserved, and they will get assigned to the desired volumes correctly when recreated.

We can identify the pods using the following command:

kubectl get pvc

Result:

Managing state is crucial to app functionality

Kubernetes StatefulSets allows users to easily create and manage stateful applications or services within a Kubernetes cluster. However, these StatefulSets configurations involve some complexity—so you must carefully plan your deployments before them carrying out.

Additionally, StatefulSets are the ideal solution for dealing with database applications, payment services, etc., where managing state is a crucial part of the application functionality.

Related reading

• BMC DevOps Blog
• Kubernetes Multi-Clusters: How & Why To Use Them
• Kubernetes Best Practices for Enhanced Cluster Efficiency
• 3 Kubernetes Patterns for Cloud Native Applications
• The State of Containers Today
• Containerized Machine Learning: An Intro to ML in Containers

Containerized deployments offer more scalability and availability improvements over traditional deployments. Even with these improvements, complex applications can quickly overwhelm containerized environments without proper management. Kubernetes helps organizations to:

• Effectively orchestrate containerized environments
• Efficiently manage the underlying resources and user demands

However, there are situations where a single Kubernetes cluster is unable to handle the application load or properly distribute the application to end-users. In such instances, multi-cluster Kubernetes solutions are ideal for distributing the work across multiple clusters.

In this article, we’ll take a look at Kubernetes multi-cluster implementations.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What is a Kubernetes multi-cluster?

Kubernetes multi-cluster is an environment with multiple Kubernetes clusters. They can be configured in several ways:

• Within a single physical host
• With different multiple hosts in the same data center
• In different regions within a single cloud provider
• Utilizing multiple cloud providers and distributed across multiple regions to provide a truly global multi-cluster environment

A multi-cluster environment is not simply running multiple Kubernetes clusters.

Instead, it should consist of proper application deployment strategies to distribute and maintain the application across multiple clusters with appropriate tools and practices baked into the DevOps process.

(See how containers & K8s work together.)

Why use Kubernetes multi-cluster?

Configuring and managing multi-cluster Kubernetes deployments is not an easy task. So, why would an organization allocate valuable resources for this endeavor?

The reason: user experience.

When it comes to maintaining large-scale applications with thousands of users, multi-clusters are the core component to providing an excellent user experience. Let’s see all the benefits a multi-cluster environment offers.

Increased scalability & availability

With multi-clusters, users can distribute the application across different regions, exponentially increasing the availability. With that, end users will have a much better experience interacting with the applications with:

• Less latency
• Faster performance

The same can be applied for scaling instead of a single cluster trying to handle all the scaling requirements. Multi-cluster deployments are distributed across multiple clusters and scales according to a load of a specific cluster. It also drastically reduces resource consumption of a single cluster and leads to less load on backend services like databases.

Application isolation

Multi-cluster deployments allow organizations to achieve true application isolation with separate Kubernetes clusters. They can be either:

• Different clusters for staging and production (simple separation)
• Different clusters facilitating distinct applications

With different clusters, any cluster failure or configuration change affects only that specific cluster. This isolation helps users to:

• Easily diagnose issues
• Experiment with new features
• Carry out optimizations without causing disruptions to other clusters

Regulatory compliance

With applications serving users across the globe, one major considerations is adhering to regulations and policies of different regions and countries—policies like GDPR and the California Consumer Privacy Act (CCPA). This is particularly prevalent when user data is involved, as most regions require the user data to reside within geographical limits.

Importantly, applications hosted in a single cluster would not be able to comply with all these compliance requirements. Multi-cluster deployments, on the other hand, enable organizations to meet these requirements by locating the Kubernetes cluster in specific geographical regions.

Applications that cater to specific geographic regions with isolated Kubernetes clusters can:

• Limit the scope of regulatory requirements
• Enables organizations to target specific clusters to meet different requirements and gain regulatory compliance relatively easily

Cluster management

Setting up and managing multi-cluster deployments can be a complex and time-consuming task. Fortunately, a properly configured multi-cluster environment can reduce the overall management complexity.

With proper cluster management, you can have better visibility with standardized processes and proper DevOps practices while managing all the infrastructure and applications from a centralized location. This can be a significant factor in managing operations expenditure—users can effectively allocate resources.

Additionally, in a multi-cluster environment, the production issues are isolated to that specific cluster. That means developers only need to investigate a limited scope to identify and fix issues without affecting other clusters. Crucially, a complete cluster failure would not cause any downtime in a multi-cluster environment. The other clusters step in to ensure availability.

Users are not vendor-locked into a true multi-cloud deployment. When new features or cost savings are available with new cloud providers, users can seamlessly transfer applications, with minimal modifications, between different providers or Kubernetes clusters.

Distributed applications

With increasing edge computing requirements and the popularity of Internet of Things (IoT) services, distributed computing is becoming the preferred solution.

A multi-cluster Kubernetes environment is the ideal platform for that.

Developers can simply create containerized applications that can be deployed in Kubernetes while providing a powerful orchestration engine distributed across multiple regions. This eliminates the need to cater applications to different environments and infrastructure since these containers can act as the edge nodes to facilitate a global IoT network.

Kubernetes multi-cluster architecture

When it comes to multi-cluster architecture, there are two considerations:

• The Kubernetes multi-cluster configuration
• The application architecture

Kubernetes multi-cluster architecture

There are primarily two methods to handle a multi-cluster Kubernetes environment:

• The Kubernetes-centric federated method
• The network-centric method

Different vendors offer variations and improvements upon these methods.

A Kubernetes-centric method uses tools like kubefed to manage multiple Kubernetes clusters from a centralized location (federated clusters). Other tools like Shipper and Admiralty can facilitate:

• Rollout strategies
• Multi-cluster orchestration
• Workload distribution

The network-centric method aims to provide a robust network to facilitate communication between different clusters. The clusters act as separate entities, yet applications can communicate with each other using a network. The network approach is based on mesh networking principles and has adapted service mesh concepts to the infrastructure. An example of this approach is:

• Linkerd service mirroring

Most organizations prefer the network-centric method due to the lack of maturity of Kubernetes-centric multi-cluster configurations. Even with the increased complexity of managing a mesh network, this approach effectively handles multi-cluster environments with automated pipelines and standardized configurations with GitOps, reducing management complexity.

Multi-cluster application architecture

There are two application architectures to consider when developing multi-cluster applications—replicated apps and service-based apps.

Replicated application architecture

This approach replicates the complete application across multiple Kubernetes clusters, and it is the simplest way to deploy a multi-cluster environment. With the complete application distributed across multiple clusters in different regions, end users can be directed to the nearest cluster, helping you to both:

• Manage the load effectively
• Provide the best user experience

Coupling this replicated application architecture with proper networking infrastructure and load balancers can handle cluster failure with ease. When a cluster faces an issue, users can be redirected to a different cluster without any effect on the end-user or application functionality.

Service-based application architecture

Another development architecture is to divide the application into multiple components or services and distribute them across clusters. It provides the best isolation and is the best option for obtaining regulatory compliance. However, all this comes with increased application complexity.

With different components running on different clusters, the communication layer between each component should be designed to handle user and application traffic. When configuring a fault-tolerant architecture, organizations will need to:

1. Treat a set of clusters with different components of the application as a single application
2. Replicate it across a different region

This increases the number of overall clusters and operational costs.

Even with the increased complexity, however, this approach provides the best security and performance for the application. With its component-based approach, the regulatory scope can be confined to a single component in a defined region without having to modify the complete application.

This approach also enables target troubleshooting and optimizations for specific components.

Platforms for Kubernetes multi-clusters

There are a lot of tools that support configuring or managing Kubernetes multi-cluster environments. Following is a list of such platforms.

• Rancher is an enterprise-grade container management platform with built-in support for Kubernetes multi-cluster management on various platforms from on-premise deployment to multi-cloud.
• Fleet is a GitOps-at-scale project designed to facilitate and manage a multi-cluster environment. It started as an extension of the Rancher but has since developed into its own project.
• Google Anthos is designed to extend the Google Kubernetes engine across hybrid and multi-cluster environments. This enables users to provide a consistent Kubernetes experience while having a centralized multi-cluster management environment.
• Microsoft Azure Arch is designed to extend Azure management and services across hybrid and multi-cloud environments. It is not limited to Kubernetes but can be used to extend Kubernetes clusters and Azure Arch providing a centralized location to manage, secure, and govern multi-cluster Kubernetes deployment.
• VMWare Tanzu is platform offered by VMWare to manage and secure Kubernetes infrastructure across multi-cloud environments.

Multi-clusters offer multi benefits

Kubernetes multi-clusters is an extensive field composed of multiple implementations and integration methods dictating multi-cluster deployments.

With all the advantages it offers for large-scale applications, it is paramount that you select the appropriate architecture for your multi-cluster environment and tool stack to manage the Kubernetes multi-cluster centrally.

Related reading

• BMC DevOps Blog
• Kubernetes Deployments Fully Explained
• How To Set Up a Continuous Integration & Delivery (CI/CD) Pipeline
• Continuous Delivery Metrics
• How to Setup a MongoDB Cluster
• What Is DevSecOps? Combining Development, Security & Operations

In any Kubernetes cluster, applications must be able to connect with the outside world to provide end-users with access to the application. Kubernetes Ingress is one option available—let’s take a look.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What is Kubernetes Ingress?

Kubernetes Ingress is an API object that provides routing rules to manage access to the services within a Kubernetes cluster. This typically uses HTTPS and HTTP protocols to facilitate the routing.

Ingress is the ideal choice for a production environment. Users can expose services within the Kubernetes cluster without having to create multiple load balancers or manually exposing services.

Moreover, K8s Ingress offers a single entry point for the cluster, allowing administrators to manage the applications easily and diagnose any routing issues. This decreases the attack surface of the cluster, automatically increasing the overall security.

Some use cases of Kubernetes Ingress include:

• Providing externally reachable URLs for services
• Load balancing traffic
• Offering name-based virtual hosting
• Terminating SSL (secure sockets layer) or TLS (transport layer security)

Kubernetes Ingress supports multiple protocols, authentication, content-based routing, etc., and allows users to manage and configure them in Kubernetes clusters easily.

Kubernetes Ingress consists of two core components:

• Ingress API object. The API object indicates the services that need to be exposed outside the cluster. It consists of the routing rules.
• Ingress Controller. Ingress Controller is the actual implementation of Ingress. It is usually a load balancer that routes traffic from the API to the desired services within the Kubernetes cluster.

What is an Ingress controller?

The actual implementation of Kubernetes Ingress, the Ingress controller is responsible for the traffic routing from external sources to the appropriate services in the Kubernetes cluster. Ingress controllers support routing through both the transport layer (OSI – Layer 4) and the application layer (OSI – Layer 7) in the OSI model.

The application layer routing is preferred over simple transport layer routing because it offers greater control, such as load balancing external traffic based on requests. The Ingress controllers are neither automatically started with a Kubernetes cluster nor a part of the default “kube-controller-manager” binary.

Kubernetes offers built-in support for AWS, GCE, and nginx ingress controllers and also supports integration with third-party ingress controllers. The choice of the Ingress controller depends on:

• Your application requirements
• The host environment

Simply put, the Ingress controller is an application that runs within the Kubernetes cluster and provisions a load balancer according to the requirements of Ingress. (The load balancer can be a software load balancer, external hardware load balancer, or cloud load balancer, each type requiring different controller implementations.)

Ingress vs alternatives for exposing services

Using Ingress is not the only way to expose services.

A Kubernetes service resource provides an abstract way to expose a set of targeted application pods as a network service. At the same time, Kubernetes uses ClusterIP, NodePort, and Load Balancer methods to expose services both internally within nodes and externally outside the cluster.

• ClusterIP is the default service type that exposes services using an internal cluster IP. This method exposes only the service within the cluster and is unreachable externally.
• NodePort exposes the service using a static port associated with the desired node. NodePort configuration will automatically create the ClusterIP to route the traffic internally. Users can access the service externally by using the IP address of the node and the exposed port.
• LoadBalancer exposes the service using the cloud provider’s load balancer and automatically creates the necessary NodePorts and ClusterIPs to route the desired traffic.

(Learn about NodePort, Port & TargetPort.)

Individually exposing and managing services is inefficient and not scalable in large Kubernetes clusters. That’s how Ingress becomes the ideal solution—by acting as the entry point for the whole Kubernetes cluster. This enables users to:

• Manage all the routing rules from a centralized resource
• Expose multiple services under a single IP address

Creating an Ingress resource

Now let’s have a look at how to create an Ingress resource.

As with any other resource in Kubernetes, Ingress resource also consists of apiVersion, kind, and metadata fields with the desired rules and configurations under the spec section.

Simple Ingress configuration

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/app-root: /test_app
  name: testapproot
  namespace: default
spec:
  rules:
  - host: testapproot.mycluster.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /

When defining an ingress resource, the name of the ingress object needs to adhere to valid subdomain DNS naming conventions. Furthermore, annotations can be used to configure ingress resources with specific options. In the above YAML file, we use the “nginx.ingress.kubernetes.io/app-root” option to define the application root that the ingress controller needs to redirect when the root path is called (path: /).

Different types of ingress controls will include different annotation support, and the user needs to modify the annotations to suit the desired controller.

Ingress rules

Ingress rules are the core of Kubernetes Ingress. They define what happens to the incoming traffic and to which services that traffic should be directed. Each HTTP rule consists of multiple fields to define the rule properly.

• Host is an optional parameter that defines which host the rule gets applied. When the host parameter is not defined, the rule is applied to all incoming HTTP traffic.
• Paths refers to the list of paths that are associated with a backend. Both the host and path should match to route the traffic to the desired service properly.
• Backend is the destination to which the incoming traffic should be ultimately directed. This can be a Kubernetes service or a custom resource backend to indicate another resource under the same namespace (e.g., storage bucket).

Any request that does not match any of the defined rules will be directed to the default backend. This default backend is a part of the ingress controller itself and not specified in the ingress resource.

Defining multiple Ingress rules

Now that we have a better understanding of the ingress resource, we can see how to define multiple ingress rules with different hosts and backends.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: myapp
spec:
  rules:
  - host: "app.mycluster.com"
    http:
      paths:
      - pathType: Prefix
        path: "/home"
        backend:
          serviceName: http-app
          servicePort: 80
      - pathType: Prefix
        path: "/admin"
        backend:
          serviceName: http-app
          servicePort: 8080
  - host: "data.mycluster.com"
    http:
      paths:
      - pathType: Prefix
        path: "/charts"
        backend:
          serviceName: http-charts
          servicePort: 81

In the above configuration, we have configured three routing rules for two different hosts that consume different services.

host: “app.mycluster.com”

This section consists of two rules that only apply to traffic directed to app.mycluster.com. If the URL path matches either /home or /admin, the traffic will get directed to different ports of the same service.

host: “data.mycluster.com”

The single rule configured will apply to traffic directed at data.mycluster.com host, and if the URL path matches /charts, it will direct the traffic to port 81 of the “http-charts” service.

Kubernetes Ingress path types

When defining rules, the path must include a corresponding path type to validate the rules properly. Kubernetes has three path types:

• This matches the configurations of an IngressClass defined in an Ingress Controller. It will be treated as a separate path type or the same as Prefix or Exact types, depending on the implementation of the class.
• We can use this type when the user needs to match a URL path exactly with case sensitivity.
• This will match the URL based on the path prefix split by the forward-slash (/). This is also case sensitive, and the matching process is done from element to element in the URL.

Let’s now look at some path matching scenarios.

In instances where multiple matches are available for a request, the request with the longest matching path will be given preference. If both paths are an exact match, the ingress controller will select the exact path type over the prefix to apply the necessary ingress rule.

(Dive deeper into all the matching scenarios in the official Kubernetes documentation.)

Types of Ingress

There are three types of Ingress in Kubernetes;

• Single service. This is Ingress backed by a single service where a single service is exposed. To define this, specify a default backend without any rules.
• Simple fanout. The fanout type is where a single entry point (single IP address) is used to expose multiple services. There, URL traffic gets routed to the desired destination depending on the request. This method allows users to reduce the number of load balancers and easily manage the routing within the cluster.
• Name-based virtual hosting. This method can be used to route traffic from a single entry (IP address) to different hosts within the cluster. Here, the traffic is first targeted towards a specified host before any other routing is done.

Users can secure an ingress by specifying a secret with a supported TLS key and a certificate. The ingress resource supports a single TLS port (443) and assumes TLS termination at the ingress point. Then the internal communications happen via plain text.

Manage services better, with improved security

Kubernetes Ingress provides a unified resource to manage and route traffic to your applications in a Kubernetes cluster without having to manage services and load balances manually.

Related reading

• BMC DevOps Blog
• Kubernetes Best Practices for Enhanced Cluster Efficiency
• How Containers & Kubernetes Work Together
• Kubernetes vs Docker Swarm: Comparing Container Orchestration Tools
• Docker Security: 14 Best Practices for Securing Docker Containers
• The State of Containers Today: A Report Summary

In this era of fast-paced technology, more and more organizations and teams are moving towards agile practices with rapidly evolving software development lifecycles. With that, Continuous Delivery (CD) has become a major part of the DevOps process, where the whole software release process is automated—the build, the testing, the deployment.

With the popularity of Kubernetes for developing containerized applications, the need is growing for a reliable continuous delivery platform with native support for deployments on Kubernetes clusters.

In this article, we will explore Spinnaker, one tool that facilitates integrating Kubernetes into a CD pipeline.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What is Spinnaker?

Spinnaker is an open-source continuous delivery platform targeted at multi-cloud deployments. Initially developed by Netflix as a successor for their internal Asgard platform, Spinnaker has now evolved into a standalone continuous delivery platform.

Spinnaker provides the necessary tools to manage your code from commit to multi-cloud delivery with its powerful abstraction layer.

(Read our comprehensive Spinnaker introduction.)

Spinnaker features

Let’s look at Spinnaker’s key features.

Multi-cloud deployment

Spinnaker natively supports deploying applications in multiple cloud providers. It supports cloud platforms such as AWS, Azure, GCP, Oracle Cloud, Cloud Foundry, and OpenStack.

CI integrations

Spinnaker’s robust and automated release pipelines can be easily integrated with other tools such as:

• Jenkins
• Travis CI
• Git event
• CRON jobs

This allows developers to effectively carry out various tasks from artifact collection to pipeline triggering.

Monitoring & notification integrations

Spinnaker can be easily integrated with monitoring solutions such as Datadog, Prometheus, SignalFx, and Stackdriver for metrics collection and analysis.

Additionally, you can configure Spinnaker to work with services such as Slack, HipChat, SMS to provide notifications.

Support for different deployment strategies

Spinnaker includes built-in deployment strategies like highlander, canary, and red/black and can be easily configured to adapt to any custom deployment strategy.

Additionally, Spinnaker’s ability to create and deploy immutable images allows developers to:

• Deploy faster with easier rollbacks
• Handle config drift issues much more effectively.

Baked-in security practices

Spinnaker comes with strong security features such as built-in role-based access control with support for multiple authentication mechanisms, including Oauth, LDAP, X.509 certs, Azure Groups, etc. Moreover, it allows convenient isolation of projects for maximum security.

Other features of Spinnaker, such as manual judgments and chaos monkey integrations, enable developers to review and test the deployments for instance failures before releasing them.

Continuous Delivery with Spinnaker

Spinnaker consists of two core feature sets that are helpful in the deployment:

• Application management
• Application deployment

Application management

The application management feature consists of all the necessary tools to view and manage cloud infrastructure. Spinnaker operates on a services model where multiple services are used to facilitate a product. These types of services are sometimes referred to as “applications” or “microservices.”

Applications, clusters, and server groups are the key components in the application management feature. They define services. Then, load balancers and firewalls expose those services to the wider world.

• Applications are any collection of clusters with load balancers and firewalls. It represents a service that needs to be deployed with all the service and infrastructure configurations.
• Clusters are the logical groupings of server groups. Importantly, clusters do not correspond to Kubernetes clusters. They are simply collections of server groups regardless of the underlying hardware or software architecture.
• Server groups are the base resource of Spinnaker, which identifies the deployable artifact (container image, source, VM image, etc.) with the basic configurations such as no of instances, scaling policies, metadata, etc. When deployed, these server groups correspond to a collection of instances of running software like Kubernetes pods or VM instances. Server groups can also be associated with load balancers and firewalls.
• Load balancer manages the traffic to each instance of Server Groups by associating with ingress protocols and ports.
• Firewall acts as a gateway for managing network traffic access. Furthermore, it incorporates firewall rules defined by IP ranges, communication protocols, and ports to allow traffic.

Application deployment

The deployment feature set creates and manages the continuous delivery pipelines. It consists of:

• Pipeline. The core component of the Spinnaker application deployment, the pipeline consists of multiple stages where different tasks are performed to deploy the application. These pipelines can be integrated with other CI tools and notification methods to build a seamless CI/CD pipeline.
• Stage. This is a set of sequential tasks to be carried out in the pipeline. Spinnaker provides some common stages that can be easily integrated into the pipeline. For instance, we can provide stages such as Deploy, Path, Scale, Undo Rollout, etc., for Kubernetes. Additionally, users can create custom stages utilizing webhooks and run jobs.
• Task. The smallest component of the deployment feature, a task consists of one specific task to be carried out within a stage in the pipeline.

Simple example: Spinnaker CD pipeline

The diagram illustrates a simple CI/CD workflow that incorporates Spinnaker to build a container image and deploy it in a Kubernetes cluster:

Let’s look at each step of the workflow:

1. Developers commit/push code to a centralized repository.
2. The completed code triggers Jenkins to test and build a container.
3. This container gets pushed to a container registry.
4. Spinnaker listens for a new image in the container registry and triggers the delivery pipeline for deployment.
5. The Spinnaker pipeline creates the necessary configurations (application configs, instances, scaling, etc.) and deploys the container in a target Kubernetes cluster after manual review.

Creating a solid CI/CD workflow allows developers to easily automate almost all the stages of the development and deployment process.

On top of that, leveraging tools like Spinnaker, which offers a solid continuous delivery pipeline with Kubernetes, will lead to faster, efficient, and less error-prone software development lifecycles. Refer to the official Hello Deployment tutorial for a complete deployment pipeline setup.

(Set up your own CI/CD pipeline.)

Extending Spinnaker pipelines

In this section, let us expand the above workflow a bit further. Consider a scenario with the following requirements:

• Deploy the application to a staging environment and then push it to the production environment after testing and manual review.
• Deploy the application to Kubernetes Clusters hosted on both Azure and AWS.

We can simply configure a Spinnaker pipeline to accommodate all the above requirements, as shown below.

In the above pipeline, we have defined multiple stages to accommodate deployments in two different cloud providers. (This is possible all thanks to Spinnaker’s robust multi-cloud support.) Once the initial configurations are made, the pipeline will execute multiple stages simultaneously to bake and deploy the images in Kubernetes clusters in both Azure and AWS.

Then, you can verify the images and staging deployment at the manual judgment stage and finally deploy them to the production environments of AWS and Azure. Spinnaker’s capabilities eliminate the need to create separate pipelines or define different workflows to handle deployments and infrastructure in multi-cloud deployments. All the necessary configurations and infrastructure are managed through Spinnaker.

These pipelines can then be further extended to provide notifications to inform the DevOps teams of the pipeline progression, deployment state, and so on.

Moreover, the ability to integrate chaos monkey allows developers to easily test the resiliency of a deployed application by simulating instance failures and address any identified issues.

The use cases for Spinnaker are endless. Whatever the requirement, Spinnaker can be adopted to facilitate it. Spinnaker can even be deployed inside a Kubernetes Cluster and utilize pipelines to deploy applications in the host cluster itself.

Robust CI/CD pipeline

The combination of Spinnaker and Kubernetes helps users to achieve the end goal of having a robust and automated CI/CD pipeline for containerized application deployments with baked-in best practices.

We can enjoy the following benefits by leveraging these two technologies.

• Seamless resource mapping using the Kubernetes provider v2
• Easily integrate using advanced development strategies
• Easily rollback deployments. Spinnaker will utilize the versioned manifests of the applications for these rollbacks
• Easily deploy with native K8s support
• Monitor health for the deployed applications and the Kubernetes cluster
• Integrate with multiple artifact stores and image registries to easily obtain applications and container images
• Leverage excellent Kubernetes features like scaling, load balancing, and high availability in the application deployment
• Easily integrate and extend using third-party tools within the CI/CD pipeline

Get the most out of containers

Spinnaker and Kubernetes complement each other to get the best out of containerized application deployment. Spinnaker is the leading platform for managing multi-cloud continuous delivery with an active community that includes Netflix, Google, Microsoft, etc.

Related reading

• BMC DevOps Blog
• Deploying vs Releasing Software: What’s The Difference?
• Deployment Pipelines (CI/CD) in Software Engineering
• Testing Automation Explained: Why & How To Automate Testing
• 6 Benefits of Deployment Automation
• Top 5 Best Practices for Software Development

A Kubernetes service is a logical collection of pods in a Kubernetes cluster. We can define a K8s service as an abstract way to load balance across the pods and expose an application deployed on a set of Pods. Moreover, using the inbuilt service mechanism in Kubernetes eliminates the need for implementing a separate service discovery mechanism.

In this article, we will discuss the structure of a Kubernetes service and how to utilize it.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What are Kubernetes services?

A Kubernetes service can be used to easily expose an application deployed on a set of pods using a single endpoint. A service is both a REST object and an abstraction that defines:

• A set of pods
• A policy to access them

Pods in a Kubernetes deployment are regularly created and destroyed, causing their IP addresses to change constantly. It will create discoverability issues for the deployed, application making it difficult for the application frontend to identify which pods to connect.

This is where the strengths of Kubernetes services come into play: services keep track of the changes in IP addresses and DNS names of the pods and expose them to the end-user as a single IP or DNS.

Kubernetes services utilize selectors to target a set of pods:

• For native Kubernetes applications (which use Kubernetes APIs for service discovery), the endpoint API will be updated whenever there are changes to the pods in the service.
• Non-native applications can use virtual-IP-based bridge or load balancer implementation methods offered by Kubernetes to direct traffic to the backend pods.

Attributes of a Kubernetes service

Here are general attributes of a service:

• A service is assigned an IP address (“cluster IP”), which the service proxies use.
• A service can map an incoming port to any targetPort. (By default, the targetPort is set to the same value of the port field, and it can be defined as a string.)
• The port number assigned to each name can vary in each backend pod. (For example, you can change the port number that pods expose in the next version of your backend software without breaking clients.)
• Services support TCP (default), UDP, and SCTP for protocols.
• Services can be defined with or without a selector.
• Services support a variety of port definitions

Now let’s see how services work.

Defining a Kubernetes service

As mentioned above, we can define Kubernetes services with or without selectors—let’s do both!

Defining a service with a selector

In the following example, we have defined a simple service exposing port 80 in the service while targeting port 8080 in the Pods with the selector label “app=webserver-nginx”.

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: webserver-nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Kubernetes will automatically assign an IP address (“Cluster IP”) which will then be used by service proxies to route the traffic. The controller for the selector will consistently monitor for Pods matching the defined label.

Some applications will require multiple ports to be exposed via the service. Kubernetes facilitates this using multi-port services where a user can define multiple ports in a single service object. When defining a multi-port service, the main requirement is to provide names for each port exposed so that they are unambiguous. These port names can:

• Only contain alphanumeric characters and the dash symbol
• Should start and end with an alphanumeric character.

In this example, we have exposed both ports 80 and 443 to target ports 8080 and 8090 to route HTTP and HTTPS traffic to underlying pods using the selector “app=webserver-nginx-multiport”

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: webserver-nginx-multiport
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
    - name: https
      protocol: TCP
      port: 443
      targetPort: 8090

Defining a service without a selector

When defining a service without a selector, you need to manually map the service to the corresponding IP address and port by adding an endpoints object. The reason is that the endpoint objects are not created automatically like with a selector since Kubernetes does not know to which Pods the service should be connected.

Service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Endpoint:

apiVersion: v1
kind: Endpoints
metadata:
  name: nginx-service
subsets:
  - addresses:
      - ip: 192.10.0.1
    ports:
      - port: 8080

When defining a service and the endpoint, the main consideration is that both the name of the service and the endpoint must be an exact match.

Some use cases for services without selectors include:

• Connecting to a different service on another Namespace or another cluster
• Communicating with external services, data migration, testing services, deployments, etc.

Creating a Kubernetes service

Now that we know the basic structure of a Kubernetes service, let’s use a practical example to better understand how to create a service. First, we’ll create a small deployment with four replicas of an apache web server.

apache-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-deployment
  labels:
    app: webserver
spec:
  replicas: 4
  selector:
    matchLabels:
      app: webserver
  template:
    metadata:
      labels:
        app: webserver
    spec:
      containers: 
      - name: apache
        image: bitnami/apache:latest
        ports:
        - containerPort: 80

Create the deployment.

kubectl apply -f apache-deployment.yaml

Result:

Check the status of the deployment.

kubectl get deployment apache-deployment

Result:

Now we have deployed the application to the apache server.

Next, we will create a service to access our deployed application. There are two methods to do that, either:

1. Create a YAML manifest for a service
2. Use the “kubectl expose” command

The expose command allows users to create the service from the command line directly.

We can use the following command to expose the “apache-deployment” with the ClusterIP service type. We will point the service to port 8080 as the “bitnami/apache” image uses port 8080 as the HTTP port.

kubectl expose deployment apache-deployment --name=apache-http-service --type=ClusterIP --port=8080 --target-port=8080

Result:

Check whether the service is created correctly.

kubectl get service apache-http-service

Result:

Now we need to forward the traffic to the service we created, and we can use the “kubectl port-forward” command for that. We will be directing traffic from host port 8090 to service port 8080.

kubectl port-forward service/apache-http-service 8090:8080

Result:

When we navigate to the “http://localhost:8090“, we will be directed to an Nginx landing page like the following.

We could have also defined a service using a YAML file for the above scenario, as shown below. There we have set the selector as “app=webserver.”

apiVersion: v1
kind: Service
metadata:
  name: apache-http-service
spec:
  selector:
    app: webserver
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

Types of Kubernetes services

There are four types of Kubernetes services:

ClusterIP

This is the default type that exposes the service on an internal IP of the cluster. These services are only accessible within the cluster. So, users need to implement port forwarding or a proxy to expose a ClusterIP to a wider ingress of traffic.

NodePort

A NodePort service exposes the service on the IP of each node at a static port. A ClusterIP service is created automatically to route the traffic to the NordPort service. Users can communicate with the service from the outside by requesting <NodeIP>:<NodePort>

LoadBalancer

This is the preferred solution to expose the cluster to the wider internet. The LoadBalancer type of service will create a load balancer (load balancer type depends on the cloud provider) and expose the service externally.

It will also automatically create ClusterIP and NodePort services and route traffic accordingly.

ExternalName

This type of service maps the service to the contents of the externalName field (Ex: app.test.com). It does this by returning a value for the CNAME record.

Service proxy implementations

The kube-proxy process implements a form of virtual IPs for all the service types, excluding the ExternalName type. Using proxying for services enables users to mitigate some DNS resolution issues.

For example, if a DNS record is expired, there might be instances where previous DNS lookups are still cached with the expired record, leading to resolution issues. Similarly, DNS records not getting properly propagated can also lead to resolution issues.

The following three methods can be used by the kube-proxy for routing traffic:

• Proxy-mode: userspace
• Proxy-mode: iptables
• Proxy-mode: ipvs

Proxy-mode: userspace

In this mode, kube-proxy monitors the Kubernetes master, and whenever a service or an endpoint gets created or removed, it allocates or deallocates a random port on the local node.

All the connections to this “proxy port” are proxied to the service and the necessary endpoints.

Proxy-mode: iptables

This mode also monitors the Kubernetes master for services and endpoints. However, it will also install iptable rules to capture traffic to the ClusterIp of the service and port and then redirect the traffic.

Additionally, it will install iptable rules for endpoint objects to select the backend Pod, while the default behavior is to select a random backend. (Pod)

Proxy-mode: ipvs

In this mode, the kube-proxy watches the services and endpoints and then calls the Netlink interface to create appropriate ipvs (IP Virtual Server) rules. This mode will periodically sync the ipvs rules with the services and endpoint to keep everything up to date.

Discovering Kubernetes services

You can use two methods to discover a service—DNS or Environment Variable.

DNS

This is the preferred method for service discovery. Here, the DNS server is added to the Kubernetes cluster that watches the Kubernetes API and creates DNS records for each new service. When the DNS is enabled, cluster-wide all Pods will be able to perform name resolution of services.

Environment Variables

In this method, kubelet adds environment variables to Pods for each active service. When using this method, the desired service must be created before creating the Pods which will use that service. Otherwise, the Pods would not get the necessary environment variables for the desired service.

Headless Services

When load-balancing and single service IP are not required, users can create a headless service by explicitly specifying “none” for the cluster IP field (.spec.clusterIP). These headless services do not have associated Cluster IPs, and no load balancing or proxying is provided for them by the platform. The Kube-proxy also ignores these services.

DNS configuration of these services depends on the selectors defined in the services.

• Headless service with selectors. The endpoint controller will create the endpoint records in the API, modifying the DNS record to return an A record that points to the necessary Pods.
• Headless service without selectors. The endpoint controller will not create any endpoint records without having the selectors configured.

Summing up K8s services

With that, we’ve covered all the basics of Kubernetes services. We now know about services, including the different types, service discovery, and service proxy usages. Services are an integral part of Kubernetes that provide means to facilitate communication between pods, external services, and users.

Related reading

• BMC DevOps Blog
• Kubernetes Guide, with 20+ articles and tutorials
• Kubernetes Deployments Fully Explained
• The State of Kubernetes Today
• Kubernetes vs Docker Swarm: Comparing Container Orchestration Tools
• The State of Containers Today: A Report Summary

Kubernetes networking is an integral part of managing and making communication easier within a Kubernetes cluster. It manages a wide range of operations, such as:

• Handling internal container communication
• Exposing the containers to the internet

This article introduces you to key Kubernetes networking concepts.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

Networking in Kubernetes

First, let’s review the basic networking types in a Kubernetes cluster:

• Container-to-container networking
• Pod-to-pod networking
• Pod-to-service networking
• Internet-to-service networking

Now, let’s take a look at how each type operates to provide connectivity within the cluster.

Container-to-container networking

In the most basic configurations, container to container communication within a single pod takes place via the localhost and port numbers. This is possible because the containers in the pod are located in the same network namespace.

A network namespace is a logical networking stack that includes routes, firewall rules, and network devices which provides a complete network stack for processes within the namespace. There can be multiple namespaces on the same virtual machine.

In Kubernetes, there is a hidden container called a pause container that runs on all the pods in Kubernetes. This container keeps the namespace open even if all the other containers in the pod are nonfunctional.

A new networking namespace is assigned to each pod created. Containers within the pod can use this networking namespace via localhost. However, a user must be aware of the port conflicts: If the containers use the same port, networking issues will arise within the containers.

Pod-to-pod networking

Each pod in a Kubernetes node gets assigned a dedicated IP address with a dedicated namespace. Due to an automatic assignment, users do not need to explicitly create links between pods. This allows us to tread pod networking operations such as port allocation, load balancing, and naming similar to a virtual machine or a physical host.

Kubernetes imposes three fundamental requirements on any network.

1. Pods on a node can communicate with all pods on all nodes without NAT.
2. Agents on a node (system daemons, kubelet) can communicate with all the pods on that specific node.
3. Pods in a host network of a node can communicate with all pods on all nodes without NAT (Only for platforms such as Linux, which supports pods running on the host network)

Communication between pods in the same node

Kubernetes creates a virtual ethernet adapter for each pod, and it is connected to the network adaptor of the node.

When a network request is made, the pod connects through the virtual ethernet device associated with the pod and tunnels the traffic to the ethernet device of the node.

In a Kubernetes node, there is a network bridge called cbr0, which facilitates the communication between pods in a node. All the pods are a part of this network bridge. When a network request is made, the bridge checks for the correct destination (pod) and directs the traffic.

Communication between pods in different nodes

When the requested IP address cannot be found within the pod, the network bridge directs the traffic to the default gateway. This would then look for the IP within the cluster level.

Kubernetes keeps a record of all the IP ranges associated with each node. Then an IP address is assigned to the pods within the nodes from the range assigned to the node. When a request is made to an IP address in the cluster, it will:

1. Look for the IP range for the requested IP.
2. Then direct it to the specific node and then to the correct pod.

Pod-to-service networking

A service is an abstraction that routes traffic to a set of pods. In other words, a service will map a single IP address to a set of pods.

When a network request is made, the service proxies the request to the necessary pod. This proxy service happens via a process called Kube-proxy that runs inside each node.

The service would get its IP within the cluster when a request first reaches the service IP before being forwarded to the actual IP of the pod. This is an important feature in Kubernetes—it decouples the dependency of networking directly to each pod. The pods can be created and destroyed without worrying about network connectivity. This is because the service will automatically update its endpoints when the pods change with the IP of the pod.

The service would get its IP within the cluster. Any request first reaches the service IP before being forwarded to the actual IP of the Pod. This is an important feature in Kubernetes as it decouples the dependency of networking directly to each Pod. Thus, pods can get created and destroyed without worrying about network connectivity as the service will automatically update its endpoints with the IP of the pod that it targets as the Pods changes.

A service knows which Pods to target using the label selector. The label selector is the core grouping primitive in Kubernetes; via a label selector, the client or user can identify a set of objects.

• The label selector will look for the specified labels in each Pod and match them with the service accordingly.
• Without the selector, properly configured services cannot keep track of the Pods and will lead to communication issues within the cluster.

DNS for internal routing in a Kubernetes cluster

Each cluster comes with inbuilt service for DNS resolution.

• A domain name is assigned to each service in the cluster
• A DNS name is assigned to the Pods.

(These can be manually configured via the YAML file using the hostname and subdomain fields.)

When a request is made via the domain name, the Kubernetes DNS service will resolve the request and point it to the necessary internal service from the service. The Kube-proxy process will point it to the endpoint pod.

Internet-to-service networking

In the above sections, we have covered the internal networking basics of a Kubernetes cluster. The next step is to expose the containerized application to the internet.

Unlike the previous networking types, exposing the Kubernetes cluster to the internet depends on both the:

• Underlying cluster infrastructure
• Network configurations

The most common method used to handle traffic is by using a load balancer. In Kubernetes, we can configure load balancers. When a service is created, users can:

• Specify a load balancer that will expose the IP address of the load balancer.
• Direct traffic to the load balancer and communicate with the service through it.

Tools for Kubernetes networking

There are plenty of tools that we can use to configure networking within Kubernetes while adhering to all the guidelines and requirements imposed by the cluster.

• Cilium is an open-source software for providing and securing network connectivity between application containers. Cilium is L7/HTTP aware and can enforce network policies from L3-L7 using identity-based security models.
• Flannel is a simple network overlay that meets Kubernetes requirements.
• Kube-router is a turnkey solution for Kubernetes networking. Kube-router provides a lean and powerful alternative to default Kubernetes network components from a single DaemonSet/Binary.
• Antrea is a Kubernetes-native open source networking solution that leverages Open vSwitch as the networking data plane.

And here are cloud vendor-specific container network interfaces:

• AWS VPC CNI for Kubernetes is the Amazon Web Services specific CNI.
• Azure CNI for Kubernetes is the Microsoft Azure specific CNI.

Please refer to the official Kubernetes documentation for the full list of available network tools.

K8s networking optimizes performance

Kubernetes cluster networking can be configured to suit any user requirements to optimize the performance of the containerized application and the cluster.

Related reading

• BMC DevOps Blog
• Kubernetes Guide, with 20+ articles and tutorials
• Bring Kubernetes to the Serverless Party
• The State of Kubernetes Today
• Kubernetes vs Docker Swarm: Comparing Container Orchestration Tools
• The State of Containers Today: A Report Summary

Kubernetes Deployment is the process of providing declarative updates to Pods and ReplicaSets. It allows users to declare the desired state in the manifest (YAML) file, and the controller will change the current state to the declared state.

So, let’s look at how to create and use Kubernetes deployments. I’ll walk you through how to utilize Kubernetes deployments to simplify the deployment process and manage, scale, and roll back deployments.

For the purpose of this article, we will be using a locally deployed Kubernetes cluster in a Windows environment utilizing the Windows Subsystem for Linux (WSL).

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

Creating a Kubernetes deployment

Let’s first create a simple Nginx deployment with four replicas. Like any other Kubernetes configuration, a deployment file will contain:

• apiVersion (apps/v1)
• Kind (Deployment)
• The metadata with a spec section to define the replicas and configurations related to the deployment

nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  # Define the Deployment Name
  name: nginx-deployment
  labels:
    app: webserver
spec:
  # Define the Number of Pods
  replicas: 4
  # Define the Selector
  selector:
    matchLabels:
      app: webserver
  template:
    metadata:
      labels:
        app: webserver
    spec:
      containers: # Container Details
      - name: nginx
        image: nginx:latest # Image
        ports:
        - containerPort: 80

We can create the deployment using the following command. We have added the –record flag to save the command.

kubectl apply -f nginx-deployment.yaml --record

Result:

We have defined the following entries in the metadata and specs sections of the above deployment file.

• The deployment name and a label (app: webserver).
• The number of replicas (pods), the selector in which the controller will select the targeted pods. (The label is used to select the necessary pods using the matchLabels field.) The template section contains the actual template for the pod. This section defines the metadata that each pod will have with the specs (container definition). In this instance, we have defined the Nginx image and the container port as 80.

If we check the Kubernetes cluster after some time, we can see that the Pods are deployed with the given template. Additionally, we can retrieve details of the deployment using the describe command.

kubectl get all

Result:

kubectl describe deployment nginx-deployment

Result:

Exposing the ReplicaSet

Now we have created a deployment and need to verify if the Nginx web servers were deployed correctly.

The straightforward way to achieve this is to create a service object that exposes the deployment. An important fact to note here is that the way we expose the deployment and the parameters can vary depending on the configurations of the Kubernetes cluster.

kubectl expose deployment nginx-deployment --type=LoadBalancer --name=nginx-web-server

Result:

kubectl get services

Result:

Then we can navigate to the localhost port 80 to check if we can see the default Nginx server landing page as below.

Discovering the Kubernetes deployment details

When managing a Kubernetes cluster, the initial step would be to check for a successful deployment. For this purpose, we can use the kubectl rollout status and kubectl get deployment commands.

• kubectl rollout status informs the user if the deployment was successful.
• kubectl get deployment shows the desired and updated number of replicas, the number of replicas running, and their availability. As mentioned previously, we can use the kubectl describe command to a complete picture of the deployment.

kubectl rollout status deployment nginx-deployment
kubectl get deployment nginx-deployment

Result:

We can fetch information about the ReplicaSets created during deployment using the kubectl get ReplicaSet command.

By default, Kubernetes will automatically append a pod-template-hash value to the ReplicaSet name. However, do not rename the ReplicaSet as it will break the deployment.

kubectl get replicaset

Result:

The kubectl get pod command can be used to get only the information about the pods related to the deployment while defining a selector. In this instance, we will be using the “app:webserver” label as the selector.

kubectl get pod --selector=app=webserver

Result:

Managing Kubernetes deployments

Now we know how to create a deployment and retrieve information about the said deployment. The next stage is to manage the deployment. What differentiates deployments from a simple ReplicaSet is that deployments enable users to update the pods (pod templates) without causing any interruption to the underlying application.

Performing Rolling Update on a deployment

Let’s assume that we need to change the Nginx server version in our deployment to target our application to a specific server version. We can do this by either:

• Using the kubectl set image command
• Changing the deployment configuration file

Using the set image command

The set image command can be used with the container name of the template and the required image name to update the pods.

kubectl set image deployment nginx-deployment nginx=nginx:1.19.10

Result:

We will get the rollout process if we run the get rollout status command immediately.

kubectl rollout status deployment nginx-deployment

Result:

Changing the deployment configuration file

We can use the edit deployment command to edit the configuration file. Navigate the relevant section (container image) and make necessary changes. Kubernetes will start the process of updating the pods the moment we save the new configuration.

kubectl edit deployment nginx-deployment

Deployment Edit View (nginx-deployment):

Deployment strategies

Kubernetes uses two deployment strategies called “Recreate” and “RollingUpdate” to recreate pods. We can define those strategies in .spec.strategy.type field. The RollingUpdate strategy is the default for deployments.

• Recreate will delete all the existing pods before creating the new pods.
• RollingUpdate will recreate the pods in a rolling update fashion. Moreover, it will delete and recreate pods gradually without interrupting the application availability. This strategy utilizes the maxUnavailable and maxSurge values to control the rolling update process..
  • maxUnavailable defines the maximum number of pods that can be unavailable in the update process.
  • maxSurge defines the maximum number of pods that can be created.

In our deployment, we haven’t explicitly defined a strategy so that Kubernetes will use the default RollingUpdate strategy. We can use the describe command to verify the current strategy for the deployment.

kubectl describe deployment nginx-deployment | grep Strategy

Result:

From the above output, we can discern that the default configurations for the RollingUpdate strategy are 25% for max unavailable and 25% for the max surge values. We are creating four replicas in our configuration. According to the above configuration in the update process, a single pod will get destroyed while a single pod is created (25% of 4 is 1).

We can view the Events in the deployment using the describe command to gain a better understanding of the update process.

kubectl describe deployment nginx-deployment

Result:

If we look at the current ReplicaSet, we can notice that it has four pods whereas the old ReplicaSet does not contain any pods.

kubectl get replicasets

Result:

Pausing & resuming deployments

Kubernetes deployments provide the ability to pause and resume deployments. This enables users to modify and address issues without triggering a new ReplicaSet rollout.

We can use the “rollout pause deploy” command to pause the deployment.

kubectl rollout pause deploy nginx-deployment

Result:

Now, if we update the Nginx image in the paused status, the controller will accept the change, yet it will not trigger the new ReplicaSet rollout. If we look at the rollout status, it will indicate a pending change.

kubectl set image deployment nginx-deployment nginx=nginx:1.20 --record
kubectl rollout status deployment nginx-deployment

Result:

You can simply run the “rollout resume deploy” command to resume the deployment.

kubectl rollout resume deploy nginx-deployment
kubectl rollout status deployment nginx-deployment

Result:

Scaling deployments

As the Deployments rely on ReplicaSets to manage the pods, we can scale up or down the number of pods. This scaling can be done either:

• Manually
• By configuring an auto-scaling rule

Manual scaling

We can use the scale command with the replica parameter to scale the deployment to the desired number. For instance, we will use the following command to scale up our deployment from 4 pods to 8 pods.

kubectl scale deployment nginx-deployment --replicas=8
kubectl rollout status deployment nginx-deployment

Result:

If we look at the pods associated with this deployment, we can see that it has eight pods now.

kubectl get pod --selector=app=webserver

Result:

Autoscaling

The best practice for scaling deployments would be to configure an auto-scaling rule so that the pods will scale according to predefined thresholds.

So, let’s go ahead and create an autoscaling rule for our deployment, which will scale according to the CPU load of the node.

kubectl autoscale deployment nginx-deployment --min=5 --max=10 --cpu-percent=70

Result:

According to the above configuration, if the CPU load is greater than 70%, the deployment will scale until the maximum number of pods is reached (maximum ten pods). On the other hand, it will scale back gradually until there are five pods (minimum five pods) when the load is reduced.

Rolling back a deployment

Kubernetes also supports rolling back deployments to the previous revision. This is a crucial feature enabling users to undo changes in deployment.

For instance, if a critical production bug was deployed to the cluster, we can simply roll back the deployment to the previous revision easily with no downtime until the bug is fixed.

Let’s assume that we have updated our configuration with an incorrect image. (We will be using the caddy webserver in this example.)

kubectl set image deployment nginx-deployment nginx=caddy:latest --record

Result:

This is where the Deployment controller’s history function comes into play. The controller keeps track of any changes to the pod template and keeps them in history. When we specify the record flag in a command, it will be reflected in the history.

kubectl rollout history deployment nginx-deployment

Result:

We can use the revision number to inform the deployment controller to roll back our deployment to the previous revision.

First, let’s verify if the revision we are going to roll back is the correct deployment. Here, we are trying to roll back to the third revision. Simply specify the revision in the history command to get the details of the indicated revision.

kubectl rollout history deployment nginx-deployment --revision=3

Result:

After confirming the revision, we can utilize the “kubectl rollout undo” command with the “to-revision” parameter to roll back the deployment.

kubectl rollout undo deployment nginx-deployment --to-revision=3
kubectl rollout status deployment nginx-deployment

Result:

That’s it! Now we have successfully rolled back the deployment.

After the rollback, we won’t be able to see the third revision as we have deployed it as the current configuration. However, we will be able to see a new record with a higher revision number for the newly rolled back deployment.

kubectl rollout history deployment nginx-deployment

Result:

Summing up K8s deployments

In this article, we have only scratched the surface of the capabilities of Kubernetes deployments. Users can create more robust containerized applications to suit any need by combining deployments with all the other Kubernetes features.

Related reading

• BMC DevOps Blog
• Kubernetes Guide, with 20+ articles and tutorials
• Bring Kubernetes to the Serverless Party
• The State of Kubernetes Today
• Kubernetes vs Docker Swarm: Comparing Container Orchestration Tools
• The State of Containers Today: A Report Summary

Kubernetes is a leading open-source engine that orchestrates containerized applications.

In this article, we will have a look at the DaemonSet feature offered by Kubernetes. We’ll walk you through use cases and how to create, update, communicate with, and delete DaemonSets.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What is a Kubernetes DaemonSet?

The DaemonSet feature is used to ensure that some or all of your pods are scheduled and running on every single available node. This essentially runs a copy of the desired pod across all nodes.

• When a new node is added to a Kubernetes cluster, a new pod will be added to that newly attached node.
• When a node is removed, the DaemonSet controller ensures that the pod associated with that node is garbage collected. Deleting a DaemonSet will clean up all the pods that DaemonSet has created.

DaemonSets are an integral part of the Kubernetes cluster facilitating administrators to easily configure services (pods) across all or a subset of nodes.

DaemonSet use cases

DaemonSets can improve the performance of a Kubernetes cluster by distributing maintenance tasks and support services via deploying Pods across all nodes. They are well suited for long-running services like monitoring or log collection. Following are some example use cases of DaemonSets:

• To run a daemon for cluster storage on each node, such as glusterd and ceph.
• To run a daemon for logs collection on each node, such as Fluentd and logstash.
• To run a daemon for node monitoring on every note, such as Prometheus Node Exporter, collectd, or Datadog agent.

Depending on the requirement, you can set up multiple DaemonSets for a single type of daemon, with different flags, memory, CPU, etc. that supports multiple configurations and hardware types.

Scheduling DaemonSet pods

By default, the node that a pod runs on is decided by the Kubernetes scheduler. However, DaemonSet pods are created and scheduled by the DaemonSet controller. Using the DaemonSet controller can lead to Inconsistent Pod behavior and issues in Pod priority preemption.

To mitigate these issues, Kubernetes (ScheduleDaemonSetPods) allows users to schedule DaemonSets using the default scheduler instead of the DaemonSet controller. This is done by adding the NodeAffinity term to the DaemonSet pods instead of the .spec.nodeName term. The default scheduler is then used to bind the Pod to the target host.

The following is a sample NodeAffinity configuration:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          # Key Name
          - key: disktype
            operator: In
            # Value
            values:
            - ssd            
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent

Above, we configured NodeAffinity so that a pod will only be created on a node that has the “disktype=ssd” label.

Additionally, DaemonSet pods adhere to taints and tolerations in Kubernetes. The node.kubernetes.io/unschedulable:NoSchedule toleration is automatically added to DaemonSet pods. (For more information about taints and tolerations, please refer to the official Kubernetes documentation.)

How to create a DaemonSet

As for every other component in Kubernetes, DaemonSets are configured using a YAML file. Let’s have a look at the structure of a DaemonSet file.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: test-daemonset
  namespace: test-daemonset-namespace
  Labels:
    app-type: test-app-type
spec:
  template:
    metadata:
      labels:
        name: test-daemonset-container
  selector:
    matchLabels:
      name: test-daemonset-container

As you can notice in the above structure, the apiVersion, kind, and metadata are required fields in every Kubernetes manifest. The DaemonSet specific fields come under the spec section—these fields are both mandatory.

• template. This is the pod definition for the Pod that needs to be deployed across all the nodes. A pod template in a DaemonSet must have its RestartPolicy set to “Always,” and by default it will take “Always” if you havne’t specified a RestartPolicy.
• selector. The selector for the pods managed by the DaemonSet. This value must be a label specified in the pod template. (In the above example, we have used the name: test-daemonset-container as the selector.) This value is fixed and cannot be changed after the initial creation of the DaemonSet. Changing this value will cause pods created via that DaemonSet to be orphaned. Kubernetes offers two ways to match matchLabels and matchExpressions for creating complex selectors.

Other optional fields

• template.spec.nodeSelector – This can be used to specify a subset of nodes that will create the Pod matching the specified selector.
• template.spec.affinity – This field can be configured to set the affinity that would run the pod only on nodes that match the configured affinity.

Creating a DaemonSet

Now let’s go ahead with creating a sample DaemonSet. Here, we will be using a “fluentd-elasticsearch” image that will run on every node in a Kubernetes cluster. Each pod would then collect logs and send the data to ElasticSearch.

daemonset-example.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-elasticsearch-test
  namespace: default # Name Space
  labels:
    k8s-app: fluentd-logging
spec:
  selector: # Selector
    matchLabels: 
      name: fluentd-elasticsearch-test-deamonset
  template: # Pod Template
    metadata:
      labels:
        name: fluentd-elasticsearch-test-deamonset
    spec:
      tolerations: # Tolerations
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers: # Container Details
      - name: fluentd-elasticsearch
        image: quay.io/fluentd_elasticsearch/fluentd:v2.5.2
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

First, let’s create the DaemonSet using the kubectl create command and retrieve the DaemonSet and pod information as follows:

kubectl create -f daemonset-example.yaml

kubectl get daemonset

kubectl get pod -o wide

As you can see from the above output, our DaemonSet has been successfully deployed.

Depending on the nodes available on the cluster, it will scale automatically to match the number of nodes or a subset of nodes on the configuration. (Here, the number of nodes will be one as we are running this on a test environment with a single node Kubernetes cluster.)

Updating DaemonSets

When it comes to updating DaemonSets, if a node label is changed, DaemonSet will automatically add new pods to matching nodes while deleting pods from non-matching nodes. We can use the “kubectl apply” command to update a DaemonSet, as shown below.

There are two strategies that can be followed when updating DaemonSets:

• The default strategy in Kubernetes, this will delete old DaemonSet pods and automatically create new pods when a DaemonSet template is updated.
• When using this option, new DaemonSet pods are created only after a user manually deletes old DaemonSet pods.

These strategies can be configured using the spec.updateStrategy.type option.

Deleting DaemonSets

Deleting a DaemonSet is a simple task. To do that, simply run the kubectl delete command with the DaemonSet. This would delete the DaemonSet with all the underlying pods it has created.

We can use the cascade=false flag in the kubectl delete command to only delete the DaemonSet without deleting the pods.

Communicating with pods created by DaemonSet

There are multiple methods to communicate with pods created by DaemonSets. Here are some available options:

• Push. This way, pods can be configured to send information to other services (monitoring service, stats database). However, they do not receive any data.
• NodeIP & Known Port. Pods are reachable via the node IPs using hostPort. Users can then utilize the known ports and the node IP to communicate with the pods.
• DNS. In this method, users can configure a headless service with the same pod selector to discover DaemonSet using the endpoints resource.
• Service.  To select a random node in a DaemonSet, which we can use to create a service with the same pod selector.

DaemonSet summary

In this article, we learned about Kubernetes DaemonSets. These configurations can easily facilitate monitoring, storage, or logging services that can be used to increase the performance and reliability of both the Kubernetes cluster and the containers.

Related reading

• BMC DevOps Blog
• Kubernetes Guide, with 20+ articles and tutorials
• Kubernetes Best Practices for Enhanced Cluster Efficiency
• Kubernetes vs Docker Swarm: Comparing Container Orchestration Tools
• The State of Kubernetes Today
• Container Management Platforms: Which Are Most Popular?

In programming, we use env files or separate configuration files to store settings, configurations, or variables that are required to execute the program. In Kubernetes, we can use ConfigMaps to achieve the same functionality.

To understand ConfigMap properly, you should have some knowledge of Kubernetes, pods, and basic Kubernetes cluster management.

(This article is part of our Kubernetes Guide. Use the right-hand menu to navigate.)

What is a ConfigMap?

A ConfigMap is a Kubernetes API object that can be used to store data as key-value pairs. Kubernetes pods can use the created ConfigMaps as a:

• Configuration file
• Environment variable
• Command-line argument

ConfigMaps provides the ability to make applications portable by decoupling environment-specific configurations from the containers.

Importantly, ConfigMaps are not suitable for storing confidential data. They do not provide any kind of encryption, and all the data in them are visible to anyone who has access to the file. (Kubernetes provides secrets that can be used to store sensitive information.)

Another consideration of ConfigMaps is the size of the file, as we are trying to store application configuration ConfigMap files limited to 1MB. For larger data sets, it’s better to use separate file mounts, databases, or file services.

ConfigMap example

kind: ConfigMap
apiVersion: v1
metadata:
  name: example-configmap
  namespace: default
data:
  # Configuration Values are stored as key-value pairs
  system.data.name: "app-name"
  system.data.url: "https://app-name.com"
  system.data.type_one: "app-type-xxx"
  system.data.value: "3"
  # File like Keys
  system.interface.properties: |
    ui.type=2
    ui.color1=red
    ui.color2=green

In a ConfigMap, the required information can be stored in the data field. We can store values in two ways:

• As individual key pair properties
• In a granular format where they are fragments of a configuration format. (File Like Keys)

How to create ConfigMaps

ConfigMaps and pods go hand in hand as ConfigMaps can be used as environment variables and configuration information in a Kubernetes pod.

In this section, we will have a look at how to create ConfigMaps. Here are some notes before we get started:

• We will be using a windows environment with the windows subsystem for Linux (Ubuntu) as the terminal environment.
• The Docker desktop will be configured to facilitate a Kubernetes environment.
• We will be using the official sample files provided by Kubernetes to demonstrate the functionality of ConfigMap.

Creating ConfigMaps from directories

We can use the following command to create ConfigMap directories.

kubectl create configmap

It will look for appropriate files (regular files) within a specific directory that can be used to create a ConfigMap while ignoring any other file types (hidden files, subdirectories, symlinks, etc.)

First, let’s create a directory using this command:

mkdir configmap-example

Then we’ll download the required sample files to the directory. These files will be used to generate the ConfigMap.

wget https://kubernetes.io/examples/configmap/game.properties -O configmap-example/game.properties

wget https://kubernetes.io/examples/configmap/ui.properties -O configmap-example/ui.properties

Now let’s have a look at the file contents using the following commands.

cat game.properties
cat ui.properties

When creating ConfigMaps using directories, the most important factor is that you have to correctly define the key-value pairs within each file.

After that, let’s create the ConfigMap using the create configmap command.

kubectl create configmap game-config-example --from-file=configmap-example/

This command will package the files within the specified directory and create a ConfigMap file. We can use the kubectl describe command to view the ConfigMap file.

kubectl describe configmaps game-config-example

We can get the ConfigMap in YAML format using the following command.

kubectl get configmaps game-config-example -o yaml

Creating ConfigMaps from files

In the same way we created ConfigMaps using directories, we can also create ConfigMaps using files by using the –from-file parameter to point to a single file in the kubectl create configmap command. So, let’s create a ConfigMap using the game.properties file as shown below.

kubectl create configmap game-config-example-2 --from-file=configmap-example/game.properties

kubectl describe configmap game-config-example-2

We can define multiple –from-file arguments multiple times to create a single ConfigMap file using several different files.

kubectl create configmap game-config-example-2 --from-file=c

Creating ConfigMaps from an environment file

Kubernetes allows users to create ConfigMaps using env files. We can use the –from-env-file argument when defining an env file. This argument can also be used multiple times to define multiple env files.

When using env files, each line should adhere to the <name>=<value> format. Empty lines and comments will be ignored, while quotation marks will be a part of ConfigMap.

cat configmap-example/game-env-file.properties

kubectl create configmap game-config-env-file-example --from-env-file=configmap-example/game-env-file.properties

kubectl get configmap game-config-env-file-example -o yaml

Creating ConfigMap from a file with a predefined key

When creating a ConfigMap, we can use the following format in –from-file argument to define a key name that will overwrite the file name used in the data section.

The following example demonstrates how to define a key while creating a ConfigMap.

kubectl create configmap game-config-key-example --from-file=game-key-example-data=configmap-example/game.properties

kubectl get configmap game-config-key-example -o yaml

Creating ConfigMaps from values

Another way to create ConfigMaps is to provide literal values as parameters in the create configmap command. For this, we can use the –from-literal argument to pass each key pair. This is especially handy when we need to create ConfigMaps on the fly.

kubectl create configmap config-example-values --from-literal=example.value=one --from-literal=example-type=2 --from-literal=example.url="http://example.com"

kubectl get configmap config-example-values -o yaml

Utilizing ConfigMaps in pods

Now we have a basic understanding of how to create ConfigMaps. The next step is to use the created ConfigMaps for creating a Pod. In this section, we will create a simple ConfigMap and use it when creating a pod in Kubernetes.

As the first step, let’s create a file named “app-basic.properties” and include two key-value pairs.

app-basic.properties

system.type="TESTING CONFIGMAP"
system.number=12345

We will create a ConfigMap named “app-basic-configmap” using the above file and the –from-file option.

kubectl create configmap app-basic-configmap --from-file=configmap-example/app-basic.properties

kubectl get configmap app-basic-configmap -o yaml

Finally, let’s create a Pod referencing the newly created ConfigMap. We will be using the following YAML file to create the Pod.

example-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: configmap-example-pod
spec:
  containers:
    - name: configmap-example-busybox
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      envFrom:
        # Load the Complete ConfigMap
        - configMapRef:
            name: app-basic-configmap
  restartPolicy: Never

As you can see from the above example, We are going to load the complete ConfigMap we created to the Kubernetes Pod.

kubectl create -f example-pod.yaml
kubectl get pods

kubectl logs configmap-example-pod | grep system.number

The above result indicates that the ConfigMap “app-basic-configmap” was successfully loaded when creating the Kubernetes Pod.

Mapping keys from ConfigMaps to pods

Another way we can use ConfigMaps is to directly map values from ConfigMaps to the specific environmental variables in the Pod.

In this section, we will create two simple configmap files manually and load and map the values directly to the Kubernetes Pod. There, we will define the ConfigMaps as YAML files and then use the kubectl create command to generate the ConfigMaps.

application-defaults.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: application-configs
  namespace: default
data:
  app.value: "45000"
  app.type: test-application
  app.ui: web

application-logs.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: application-log-configs
  namespace: default
data:
  log_level: WARNING
  log_type: TEXT

kubectl create -f application-defaults.yaml
kubectl create -f application-logs.yaml

example-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: configmap-example-pod
spec:
  containers:
    - name: configmap-example-busybox
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      env:
        - name: APPLICATION_TYPE
          valueFrom:
            configMapKeyRef:
              name: application-configs
              key: app.type
        - name: APPLICATION_UI_TYPE
          valueFrom:
            configMapKeyRef:
              name: application-configs
              key: app.ui
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: application-log-configs
              key: log_level
  restartPolicy: Never

In this configuration, we are mapping environmental variables to values within each ConfigMap.

The following is the basic structure for mapping a value. In the environment section in the YAML file, we define a variable name and  reference the ConfigMap via the “configMapKeyRef” element using the “valueFrom.” Here we will provide:

• The ConfigMap name
• The key where the value should be mapped from

Next, we will create the Pod using the kubectl create command as shown below.

kubectl create -f example-pod.yaml
kubectl get pods

After successfully creating the Pod, we can explore the environment variables as shown below.

kubectl logs configmap-example-pod | grep APPLICATION_TYPE
kubectl logs configmap-example-pod | grep APPLICATION_UI_TYPE
kubectl logs configmap-example-pod | grep LOG_LEVEL

The above results indicate that the values were correctly mapped to environment variables with custom names within the Kubernetes pod.

ConfigMap defined environment variables in pod commands

Another way we can utilize ConfigMap defined environmental variables is by using them in Pod Commands. This can be done for both the command and args elements in a YAML file using the $(VARIABLE_NAME) Kubernetes substitution syntax.

The following code block demonstrates how to use these environment variables in the command element using example-pod.yaml as the base.

apiVersion: v1
kind: Pod
metadata:
  name: configmap-example-pod
spec:
  containers:
    - name: configmap-example-busybox
      image: k8s.gcr.io/busybox
      command: [ "/bin/echo", "Application Type $(APPLICATION_TYPE) - $(APPLICATION_UI_TYPE)" ]
      env:
        - name: APPLICATION_TYPE
          valueFrom:
            configMapKeyRef:
              name: application-configs
              key: app.type
        - name: APPLICATION_UI_TYPE
          valueFrom:
            configMapKeyRef:
              name: application-configs
              key: app.ui
        - name: LOG_LEVEL
          valueFrom:
            configMapKeyRef:
              name: application-log-configs
              key: log_level
  restartPolicy: Never

In this instance, the environmental variables are identified at the execution of the command (at the container start), and they will be directly displayed in the terminal.

Adding ConfigMap data to a volume

Users can consume ConfigMaps by mounting the ConfigMap data into a Kubernetes volume. In the following example, we are mounting the “application-log-config” ConfigMap data to a volume called “config-volume” mounted in “/etc/config” in the container. Then we have configured a command that would list all the files within the /etc/config directory.

apiVersion: v1
kind: Pod
metadata:
  name: configmap-example-volume-pod
spec:
  containers:
    - name: configmap-volume-example-busybox
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "ls /etc/config/" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: application-log-configs
  restartPolicy: Never

Mounted ConfigMaps are automatically updated. Kubelt will periodically check if the mounted ConfigMap is up to date and update the mount accordingly. However, this auto-update mechanism does not apply to volumes mapped as a SubPath volume.

That concludes this tutorial. Explore more Kubernetes topics with the right-hand menu.

ConfigMaps are essential to K8s clusters

In this article, we have learned about Kubernetes ConfigMaps, including multiple ways that can be used to create ConfigMaps and how to utilize ConfigMaps in a Kubernetes Pod. ConfigMaps are an essential part of any Kubernetes cluster, providing a robust method to store simple and frequently accessed application or container data.

Related reading

• BMC DevOps Blog
• Kubernetes Guide, with 20+ tutorials
• The State of Kubernetes Today
• Kubernetes Certifications: How & Why to Get Certified
• The 12-Factor App Methodology Explained
• How & Why To Become a Software Factory