With over 12 years of experience in DevOps tools and practices, I have many insights and tips to share with the community.

So, I started a new initiative to publish a tip, guide, or latest technology trend report every weekday.

Subscribe Here: dailydevopstips.com

If you subscribe, you'll receive tips and guides on DevOps, MLOps, Kubernetes, and more, delivered straight to your inbox!

Please note: I’ll be discontinuing the Substack subscription soon, so make sure to follow us on the new platform!

I’m also moving our newsletter to a new domain: blog.techiescamp.com.

As a valued subscriber, you've been automatically added to the new newsletter.

Check out the first issue here: Issue #1: Unlocking Jenkins on Kubernetes.

Please note, I’ll be discontinuing the old subscription soon, so make sure to follow us on the new platform!

It is a Highly efficient sandboxed VM in the Linux kernel, making the Linux kernel programmable at native execution speed.

It helps you extend Kernel capabilities without actually changing the kernel source code.

For example, where there is a read system call event, you can run a BPF program.

Following are the primary use cases for eBPF.

1. Security

2. Networking Tracing

3. Profiling

4. Observability

5. Monitoring

Companies like Google, Facebook, and Netflix have already implemented eBPF for various use cases for their production system.

When it comes to Kubernetes, the open source Network plugin Cilium uses BPF for Kubernetes networking.

Also, Linux kernel development community announced bpfilter, that will replac in-kernel iptables implementation with high-performance Linux-based BPF network filtering.

BPF Learning resources:

[1]. Getting Started With eBPF

[2]. How to Make Linux Microservice-Aware with Cilium and eBPF — [[Video]

[3]. Brendan Gregg, Senior Performance Engineer, Netflix Performance and OS Team, explores the past, present and future of BPF, and describes use cases.

[4]. BPF Comes to Firewall

[5]. How companies like Facebook and Google use BPF to patch 0-day exploits

[6] Cloudflare Production ready eBPF

[7]. Replacing iptables with eBPF in Kubernetes with Cilium

[8]. Cilium Kubernetes Network Plugin

[9]. eBPF: exploring use case of BPF kernel infrastructure

[10]. BPF - the forgotten bytecode

Linux Foundation certification cost increases from August 1.

You can make use of the current sysadmin sale to save up to $200.

1. Use code SYSADMINDAY100 at https://kube.promo/k8s to save $100 on CKA, CKAD, CKS, KCNA, LFCS and other certifications.

2. Use code SYSADMINDAY200 at https://kube.promo/bootcamps to save $200 on bootcamps.

3. Save $200 on certification powers bundles with code SYSADMINDAY200 at https://kube.promo/sysadmin.

Take advantage of this Sysadmin sale to save money before the price increase. You get one year validity to appear for the exams.

Also, share this offer with friends who are planning to take the certification before the price increase,

There are also discounts on boot camp and power bundles (Courses + certifications). All details are available on the offer page.

The offer expires on July 29th.

1. After purchase you have 12 months to take the exams. Which is more than enough to prepare of the exam.

2. You get a free retake. (2 attempts in total)

3. Also, you get free access to killer.sh practice environment, which is more than enough to clear the exam.

If you are learning Kubernetes, checkout 30+ comprehensive Kubernetes tutorials

In cloud environments, When we create virtual networks, firewalls, route tables, etc.

It happens in a matter of seconds to minutes.

It's not any magic; there is hardware sitting at some place in the data center. All those complex configurations are abstracted by virtualization technologies, Sofware-defined networking, and cloud providers proprietary software.

Let's take the example of Google cloud.

When it comes to Google Cloud, its cloud network virtualization stack using Andromeda SDN.

It is the orchestration point for provisioning, configuring, and managing virtual networks and in-network packet processing.

All the cloud platform networking services with high performance, availability, isolation, and security are delivered using Andromeda.

For example, when you create firewalls, routing, and forwarding rules on google cloud, it uses the backend Andromeda APIs and infrastructure.

Google first announced Andromeda in 2014.

In 2019, it released Andromeda 2.2 for high-throughput VMs.

Few more information on Google Cloud Data Cernter.

1. Google cloud uses Clos topology for its network.

2. Centralized software control stack to manage all switches within the data center.

3. Google doesn't rely on standard internet protocols. Instead, they build their custom software and hardware tailored to the data center.

Sources & References

[1]. Google Cloud Platform's latest networking stack

[2]. How Andromeda 2.2 enables high-throughput VMs

[3]. A look inside Google’s Data Center Networks

[4]. A Decade of Clos Topologies and Centralized Control in Google’s Datacenter Network

So, where are we with the IPV6 rollout?

IPv6 rollout officially started on 6 June 2012

As of July, global IPV6 adoption stands at 37%.

Few other info you should know.

1. IPv6 addresses are written using hexadecimal instead of a dotted decimal in IPv4.

2. The total number of IPV6 addresses is 340 trillion trillion trillion addresses.

3. IPV6 fc00::/7 block is reserved for the private network. It is known as a Unique local address.

Google has a dedicated web page that keeps track of IPV6 adoption.

IVP6 Status Tracker: https://www.google.com/intl/en/ipv6/statistics.html

The following image shows the difference between IpV4 & IPV6 addressing.

Getting Started With IPV6

You can get started with IPV6 using the following guides.

1. IPv6 - internetsociety offical Guides

2. Understanding IPv4 and IPv6 Protocol Family

3. IPv6: Introduction to the Protocol - Pluralsight

4. IPv6 Subnetting

5. IPV6 Subnet Calcualator

Here is some key information you should know about EKS anywhere

1. It uses the same distribution of Kubernetes that powers EKS on AWS. Distribution of the same open-source Kubernetes and dependencies deployed by Amazon EKS.

2. It supports VMware vSphere and bare metal server deployments.

3. Amazon EKS Anywhere provides Bottlerocket, a Linux-based open-source operating system built by AWS, as the default node operating system, with Ubuntu as a node OS alternative.

4. It is available as open-source software that you can freely download, install on your existing hardware, and run in your own data centers. 

5. You can provision the cluster using the eksctl utility. 

6. You can manage EKS anywhere cluster on the AWS EKS console by registering it using the EKS Anywhere provider and the EKS connector agent that runs on your on-premises cluster. This lets you visualize all workloads as a single pane of glass for clusters.

7. EKS Anywhere will not manage the underlying cluster infrastructure or your cluster control plane. It is a shared responsibility. AWS provides only tooling.

8. There is no charge for using AWS anywhere. However, charges apply if you opt for enterprise support for anywhere clusters. Ie, $24,000 per cluster per year.

Note: EKS console also supports connecting GKE, AKS, Openshift, self-hosted ec2 k8s cluster, etc to the EKS console to visualize workloads using relevant provider plugins.

Further Learning Resources

[1]. EKS connector

[2]. EKS FAQs

[3]. Getting Started With EKS anywhere

[4]. EKS anywhere AWS re-invent video

[5]. Amazon EKS Anywhere Deep Dive

Learning new DevOps tools is always fun.

DevOps Engineers should keep learning new tools.

When you start learning a new DevOps tool, the first few days could be really demotivating and you might feel like you are not getting anywhere. Don't worry, it's not just you, most of us feel the same way.

Somehow I made peace with the demotivation and have a process to learn new technology.

So here is how I learn a new tool or technology.

1. First, I will go through the official documentation with the getting started guides.

2. I browse through the documentation to understand all the key concepts (Architecture, design, etc.). Then, if it is a complex tool, I try to read a book on that tool.

3. Next, I start watching the official videos of the tool. Most tools will have conference presentations on youtube. From these videos, you will learn to use the tool and the use cases other companies are solving. For example, AWS Reinvent videos, Kubecon videos, etc.

4. Then, I pick up a use case and do hands-on. It could be a use case available online or I design my own. The real learning starts here.

5. Once I gain adequate hands-on knowledge, I start reading blogs. Mostly blogs on experiences shared by engineering teams on using the tool in production. It helps me understand the best practices and learnings from others.

6. Then, I start digging into conversations on Reddit, Stackoverflow, etc. Interestingly these forums are a gold mine of information.

7. Also, I call up people in my professional circle to understand how they use the tool in their projects. If the tool is relatively new, there is less chance of getting information. But you can share your knowledge with them.

8. Finally, I write a getting started tutorial on my DevOps blog with my learnings to retain the knowledge. It acts as a reference for me, and it might help others as well.

Also, teaching someone what you have learned is a great way to retain the knowledge. Surprisingly you will get more ideas when someone asks you questions about it.

So if you are working in a team, always share your knowledge. It is also a good trait of a DevOps engineer.

Note: The above process works well for me and it takes some patience to go through the steps I mentioned. If you are on a tight project deadline with new technology, its better to follow blogs and get things done first :)

If you have got any tips for learning, Do let me know in the comments.

Also, if you are looking for a DevOps engineer role, do look at my comprehensive guide on becoming a devops engineer.

Most accessible Kubernetes API servers are found in the United States – 201,348 (nearly 53%).

As per the report, 

"Public API access does not mean that these instances are fully open or vulnerable to an attack, it is likely that this level of access was not intended, and these instances are an unnecessarily exposed attack surface. They also allow for information leakage on version and builds."

The image from shadowserver shows the exposed cluster from all the countries.

References & Further Reading

[1]. Read the full report by shadowserver here

[2] API Security Issues Hinder Application Delivery

It is a technical practice that follows the DevOps philosophy.

You define the desired infrastructure configs in git, and a tool or an operator software that sits in your infrastructure watches for any changes in git. If it detects any change in git, it applies those changes to the infrastructure and brings it to the desired state.

Also, the operator software continuously monitors the state of the infrastructure. If the infrastructure deviates from the desired state (let’s say due to a manual change), the operator software ensures the infrastructure comes back to the desired state.

For example, in git, the infra config says, for autoscaling, the minimum instance count is 3, and the max is 9. The operator software deploys the autoscaling group with the values in Git.

Assume someone does a manual change, now the autoscaling min and max count is 4 and 12 now. Since the operator software continuously monitors the infrastructure, it identifies the configuration drift as compared to the git config.

So it rollbacks the manual changes to match the desired state in git.

Now, when it comes to Gitops, you mostly hear implementations around Kubernetes using Kubernetes operators.

But the GitOps workflow can be implemented using any tool that supports the workflow.

At a high level, GitOps aims to bring the following.

1. Git as the source of truth

2. Developer-centric infra workflows

3. Good traceability of infra changes

4. Consistency and Standardization

5. Security.

6. No manual changes.

With Gitops workflow, infrastructure engineers/developers can focus more on engineering and innovation than on infrastructure management and maintenance.

Check comments for further resources to understand GitOps.

So what does immutable infrastructure really mean?

Traditionally to host an application, one would deploy a server and configure applications.

Then comes server patching, application upgrades, server configurations changes, etc. These actions would be performed on the same server manually or using tools.

Immutable — Like its literal meaning, Immutable infrastructure is a concept where you don’t make any changes to the server after deploying it.

If you want to make any changes, the existing servers should be destroyed and replaced with a new one.

A change could be patching, application upgrade, server configurations change, etc.

You can follow the immutable infrastructure model for most modern applications, including database clusters.

In an immutable model, standard best practices should be followed in terms of configurations.

For example, externalizing commonly changed configurations using config store or a service discovery tool. A classic example would be the Nginx upstream configuration.

This way, you don’t have to bounce off a server for minor changes and configurations.

When it comes to CI/CD in VM environments, a VM image would be a deployable artifact when following an immutable infrastructure model.

For example, once CI is done, you can bake the app in a VM image (AWS AMI) with tools like packer and use it to deploy in the relevant environments.

If you are aware of containers, it is the best example of immutable infrastructure. Any change to a container results in a rebuild except for externalized configurations.

You would choose a base image of your choice and start playing around with it manually or through automation.

In actual project environments, it is not that straightforward. I want to shed some light on how it happens in a real project environment.

So, here is a list of generic VM life cycle management steps followed in secured project environments.

Note: The list is to give you an overall picture of VM lifecycle management. It differs in each organization.

1. In a secured, compliant environment, you are not allowed to use the base images provided by the cloud provider.- Every organization creates a base image with standard security tools (agents), LDAP configurations, etc. (It changes as per each organization’s security policy).

2. Usually, this image is created and maintained by a central platform or security team.- The approved and certified base image will be shared with all the teams in the organizations.

3. It could be a single cloud account or shared with multiple child accounts within the organizations.-

4. Then each team can create their own images with applications on top of the approved base image. (Tools like packer is used here)-

5. The new image created by the teams will be deployed in production.- Now, the base images get new updates and patches. So a new version of the base image is released and notified to all project teams by the platform or the enterprise security team.-

6. Every organization has a patching lifecycle. Meaning security teams set guidelines on applying the updates and patcher to VMS. For example, it could be one month or once in three months.-

7. Patching could be “in-place,” meaning patching the existing instance, or it could be immutable — meaning replacing the existing one with a new image.-

8. Based on the patching lifecycle, every team will update the existing application images with the new base image and deploy it to production irrespective of whether the application code has changed.

Subscribe now

To assign a pod a certain priority, you need a priority class.

You can set a priority for a Pod using the PriorityClass object (non-namespaced) with a Value.

The value determines the priority. It can be 1,000,000,000 (one billion) or lower. Larger the number, the higher the priority.

Also, there are two default high-priority classes set by Kubernetes

1. system-node-critical: This class has a value of 2000001000. Pods like etcd, kube-apiserver, and Controller manager use this priority class.

2. system-cluster-critical: This class has a value of 2000000000. Addon Pods like coredns, calico controller, metrics server, etc use this Priority class.

How Does Pod Priority Work?

Here is how pod priority work,

1. If a pod is deployed with PriorityClassName, the priority admission controller gets the priority value using the PriorityClassName value.

2. If there are many pods in the scheduling queue, the scheduler arranges the scheduling order based on priority. Meaning, the scheduler places the high priority pod ahead of low priority pods

3. Now, if there are no nodes available with resources to accommodate a higher priority pod, the preemption logic kicks in.

4. The scheduler preempts (evicts) low priority pod from a node where it can schedule the higher priority pod. The evicted pod gets a graceful default termination time of 30 seconds. If pods have terminationGracePeriodSeconds set for preStop container Lifecycle Hooks, it overrides the default 30 seconds.

5. However, if for some reason, the scheduling requirements are not met, the scheduler goes ahead with scheduling the lower priority pods.

Note: This is an excerpt from an article published on devopscube.com. For a detailed example please visit → Pod PriorityClass Explained

But the common concern in using a distroless or minimal image is that,

How do I take an exec session to troubleshoot if something goes wrong in the application? Because these images won’t even have a shell or any utilities required for troubleshooting.

Here is where ephemeral containers come in to picture.

An ephemeral container is a concept of adding a container in an exiting pod for debugging purposes. Let’s say you have a pod running on a minimal base image with just the application binaries and dependencies. Something went wrong, and you need to debug it.

Since it is a stripped-down minimal base image without a shell, you cannot perform a “kubectl exec” command. Here, you can add a debug container to an existing pod in real-time. This debug container would have all the required utilities to debug the application. (shell, curl, custom utilities, etc)

For example, let’s say you have a running pod named frontend, and you have an image with debug utilities called debug-image. The following command will add the debug-image container to the running frontend pod and take an exec session for debugging.

kubectl debug -it pods/frontend — image=debug-image

You can also debug a pod in CrashLoopBackOff state.

Note: Ephemeral Container was introduced in k8s v1.16 as an alpha feature, and now it is in beta as of 1.23. It is part of the Kubernetes core API.

Further reading:

[1] Introducing Ephemeral Containers

[2] K8s documentation - Ephemeral Containers

No matter how many third-party tools we use to manage Kubernetes, Kubectl is something you cannot live without out in the Kubernetes ecosystem.

There are compliant environments where you cannot use third-party tools except kubectl for a user to interact with the Kubernetes cluster.

Have you ever wanted to do more with kubectl?

For example, something like,

kubectl backup cluster

Is it is even possible?

Kubectl Plugins

Here is where kubectl plugins come in to picture. Kubectl plugins help you to extend the kubectl core functionalities with custom functionalities that you need.

You can use any programing language or scripts that support the command line to write a Kubectl plugin.

Creating a plugin is very easy. Try creating one for fun!

Check out the Documentation

Before you start writing your own plugin, check out all the Kubectl community plugins. There are more than 100+ open source kubectl plugins available for different use cases.

To make it even better, there is a plugin manager called krew to manage the community plugins on your workstation.

For example, to install a plugin, all you have to do is

kubectl krew install <plugin-name>

Krew Plugin Manager

Krew Installation

Here is why it's very important.

When deploying a pod, kubernetes assigns a QoS class to pods based on the requests and limit parameters.

Let's understand Kubernetes Pod Quality of service. (QoS).

Kubernetes pod scheduling is based on the request value to ensure the node has the resources to run the pod.

However, a node can be overcommitted if pods try to utilize all its limit ranges more than the node's capacity.

Overcommitment = sum of resource request/limits > node capacity

When pods on the node try to utilize resources that are not available on the node, kubernetes uses the QoS class to determine which pod to kill first.

Types of Pod QoS Class

Following are the three types of Pod QoS class

👉 Best effort

Your pod gets the best-effort class if you do not specify any CPU/Memory requests and limits. Best-effort pods are low-priority pods. The best-effort pods get killed first if the node runs out of resources.

👉 Burstable

If you set the request lower than the limit, the pod gets burstable class. If the node runs out of resources, burstable pods get killed if no best effort pods are available.

👉 Guaranteed

The pod gets a guaranteed class if the request and limit values are the same. It is considered the highest priority pod and gets killed if there are no best-effort or burstable pods.

🧵 Pod QoS FAQ’s

Following are the discussion that happened on LinkedIn for this topic

Question 01: While the Guaranteed Pod QoS offers the highest reliability, would the design not lose out on Kubernetes' scalability benefits? To protect against pod eviction & failures, there would be a need to keep the resource request values high which means Kubernetes reserves more than needed during non-peak times.

Answer: Well, burstable is definitely out of the picture in production. But you can consider the other two classes as part of the capacity planning based on your workload types. You need to consider pod priority class as well. Here is a blog that might help answer your queries. https://sysdig.com/blog/kubernetes-resource-limits/

Question 02: What happens when there are 2 pods with guaranteed QoS and there are no best effort or burstable pods and we need resources back for cluster? Which one of the two will be killed? Am just curious to know the algorithm used in this case

Answer: Pods get ranked on priority for termination. If multiple pods have the same priority, then pods that are most over the request are terminated first.