Our Kubernetes Cluster

An overview of our adventures with Kubernetes

Modified March 2024

| What is Kubernetes?

Kubernetes is a piece of software that orchestrates containers. According to their main website:

Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.

There are many components that make up Kubernetes and many of them can be changed out with other components that implement the same interfaces. This means that while someone might say they have Kubernetes experience, the storage, networking, and more might be running differently underneath. This typically doesn't change the way a user or administrator interacts with a Kubernetes system, often referred to as a cluster due to spanning multiple nodes, but debugging or maintaining the system might be vastly different.

| Why use Kubernetes?

Now that we know Kubernetes orchestration containers, why would we want to use it? There are other options like Docker Swarm, good old bash scripts, systemd, and more.

Kubernetes exposes a single API that can manage the lifecycle of containers, the connectivity of containers, and more. This makes managing complex systems composed of various services easier than other methods of container orchestration. The abstract nature of Kubernetes allows for developers to focus on developing code while having infrastructure engineers focus on the infrastructure.

| How are we using Kubernetes?

We are using Kubernetes as the backend for most hosted applications. There are a few exceptions, but mostly what we interact with is Kubernetes. This allows us to transparently scale resources to maximize our resource utilization efficiency. We are also using Kubernetes to build cloud native applications that our customers can use.

| What are our future plans?

We are going to be improving our current infrastructure by rebuilding the cluster. We have found one of the best ways to truly learn how a technology works is dive in and learn as you go, even if suboptimal design choices are made originally. Our current cluster is stable and working well, there are some adjustments that would improve it greatly. There are a few adjustments we would like to make across several categories:

Observability
  • Monitoring is important and necessary. We are always striving to improve our current monitoring techniques and we will be improving by adding in the monitoring earlier and more thoroughly. We have seen some shortcomings of our current solutions, implemented patches to add visibility, but would like to create a more cohesive solution.
Maintainability
  • We need a more reliable way to deploy services. Currently we are using scripts that we built to gain a better understanding of the system. Now, we have matured to being able to use a continuous delivery (CD) system like ArgoCD. This will enable us to deploy services more smoothly and in a cloud native way.
  • We have had to start version pinning containers to ensure they wouldn't upgrade accidentally. We had an incident where a minor service wasn’t pinned and it wasn’t a showstopper, but it was annoying. By ensuring our systems are completely reproducible through a CD system, we can either spin up additional clusters for other clients and/or project ideas, or have the ability to reconstitute after an unfortunate incident.
Security
  • Security is not a once and done checklist item. We are looking forward to adding new runtimes like gVisor and WASMEdge to our cluster. This will enable us to provide more isolation between services and applications we use.
  • We will be implementing a service mesh to enforce network policies between services and applications.

As we continue to engineer new systems, services, and software for our clients, we are always looking for challenges that enable us to grow. Kubernetes has been a fantastic infrastructure addition and we look forward to the continued adventure of learning how we can leverage this technology more efficiently.