Kubernetes has fundamentally changed how we build and run our applications. But it hasn't changed one uncomfortable truth - things still break. Clusters go down, data gets corrupted, or even a simple human error - someone runs the wrong command at the wrong time. When that happens, how prepared you are with your recovery plan or backups strategy will be the difference between a war story and a disaster.
The tricky part is Kubernetes backup is not as straightforward as it seems. People find out the hard way that their backup strategy has holes in it - a bunch of YAML files is not a backup, an etcd snapshot is not always enough and a CSI snapshot is not the same thing as a backup - it often remains tied to the same storage infrastructure and lacks the broader application and Kubernetes resource context needed for reliable restores.
In this series we will be covering everything you need to know about backup and restore in Kubernetes - from the basics to real production experience. We go through what backup means in Kubernetes, how to pick the right tool, key strategies and policies, the operational side of running backups at scale, why a successful backup doesn't guarantee a successful restore, and finally how we do it in Vayu Kubernetes Services.
Whether you're setting this up for the first time or taking a hard look at what you already have, there's something here for you.
Kicking off this series with its first entry "Foundations" which will help us segue into rest of the series well.
Backup is not a new concept, and neither is the complexity around it. Every system that holds data of any value needs a way to recover it when something goes wrong. Distributed systems have always made backup harder with multiple components, multiple data stores, and the need for consistency across all of them. Kubernetes is no different, but it brings its own specific challenges that traditional backup approaches are not designed for.
In a traditional setup like a VM or a physical server, backup is relatively straightforward. You snapshot the disk, copy the files, and you're done. The system is largely static and what you backed up is what you restore.
Kubernetes is different. An application running on Kubernetes is not a single thing sitting in a single place. It is a collection of resources including Deployments, Services, ConfigMaps, Secrets, PersistentVolumeClaims, and more that are spread across a cluster. Pods come and go, resources are created and deleted dynamically, and the cluster itself is always doing something. So, backing up a Kubernetes application is not just about copying data. It is about capturing the complete state of a running system, its configuration, its data, its dependencies, and the relationships between all of them, in a way that can be restored into a working application.
This dynamic and declarative nature is what makes Kubernetes backup both powerful and deceptively complex. Unlike traditional systems, Kubernetes continuously works to match the actual state of the cluster to the desired state declared in its objects. A good backup must therefore preserve enough information for the reconciliation loops to resume correctly after a restore.
These three terms get used interchangeably but they solve different problems and have very different cost and recovery time implications.
High Availability is about redundancy within the same environment. If a node goes down, another takes over. The system keeps running with little to no downtime. HA does not protect you from data loss or a full cluster failure, it just reduces the impact of individual component failures.
Disaster Recovery is about replication to a secondary site or region. The data is continuously or near-continuously mirrored to another location. When the primary goes down, you failover to the secondary. Downtime is minimal (minutes at most) because the data is already there waiting.
Backup and Restore is different from both. It is a point-in-time copy of your data and configuration stored somewhere safe. When you need it, you retrieve it, restore it to a new environment, and bring the application back up. This usually takes time, potentially hours depending on data size and complexity. There is no live replication, no instant failover. The tradeoff is cost - backup and restore is significantly cheaper than maintaining a live DR environment.
Understanding these distinctions helps set realistic expectations. Many teams start with HA, assume they have protection, and only later discover the gaps when facing data corruption or widespread configuration errors.
There are two broad categories of backup in Kubernetes and understanding the difference matters because they serve different purposes and are used in different recovery scenarios.
A cluster backup captures the entire state of a Kubernetes cluster, primarily through etcd, which is the key-value store that Kubernetes uses to store all cluster state. Every resource object, every configuration, every piece of metadata lives in etcd. Backing it up gives you a snapshot of the entire cluster at a point in time.
Cluster backups are useful for full cluster recovery scenarios. If a cluster is lost entirely, an etcd backup can help bring it back. However, they come with important limitations. etcd snapshots are tightly coupled to the Kubernetes version and the specific cluster configuration. Restoring an etcd backup to a different cluster or even a slightly different version often requires careful preparation and is not always straightforward. Additionally, etcd backups do not capture persistent data stored outside etcd, which constitutes most of your application data on PersistentVolumes.
An application backup focuses on individual workloads and everything they depend on. Rather than backing up the entire cluster, you select specific applications, their manifests, their data, their configuration, in a way that is portable and recoverable independent of the original cluster.
This is the more flexible and operationally useful approach for most day-to-day scenarios. It supports use cases like restoring a single application after an incident, migrating workloads across clusters, or recovering in a different region entirely.
Most mature backup strategies use both. Cluster backups via etcd for catastrophic recovery and application backups for granular, operational resilience.
An application in Kubernetes is rarely just a Deployment. It is a collection of interdependent resources, each playing a specific role in how the application runs. A complete backup needs to capture all of them, not just the parts that are easy to see.
Kubernetes Manifests
Manifests are the core workload definitions that describe how an application is structured, how it is exposed, and how it scales. This includes Deployments, StatefulSets, Services, Ingresses, and HorizontalPodAutoscalers among others. They form the skeleton of your application and everything else is built on top of them.
Persistent Data
Persistent data is the actual content stored in PersistentVolumes, including your database records, uploaded files, logs, or anything that an application writes to disk over time. Unlike manifests, which are largely static or version-controlled, this is the living, changing part of the application. It is often the most critical element to recover correctly.
Secrets and ConfigMaps
Secrets and ConfigMaps hold the configuration and credentials your application needs to function. Secrets carry sensitive values like API keys and database passwords, while ConfigMaps hold environment-specific values or settings. Together they define the runtime context. Restoring without them can render an otherwise complete workload unusable.
Custom Resource Definitions and Custom Resources
CRDs extend what Kubernetes understands natively. Modern applications increasingly rely on operators and custom resources. Without backing up both the CRD definitions and the associated Custom Resources, restored applications may fail because the Kubernetes API no longer recognizes the resource types.
Cluster-Scoped Resources
Not everything an application depends on lives inside its namespace. ClusterRoles, ClusterRoleBindings, MutatingWebhookConfigurations or ValidatingWebhookConfigurations, and certain CRDs exist at the cluster level. Many backup tools default to namespace-scoped selection, which means these resources can silently fall outside the backup. Their absence during restore is not always obvious immediately but can cause permission errors or missing functionality.
Backup tools generally support granularity through labels, namespaces, or include/exclude filters. This allows teams to back up at the cluster level, per namespace, or per application using label selectors. Choosing the right granularity is one of the first important decisions in designing a backup strategy.
Beyond these core components, certain supporting elements often fall outside default backup scopes yet matter during restore. Examples include NetworkPolicies that control traffic flow and admission control policies such as those from OPA Gatekeeper. Restoring without them may not cause immediate breakage, but the recovered application will behave differently from its pre-incident state.
If it's in Git, it's safe
This is one of the most common assumptions in teams that have adopted GitOps. And it makes sense on the surface. If all your manifests are version controlled, you can always redeploy. But there is an important difference between reproducibility and recoverability. Reproducibility means you can redeploy an application. Recoverability means you can restore it, with its data, its runtime state, its Secrets, and everything it depends on. Git gives you the former and does nothing about the latter. A database that was wiped cannot be restored from a Deployment manifest.
Backing up etcd is backing up everything
etcd is the brain of a Kubernetes cluster. It holds the entire cluster state, every resource definition, every configuration. But the brain is not the whole body. Your actual application data including database contents, files written to PersistentVolumes, or anything your application has stored on disk, none of that lives in etcd. An etcd backup will restore your cluster topology and resource definitions, but if your PersistentVolumes are gone, your data is gone with them. etcd backup is necessary but it is not sufficient.
Stateless applications do not need backup
The label "stateless" gets used loosely and that looseness creates risk. In practice, very few applications running in a modern cluster are truly stateless. An application might be mostly stateless but write logs to a volume, cache data to a PVC, or depend on a sidecar that maintains its own state. And even if an application genuinely has no persistent data, it still has Secrets that hold credentials, ConfigMaps that define its configuration, and potentially cluster-scoped dependencies like ClusterRoles that govern what it can access. None of these are data in the traditional sense but lose them and you cannot restore the application into the same working state it was in before. Calling something stateless and skipping its backup entirely is a habit worth examining carefully.
A CSI snapshot is as good as a backup
A CSI snapshot is fast, convenient, and gives you a point-in-time copy of a PersistentVolume. But it is not equivalent to a full backup. While the underlying snapshot data is often durable in cloud environments, the VolumeSnapshot Custom Resource lives inside the Kubernetes cluster and can be lost if the control plane fails. CSI snapshots are typically crash-consistent only and also, they do not capture the surrounding Kubernetes objects (Deployments, Services, Secrets, etc.) which are needed for a complete and portable restore. A true backup must exist independent of the source infrastructure and capture the full application context.
While this gives us an overall understanding of what backups can mean for a Kubernetes cluster, the story doesn't end here. In the next few entries in this blog series, we will cover how the right backup tool can be identified and what should be the optimised backup and restore policies to get precise RTO and RPO for applications hosted on Kubernetes.
Start your backup and restore journey on Vayu Cloud's Kubernetes-as-a-Service, where you can deploy your applications and configure their backup policies through Backup Manager.