A Kubernetes backup is not one single job, it is the sum of three separate layers: the configuration in Git, the cluster state in etcd, and the data on the volumes. If you only back up one of them, you still lose something in a real disaster: either the cluster state, the actual data, or both at once.
The three layers need different tools and different schedules. An etcd snapshot saves you from a broken control plane, a volume backup saves you from deleted or corrupted data, and Git saves you from a botched change. Only all three together give you a backup you can actually trust when it matters.
I run Kubernetes clusters in production and wrote the Kubernetes Practical Guide published by Rheinwerk Computing. You can find all my Kubernetes articles collected on the Kubernetes page.
What can actually be lost in a Kubernetes cluster
Three things typically get lost when a cluster or a node fails. First, the manifests themselves: Deployments, Services, ConfigMaps, Ingress rules. Second, the current state: which Pod is running where right now, which Secrets exist, which binding between a Persistent Volume and a Persistent Volume Claim currently applies. Third, the actual data inside the volumes, meaning whatever your database or file store holds.
The second point is often underestimated. The current state of the cluster, including every object, every role binding and every internal reference, lives in etcd, the distributed key value store behind the Kubernetes API. A manifest in Git shows you what should be true. etcd shows you what is actually true right now, and that is exactly what can be gone after a total control plane failure, even if your manifests in Git are perfectly intact.
Kubernetes backup in three layers: Git, etcd, and volumes
The table below shows what each layer covers and what you use to back it up.
| Layer |
What it holds |
Lost when |
Backed up with |
| Git |
desired state of your manifests |
human error, misconfiguration |
version control, pull requests |
| etcd |
actual cluster state, secrets, bindings |
control plane failure |
etcd snapshot |
| Volumes |
application data, database contents |
node failure, accidental deletion |
Velero, CSI snapshot, filesystem backup |
If you run your clusters with GitOps, you get the first layer almost for free. When ArgoCD or a similar tool rolls out the desired state from a Git repository, every Deployment and ConfigMap definition is already versioned and recoverable. In my own clusters, that replaces a separate backup of the manifests entirely: I do not need to export running objects from the cluster, I only need to restore the repository and trigger a sync again. That does not change the fact that etcd and volumes still need their own backups, because both hold state nobody writes into Git: runtime data in the volumes, cluster generated values in etcd.
Backing up etcd: a snapshot, not trust
An etcd snapshot is a complete copy of the cluster state at one point in time: every object, every secret, every internal counter. Without that snapshot, a cluster cannot be recovered after a total control plane failure, no matter how clean your manifests look in Git, because a repository holds only what you deliberately put there. Encrypted secrets can absolutely live in Git, through SOPS for instance, and that is how I keep my own. What is not in there are the values the cluster generates itself: automatically created service account tokens, current resource states, and the internal references between objects.
The simplest route bypasses Kubernetes objects entirely: k3s takes scheduled etcd snapshots twice a day by default and can write them straight to an S3 compatible store, configured through server startup parameters (k3s documentation on etcd snapshots). If you run k3s, take that built in route, because it saves you assembling certificate paths, permissions and data directories yourself.
If you want the same flow as a cluster object, you can in principle express it as a CronJob that runs the snapshot command on a control plane node. I deliberately do not show a finished manifest here. Such a job needs access to the data directory including the etcd certificates, a pinned image version instead of a moving tag, and the S3 credentials pulled cleanly from a Secret rather than sitting in plain text in the manifest. Those details differ per installation, and a recovery path that only works on paper is more dangerous than none at all. The built in k3s schedule covers the same purpose without you assembling any of it yourself.
What matters more than taking the snapshot is putting it back, and there a k3s restore does not run across all control plane nodes at once but in a fixed order: stop k3s on all servers, run k3s server --cluster-reset --cluster-reset-restore-path=<path> on the node holding the snapshot, restart that node without the reset flag, and only then, on every other server node, back up and delete the old server/db directory before letting the node rejoin (k3s documentation on etcd snapshots). A restore always affects the entire cluster state and is a procedure for genuine disasters, not something you reach for in daily operations. For clusters not running k3s, the Kubernetes documentation on backing up an etcd cluster describes the general route.
Backing up volumes: Velero, CSI snapshots, or a filesystem backup
An etcd snapshot is not enough for the data inside your Persistent Volumes, because it only saves the description of the volume, not its content. That needs a dedicated tool, and in practice Velero has become the standard.
Velero backs up two things at once: the Kubernetes objects of a namespace, including the PVC definitions, and optionally the data of the volumes attached to them. For the data itself, you have two paths. If your storage driver supports CSI snapshots, Velero uses them natively and creates a snapshot at the storage layer. That snapshot is crash consistent, not automatically application consistent: a database with open transactions also needs a hook that brings it into a clean state before the snapshot runs. And a CSI snapshot usually sits with the same storage provider as the volume, so it does not replace a backup in a second location (Velero documentation on CSI snapshots). Without that support, Velero copies the files through a built in filesystem backup, which is slower but works with almost any storage type.
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: daily-namespace-backup
namespace: velero
spec:
schedule: "0 4 * * *"
template:
includedNamespaces:
- production
snapshotVolumes: true
ttl: 720h0m0s
storageLocation: hetzner-object-storage
This Schedule object creates a backup of the production namespace every day at four in the morning, including volumes, with a 30 day retention. The storageLocation points to a previously configured BackupStorageLocation that targets your S3 compatible store, in my case that is Hetzner Object Storage. Encryption happens at two levels: access permissions on the bucket itself, and optionally server side encryption on the object store.
Practicing a restore before you need one
A backup you have never restored is a guess, not a backup. The one test that actually counts is a full restore into an isolated environment: a separate namespace or, better still, a separate test cluster, where you restore the backup and check whether the application actually starts with the recovered data.
For Velero, that means running a velero restore create against an existing backup, with a --namespace-mappings flag that renames the namespace on restore so you do not accidentally overwrite production. For the etcd snapshot, it means running through the restore at least once on a throwaway cluster, so that in a real emergency you already know which steps come in which order and how long they take. I keep this test restore as a recurring job in a separate test environment, along the lines of Kubernetes Jobs and CronJobs Explained, so the exercise does not get forgotten.
For the schedules themselves, a simple rule of thumb has worked for me: etcd snapshots several times a day, since they are small and fast, volume backups once a day outside peak hours, since they generate more load, and a retention period based on your actual recovery time objective, not on an arbitrary number.
Frequently asked questions
Does Velero back up etcd too?
No. Velero backs up Kubernetes objects and optionally the data of volumes, but not the internal state of etcd. For the control plane you additionally need an etcd snapshot, the two backups complement each other, neither replaces the other.
Do I need Velero if I already use GitOps?
Barely, for the manifests themselves, since with GitOps they already live in a repository and can be rolled out from there again. As soon as your applications store data in Persistent Volumes, a database for example, you still need Velero or a comparable tool, because Git holds no runtime data.
How do I back up a Persistent Volume Claim in Kubernetes?
You do not back up a PVC directly, you back up the volume behind it. If your storage driver supports CSI snapshots, you can use them to create a snapshot of the volume at the storage layer, which Velero triggers automatically. That still is not application consistent, which is what a database needs a matching hook for. Without CSI support, a filesystem backup remains the only path.
How often should I back up my cluster?
That depends on your rate of change, not on a general rule. As a starting point from my own practice: etcd several times a day, since the snapshot is small and fast, volumes at least once a day. What matters more than frequency is that you regularly test whether a restore actually succeeds.
What Kubernetes backup solutions exist besides Velero?
Velero is the most common approach, because it combines objects and volumes in one tool. Besides that, you can drive CSI snapshots directly through your storage driver without Velero in between, or use filesystem tools like Kopia or Restic for pure file backups. For the etcd side, the native snapshot command remains the right choice in every case.
Where to go next
Before you worry about backups, it is worth looking at where your cluster actually runs: Kubernetes On-Premises or Cloud? covers which backup responsibilities stay with you and which a provider takes over. And if you want to revisit the basics of volumes, they are covered in Kubernetes Persistent Volumes and PVCs.
My suggestion for this week: if you have not already, set up an automated etcd snapshot, create a Velero schedule for your most important namespace, and schedule your first test restore, ideally before you actually need it.
In full detail with all examples on Persistent Volumes and how they work with StatefulSets, this is covered in chapter 6 of my Kubernetes Practical Guide (Rheinwerk Computing).
