Kubernetes architecture has two halves: the control plane, which steers the cluster, and the worker nodes, where your containers actually run. The control plane hosts kube-apiserver, etcd, kube-scheduler and kube-controller-manager; every node runs kubelet, kube-proxy and a container runtime. Everything talks through the API server, and only the API server is allowed to write to etcd.
I have been running production clusters for years, currently mostly k3s on Hetzner with three control plane nodes spread across three data centers. Architecture has its own chapter in my Kubernetes practical guide, published by Rheinwerk Computing in 2024. What follows is the short version you need to place a cluster problem with the right component. Everything I write about Kubernetes is collected on the Kubernetes page.
If you are completely new to this, read Kubernetes explained simply first. From here on, containers and clusters are taken as known.
Kubernetes cluster architecture: control plane and worker nodes
A node is a server that belongs to the cluster, whether virtual machine, bare metal or Raspberry Pi. Kubernetes gives these servers one of two roles. Control plane nodes carry the steering components: they store the desired state, accept your requests and decide where things run. Worker nodes execute: they start containers, mount volumes, hand secrets through and report back.
The point I always make first with newcomers: there is no management logic on a worker node. Workers are interchangeable. If one dies, Kubernetes rebuilds its containers elsewhere, because the desired state does not live on the node but in etcd on the control plane. That is why a cluster survives losing a server.
| Component |
Runs on |
Job |
| kube-apiserver |
control plane |
The single entry point: accepts, validates and stores requests |
| etcd |
control plane |
Key-value store holding the entire cluster state |
| kube-scheduler |
control plane |
Picks a suitable node for every new pod |
| kube-controller-manager |
control plane |
A bundle of controllers that reconcile desired and actual state |
| cloud-controller-manager |
control plane |
Connects load balancers, volumes and nodes of a cloud provider |
| kubelet |
every node |
Starts and watches the containers the API server assigns to it |
| kube-proxy |
every node |
Makes sure network packets reach the right pod |
| container runtime |
every node |
Actually runs the containers, for example containerd |
The official documentation maintains the full list under Kubernetes Components. The table above is the map for everything that follows.
kube-apiserver: the only door into the cluster
Every request to Kubernetes lands at the kube-apiserver. When you type kubectl apply, kubectl talks to it. When the scheduler places a pod, that goes through it. When a kubelet reports that a container has started, the API server takes the report. There is no second path.
The API server does more than forward. It checks whether a manifest is valid, whether the caller has the required permissions and whether quotas or rate limits apply. Only then does it write the object to etcd. It is the only component with a connection to etcd at all; everything else sees the data through its lens.
This design is called hub-and-spoke. A new component plugs into the hub instead of having to know every other one, and nothing happens in the cluster without the API server knowing. Watch requests build on that, the mechanism controllers and tools like ArgoCD use to react to changes.
etcd: the cluster's memory, and why backups are mandatory
etcd is a distributed key-value store that also exists outside Kubernetes. Inside Kubernetes it holds every object you have ever created: deployments, services, ConfigMaps, secrets, the status of every pod. As long as etcd is intact, the cluster can rebuild itself from almost any situation, because the complete desired state is stored there.
For high availability, etcd uses the Raft consensus algorithm with a quorum: a majority of members has to agree before a write counts. That is why you always run an odd number. With three, one can fail; with five, two can fail. Two members gain you nothing, because if one goes down the other no longer has a majority.
The practical consequence: etcd is the only component whose loss really hurts. Everything else can be restarted or reinstalled. A regular snapshot therefore belongs to every cluster that carries production. On k3s with embedded etcd, it is a single command:
k3s etcd-snapshot save --name before-upgrade
k3s writes the snapshot to /var/lib/rancher/k3s/server/db/snapshots/ and can upload it straight to an S3 bucket. In my setup that runs on a schedule, into a bucket outside the cluster. A backup on the same server as the database is not a backup. For kubeadm clusters, the path via etcdctl snapshot save is in the official etcd guide. Practice the restore in a test cluster before you have to do it for real.
kube-scheduler and kube-controller-manager: who decides what
The kube-scheduler answers exactly one question: which node should a new pod run on? For that it knows each node's capacity, the resources already reserved and every placement rule you set, such as node affinity, taints and tolerations. It assigns the pod to a node and records that through the API server. That is all it does; it never starts a container. If a pod sits in Pending, the scheduler usually found no fitting node, and kubectl describe pod says why in the events.
The kube-controller-manager is one program with many controllers inside. Each controller runs a loop: read the actual state, compare it with the desired state from etcd, fix the difference. The ReplicaSet controller keeps the requested number of pods running. The endpoints controller connects services to the pods behind them. The service account controller creates default accounts in new namespaces.
The most important one for operations is the node controller. Every kubelet renews a lease object in the kube-node-lease namespace, a heartbeat. If that heartbeat stops for a configurable period, the node controller marks the node NotReady, and after a further period its pods are released for rescheduling. The Nodes documentation has the details. When a node fails in one of my clusters, I see the missing heartbeat in Prometheus first and the pods coming back up elsewhere minutes later. That is the whole mechanism behind self-healing.
kubelet, kube-proxy and the container runtime on every node
The kubelet is the agent on each node. It registers the node with the API server, fetches the pods belonging on this node and makes sure they run. For that it talks to the container runtime, pulls images, starts containers, mounts volumes and places secrets and ConfigMaps as files or environment variables. It watches the containers through probes and reports their status back. The kubelet runs on control plane nodes too, since those components are containers as well.
kube-proxy takes care of networking to the pods. A service usually has a stable virtual IP, but the pods behind it change all the time. On every node, kube-proxy maintains the rules that route packets for that IP to the right pod, typically via iptables or IPVS. Some network plugins replace kube-proxy with their own mechanism; the principle stays the same.
The container runtime is the piece that finally starts the container. Kubernetes talks to it through the Container Runtime Interface, CRI. The common runtime is containerd, with CRI-O as an alternative. You no longer need Docker as the runtime; your Docker images still run, because they follow the OCI standard.
What really happens on kubectl apply
Follow one request through and the interaction gets concrete. You send a pod manifest to the cluster with kubectl apply. Simplified, this is what happens:
- The kube-apiserver authenticates you, validates the manifest and stores the pod object in etcd. The pod has no node yet.
- The kube-scheduler watches for new pods without a node, picks one based on resources and rules, and writes the assignment back through the API server.
- The kubelet on the chosen node sees the new pod, pulls the image through the container runtime, starts the container and sets up volumes, secrets and ConfigMaps.
- The kube-proxy instances update their network rules once a service points at the pod and the endpoints controller has registered it.
- The kubelet reports the pod's status to the API server, which stores it in etcd. From then on
kubectl get pods shows the pod as Running.
No step bypasses the API server, and the sequence runs again on every change, even if you only raise a memory limit. In practice you almost never create pods directly. The deployment controller sits in front of this flow and creates a ReplicaSet, and the ReplicaSet controller turns that into pods. What a pod is exactly, and why you do not create one by hand, is covered in Kubernetes pod: what is it?.
Finding the components in your own cluster
On a cluster built with kubeadm or Minikube, the control plane components run as static pods in the kube-system namespace. You can look at them like any other pod:
kubectl get pods -n kube-system
On a small cluster with one control plane node and two workers, the output looks roughly like this:
NAME READY STATUS RESTARTS AGE
coredns-76f75df574-4xk2p 1/1 Running 0 14d
coredns-76f75df574-m9zvr 1/1 Running 0 14d
etcd-cp-1 1/1 Running 0 14d
kube-apiserver-cp-1 1/1 Running 0 14d
kube-controller-manager-cp-1 1/1 Running 2 14d
kube-proxy-8h2lq 1/1 Running 0 14d
kube-proxy-tj4nw 1/1 Running 0 14d
kube-proxy-x7d9c 1/1 Running 0 14d
kube-scheduler-cp-1 1/1 Running 2 14d
You can see the pattern: the control plane components carry the name of the node they run on, kube-proxy appears once per node because it runs as a DaemonSet, and CoreDNS is the cluster DNS that comes in as an add-on. A few restarts on the controller manager and the scheduler are usually harmless on a freshly booted node. On a kubeadm cluster the kubelet starts all static control plane pods in parallel, so those two come up before the API server answers, give up and try again. That is how this one build variant behaves, not a rule: other distributions order their components differently, as the k3s example below shows.
On k3s you will not find these components there. k3s bundles API server, scheduler, controller manager and etcd into a single process that runs as a system service. There you look with systemctl status k3s and journalctl -u k3s. In both cases the kubelet is not a pod but a service on the node, because something has to start the first containers before any pods exist.
For the state of a node, kubectl describe node <name> is the most important command. Conditions shows whether the kubelet reports pressure on memory, disk or process IDs, and Allocatable tells you how much CPU and memory the scheduler may still hand out here. If you want a local cluster to try this on, the options are in Installing Kubernetes locally.
High availability: what a mid-sized company really needs
A cluster with a single control plane node works, and for development and testing it is enough. If that node fails, the containers on the workers keep running, because the kubelet holds them. But you cannot change or roll out anything, and if a worker dies now, nothing gets rescheduled. For production that is too little.
Three control plane nodes are the right size for most companies I advise: a quorum of three tolerates one failure, and losing two data centers at once is a different kind of problem. Five nodes make sense for very large clusters with many API calls, or when maintenance has to take two nodes out at once. My own clusters run three control plane nodes in three data centers, each with its own etcd member.
And when is managed Kubernetes better than running it yourself? Honest answer: whenever nobody on the team wants to operate the control plane. The provider then runs API server, etcd and backups, and the cloud-controller-manager handles load balancers and volumes. I build my clusters myself because my customers want data sovereignty and control over etcd backups, a deliberate decision with operational cost. The trade-off in detail is in Kubernetes On-Premises or Cloud?.
Frequently asked questions
Master node or control plane: what is the difference?
There is no technical difference. Master was the old name for the nodes running the steering components; the documentation has used control plane for several years, and the node label is node-role.kubernetes.io/control-plane. You will still meet master in older guides and tools, and it means the same thing.
How many control plane nodes do I need?
One for development, three for production. The odd number comes from the etcd quorum. Five only pays off for very large clusters or special maintenance requirements.
What happens if etcd fails?
Your applications keep running at first, because kubelet and the container runtime hold them. But the cluster is frozen: no deployments, no scaling, no rescheduling on node failure, and kubectl returns errors. If you lose the etcd data entirely, you need the snapshot. Without one you reapply all manifests, and objects that only ever existed inside the cluster, such as certificates or secrets, are gone.
Do regular applications run on the control plane?
Not by default. Control plane nodes carry a taint that ordinary pods do not tolerate, so the steering components always have enough resources. You can remove it, which is common on small single-node clusters and Minikube. In production you leave it in place.
What is the difference between kubelet, etcd and the scheduler?
The three sit in very different places. etcd stores the desired state on the control plane. The scheduler decides which node a new pod lands on. The kubelet carries out, on each node, what was decided for it and reports back. Memory aid: etcd knows, the scheduler plans, the kubelet does.
Where to go next
With this map you can attribute every cluster problem to a component: Pending to the scheduler, NotReady to the node controller and kubelet, unreachable services to kube-proxy and endpoints, a hanging kubectl to the API server or etcd. The next step is to see it yourself: set up a local cluster, run kubectl get pods -n kube-system and read a pod's events with kubectl describe. Then look at the smallest object travelling through the whole flow, the pod.
All of this, with every example, is in chapter 2 of my book "Kubernetes: Practical Guide for Developers and DevOps Teams" (Rheinwerk Computing).
