kubectl debug adds an extra container with an image of your choice, such as busybox, to a running Pod without restarting it. Kubernetes calls this debug container an ephemeral container. With --target, it shares your application container's process namespace: you see its processes, reach its filesystem through /proc/<PID>/root and test the network from inside the Pod. That's what you need for distroless images, where kubectl exec fails for lack of a shell. For crashing Pods, --copy-to creates a changeable copy, and kubectl debug node/<name> opens a shell on the node. Profiles set the debug container's privileges; since kubectl v1.36 the default is general.
I'm a Kubestronaut with all five CNCF certifications and the author of the Kubernetes Practical Guide published by Rheinwerk Computing. All my Kubernetes articles are on the Kubernetes page.
All output below comes from a local test cluster on October 30, 2026: minikube v1.39.0 with the Docker driver, Kubernetes v1.37.0, containerd 2.3.4, kubectl v1.37.2. It is shortened, and the Pod names are examples.
Why kubectl exec isn't enough for distroless
Per the project, distroless images contain no package manager, no shell and no other usual Linux programs. That makes them small and safer, but hard to inspect. The test Pod runs gcr.io/distroless/python3-debian13:nonroot and starts a small web server:
kubectl run shop-api --image=gcr.io/distroless/python3-debian13:nonroot \
--restart=Never --command -- python3 -m http.server 8080
kubectl exec -it shop-api -- sh
error: Internal error occurred: ... OCI runtime exec failed: exec failed:
unable to start container process: exec: "sh": executable file not found in $PATH
exec … ls / fails too. I cover the general flow with get, describe, logs and exec in kubectl Commands: The Essentials. This picks up where that ends.
kubectl debug with an ephemeral container: the basic command
kubectl debug -it shop-api --image=busybox:1.37 --target=shop-api -c debugger -- sh
Targeting container "shop-api". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
All commands and output from this session will be recorded in container logs, including credentials and sensitive information passed through the command prompt.
If you don't see a command prompt, try pressing enter.
--image picks the toolbox, --target the container whose processes you want to see, -c the name of the debug container. Inside it, these commands ran:
ps
wget -qO- http://localhost:8080/ | head -5
ls /proc/1/root/
PID USER TIME COMMAND
1 65532 0:00 python3 -m http.server 8080
19 root 0:00 sh
26 root 0:00 ps
<!DOCTYPE HTML>
<html lang="en">
...
bin boot dev etc home lib proc root run sbin sys tmp usr var
PID 1 is the distroless Python process, user ID 65532. localhost:8080 answers because a Pod's containers share the network, and /proc/1/root/ holds the application's filesystem.
Without --target, ps in the test showed only the debug container's own processes (sh and ps), not the Python process. Per the docs, the container runtime has to support --target; containerd 2.3.4 did.
Heed the second warning: everything you type, passwords included, lands in the debug container's log (kubectl logs shop-api -c debugger).
What stays in the Pod afterwards
Per the docs, an ephemeral container can't be changed or removed once added, is never restarted automatically and can't have ports, probes or resources. After exit, kubectl describe pod lists it under "Ephemeral Containers" as Terminated, while the Pod keeps running:
Ephemeral Containers:
debugger:
...
Image: busybox:1.37
...
Command:
sh
State: Terminated
Reason: Completed
Exit Code: 0
Every call adds an entry; after the test, the Pod had seven. They only go away when the Pod is replaced, for example by the next rollout.
kubectl writes to the pods/ephemeralcontainers subresource (verbs get, patch, update in the test cluster). Whoever can patch it can look inside other Pods, so grant it as carefully as pods/exec; see Kubernetes RBAC: Roles and Permissions.
Profiles: which privileges the debug container gets
--profile decides which securityContext kubectl sets. The help of kubectl v1.37.2 lists general, baseline, restricted, netadmin and sysadmin, with general as the default. Per the Kubernetes v1.36 changelog, the default changed from legacy to general, and legacy is due to be removed in v1.39; in the test it still worked, with a warning. The docs page on debugging running Pods still named legacy as the default on October 30, 2026.
Three findings from the test that cost time in practice:
general fails under Pod Security baseline. In the namespace shop labelled pod-security.kubernetes.io/enforce=baseline, the API server rejected the debug container:
Error from server (Forbidden): pods "api" is forbidden: violates PodSecurity "baseline:latest":
non-default capabilities (container "dbg-general" must not include "SYS_PTRACE" in securityContext.capabilities.add)
With --profile=baseline it ran, and ps still showed the target's processes. More on the levels in Kubernetes Pod Security Standards.
baseline sees processes, but not another container's filesystem. Without SYS_PTRACE, ls /proc/1/root as root in the debug container returned Permission denied, because the target process runs under a different user ID.
restricted with a root image never starts. busybox runs as root, and restricted requires runAsNonRoot. The debug container got stuck in CreateContainerConfigError ("container has runAsNonRoot and image will run as root"), and since ephemeral containers can't be removed, that entry stays. The fix is --custom with a partial container spec in YAML. Only with the target's user and group ID (65532 here) was the filesystem readable; runAsUser alone gave Permission denied:
# nonroot.yaml
securityContext:
runAsUser: 65532
runAsGroup: 65532
kubectl debug shop-api --image=busybox:1.37 --target=shop-api \
--profile=restricted --custom=nonroot.yaml -c dbg-nonroot -- sh -c 'id; ls /proc/1/root/'
uid=65532 gid=65532 groups=65532
bin boot dev etc home lib proc root run sbin sys tmp usr var
Per the docs, --custom can't change name, image, command, lifecycle or volumeDevices, nor the Pod.
kubectl debug --copy-to: a copy to inspect
If the application container crashes right away or needs a different command, use --copy-to. It creates a new Pod modelled on the old one, in the simplest case with an extra toolbox container:
kubectl debug shop-api --copy-to=shop-api-debug --image=busybox:1.37 -c debugger -- sleep 3600
kubectl exec shop-api-debug -c debugger -- ps
PID USER TIME COMMAND
1 65535 0:00 /pause
7 65532 0:00 python3 -m http.server 8080
13 root 0:00 sleep 3600
19 root 0:00 ps
--share-processes defaults to true, so the copy had shareProcessNamespace: true. Watch out: it had no labels (original: run=shop-api), because --keep-labels defaults to false, so no Service sends it traffic. Probes are only kept with --keep-liveness, --keep-readiness and --keep-startup.
The stronger case is a Pod in a crash loop. In the test, the deliberately broken container worker exited immediately with "CONFIG_PATH fehlt" (German for "missing"). Distroless ships a debug variant with a busybox shell for its images; for nonroot it's debug-nonroot. --set-image swaps the image in the copy, and --container with a command after -- replaces the start command:
kubectl debug worker --copy-to=worker-debug --container=worker \
--set-image=worker=gcr.io/distroless/python3-debian13:debug-nonroot \
-- sh -c 'id; ls -la /config; sleep 3600'
kubectl logs worker-debug -c worker
uid=65532(nonroot) gid=65532(nonroot) groups=65532(nonroot)
ls: /config: No such file or directory
The copy runs, and the cause is visible: /config is missing. How to narrow it down in the original is in CrashLoopBackOff: Debugging Pods. Delete the copies afterwards with kubectl delete pod shop-api-debug worker-debug, or they keep running.
kubectl debug node: a shell on the node
If the problem sits below the Pod, in the kubelet, containerd or the node's network, kubectl debug node/<name> starts a Pod on that node:
kubectl debug node/kubectl-debug-test -it --image=busybox:1.37
Creating debugging pod node-debugger-kubectl-debug-test-226bz with container debugger on node kubectl-debug-test.
With the default general profile, the test Pod ran with hostPID, hostNetwork and hostIPC, unprivileged, with the node's filesystem at /host. ls /host/var/log/pods showed the log folders of every Pod on the node. The docs name more logs under /host/var/log, such as kubelet.log, and warn that chroot /host can fail without a privileged Pod. In the minikube test, chroot /host crictl ps worked anyway; don't count on it elsewhere, and use --profile=sysadmin (privileged: true in the test) when needed.
When it ends, the Pod stays behind as Completed, or Error if the command failed; four calls left four. Delete them with kubectl delete pod <name>. And anyone who can debug this way can reach the node's filesystem: grant the right to create Pods deliberately.
Which approach when
Frequently asked questions
What is an ephemeral container?
A container that Kubernetes adds to a running Pod without a restart, with no ports, probes or resources. The feature has been stable since Kubernetes v1.25.
Why does ps in the debug container show only its own processes?
Because --target is missing, so the debug container gets its own process namespace. With --target=<container>, you see the application's processes, if the container runtime supports it.
Can I delete an ephemeral container?
No. It stays in the Pod until the Pod is replaced. So experiment with profiles on a test Pod or a copy.
Which profile does kubectl debug use by default?
In kubectl v1.37.2 it's general, the default since v1.36. Older kubectl versions use legacy. Check yours with kubectl debug --help, which shows the default next to --profile.
Where to go next
kubectl debug is the next step when describe, logs and exec no longer help. If the container never starts because the image is missing, read ImagePullBackOff in Kubernetes: Fixes. To try the commands safely, set up a cluster with Local Kubernetes: A Cluster on Your Machine.
My suggestion for the next distroless Pod: first kubectl debug -it <pod> --image=busybox --target=<container>, then ps and /proc/1/root. The fundamentals of Pods and the command line are in my Kubernetes Practical Guide (Rheinwerk Computing), and every option is in the Kubernetes docs on debugging running Pods.
