Notes on Reducing Disk Usage on Kubernetes Nodes

⚠️ This article was automatically translated by OpenAI (gpt-4-turbo). It may be edited eventually, but please be aware that it may contain incorrect information at this time.

Notes on reducing disk usage as the Node became Disk Full. Specifically, I want to delete unused container images.

Use the kubectl debug command to access the Node.

kubectl debug node/<node_name> -it --image=busybox

Start interaction with the node session using chroot.

chroot /host

Check the disk size. It's at 100% usage.

$ df -h | head -3
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/vda2                                  150G  148G     0 100% /
tmpfs                                      3.9G     0  3.9G   0% /dev/shm

Install nerdctl, a docker CLI for containerd, on the Node.

cd /tmp
wget https://github.com/containerd/nerdctl/releases/download/v1.7.6/nerdctl-1.7.6-linux-amd64.tar.gz
tar xzvf nerdctl-1.7.6-linux-amd64.tar.gz 
mv nerdctl /usr/local/bin/

Run nerdctl version to confirm access to the containerd server.

$ nerdctl version
WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
Client:
 Version:   v1.7.6
 OS/Arch:   linux/amd64
 Git commit:    845e989f69d25b420ae325fedc8e70186243fd93
 buildctl:
  Version:  

Server:
 containerd:
  Version:  v1.7.13
  GitCommit:    7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:  1.1.12
  GitCommit:    v1.1.12-0-g51d5e946

Check the container images accumulated on the Node.

nerdctl --namespace k8s.io images

Unused images are likely taking up disk space.

Use the following command to delete all images. Images that are in use and running cannot be deleted, so only unused images will be removed.

nerdctl --namespace k8s.io images | awk '{print $3}' | sort | uniq | xargs nerdctl --namespace k8s.io rmi -f

The disk usage has slightly recovered.

$ df -h  | head -3 
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/vda2                                  150G  117G   27G  82% /
tmpfs                                      3.9G     0  3.9G   0% /dev/shm

Execute exit twice to terminate.

$ kubectl get pod                                               
NAME                                          READY   STATUS      RESTARTS   AGE
node-debugger-kiwi-build-0e46b92bc3b6-nvpg5   0/1     Completed   0          8m7s

The debug pod remains in a Completed state, so feel free to delete it if it's unnecessary.