My Notes

Deleting-Stuck-Namespaces

3 min read

Deleting Kubernetes Namespaces Stuck in Terminating State

When a Kubernetes namespace gets stuck in the Terminating state, it’s usually because resources within the namespace have finalizers that prevent the namespace from being fully deleted. This guide provides a step-by-step approach to resolving this issue.

Identifying Stuck Namespaces

First, identify namespaces that are stuck in the Terminating state:

kubectl get namespaces

Look for namespaces with the Terminating status that have been in that state for an extended period.

Step 1: Check for Resources in the Namespace

Before proceeding with forced deletion, check if there are any resources still in the namespace:

# List all resources in the namespace
kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace-name>

Step 2: Check for Finalizers

Examine the namespace for finalizers:

kubectl get namespace <namespace-name> -o json

Look for the spec.finalizers field in the output. If finalizers exist, they need to be removed.

Step 3: Remove Finalizers (Standard Method)

To remove finalizers from a namespace:

  1. Export the namespace definition to a JSON file:
kubectl get namespace <namespace-name> -o json > namespace.json
  1. Edit the file to remove finalizers:
# If you have jq installed
cat namespace.json | jq '.spec.finalizers = []' > namespace-updated.json
 
# Or manually edit the file and remove the finalizers array
  1. Apply the updated namespace definition:
kubectl replace --raw "/api/v1/namespaces/<namespace-name>/finalize" -f namespace-updated.json

Step 4: Remove Finalizers (One-liner Method)

If you prefer a one-liner command:

kubectl get namespace <namespace-name> -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/<namespace-name>/finalize" -f -

Step 5: Force Delete Custom Resources

If there are custom resources with finalizers:

  1. Identify custom resources:
kubectl get <custom-resource-type> -n <namespace-name>
  1. For each resource, remove its finalizers:
kubectl patch <resource-type> <resource-name> -n <namespace-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Step 6: Force Delete the Namespace (Last Resort)

If the above methods don’t work, you can force delete the namespace:

kubectl delete namespace <namespace-name> --force --grace-period=0

⚠️ Warning: Force deletion can leave orphaned resources in your Kubernetes cluster. Use this method only when other approaches have failed.

Step 7: Check Kubernetes API Server Logs

If problems persist, check the Kubernetes API server logs for errors:

# For kubeadm clusters
kubectl logs -n kube-system -l component=kube-apiserver

Real-world Example

Here’s an example of removing finalizers from a namespace called gh-actions-runners:

# Export the namespace to a file
kubectl get namespace gh-actions-runners -o json > namespace.json
 
# Remove finalizers
cat namespace.json | jq '.spec.finalizers = []' > namespace-updated.json
 
# Apply changes
kubectl replace --raw "/api/v1/namespaces/gh-actions-runners/finalize" -f namespace-updated.json

Or as a one-liner:

kubectl get namespace gh-actions-runners -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/gh-actions-runners/finalize" -f -

Preventing Namespace Termination Issues

To prevent these issues in the future:

  1. Delete all resources in a namespace before deleting the namespace itself
  2. Ensure controllers that manage custom resources are functioning properly
  3. Be cautious with custom resource definitions (CRDs) that add finalizers
  4. Keep your Kubernetes components up to date

Note: This guide assumes you have the necessary permissions to manage namespaces in your Kubernetes cluster. Some operations may require cluster-admin privileges.

Graph View