kubectl get namespace example-ns

Output example:

NAME        STATUS        AGE
example-ns  Terminating   48d

Before you begin, it's good to describe the pod and see if you can find the reason for why it's in terminating state.

This you can do with the following command:

kubectl describe namespace <namespace-name>

For instance, I got the following output with my stucked namespace:

Output:

Status:       Terminating
Conditions:
  Type                                         Status  LastTransitionTime               Reason                Message
  ----                                         ------  ------------------               ------                -------
  NamespaceDeletionDiscoveryFailure            True    Fri, 29 Mar 2024 09:28:54 +0100  DiscoveryFailed       Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: spdx.softwarecomposition.kubescape.io/v1beta1: stale GroupVersion discovery: spdx.softwarecomposition.kubescape.io/v1beta1
  NamespaceDeletionGroupVersionParsingFailure  False   Fri, 29 Mar 2024 09:28:55 +0100  ParsedGroupVersions   All legacy kube types successfully parsed
  NamespaceDeletionContentFailure              False   Fri, 29 Mar 2024 09:28:55 +0100  ContentDeleted        All content successfully deleted, may be waiting on finalization
  NamespaceContentRemaining                    True    Fri, 29 Mar 2024 09:28:55 +0100  SomeResourcesRemain   Some resources are remaining: persistentvolumeclaims. has 1 resource instances, pods. has 1 resource instances
  NamespaceFinalizersRemaining                 True    Fri, 29 Mar 2024 09:28:55 +0100  SomeFinalizersRemain  Some content in the namespace has finalizers remaining: kubernetes.io/pvc-protection in 1 resource instances

Step 1: Identify and Attempt to Remove Stuck Resources

1. List All Resources in example-ns

   Begin by identifying all resources that might be preventing the namespace from being deleted.

❗You may imagine that the kubectl get all -n example-ns would provide all resources in the namespace, but that's simply not correct. This command will only provide most common resources like pods, deployment, configmap etc. But it will for instance not provide any CRDs.

So we need to run something that is providing the full list of namespace scoped resources:

kubectl get $(kubectl api-resources --verbs=list --namespaced -o name | awk NF | paste -sd "," -) -n example-ns --ignore-not-found

The output of this command may be pretty massive.

Attempt Graceful Resource Deletion

Now we could try to delete any resources you've identified as potentially problematic.

kubectl delete <resource-type> <resource-name> -n example-ns

If that doesn't work, we'd need to remove finalizers from the resource.

Check if there is any finalizer on that resource:

kubectl get <resource-type> <resource-name> -n example-ns -o json | jq '.metadata.finalizers'

Forcefully Remove Finalizers

Remove finalizers with caution to avoid leaving orphaned resources:

kubectl patch <resource-type> <resource-name> -n example-ns -p '{"metadata":{"finalizers":[]}}' --type=merge

❗You could also simply edit the resource and remove the finalizers manually from that object.

Step 3: Forcibly Remove the Namespace

If example-ns is still stuck, proceed to directly interact with the Kubernetes API.

1. Start a Kubernetes API Server Proxy:
   

   kubectl proxy &
   

2. Send a Request to Remove Namespace Finalizers

   Save an output from the namespace resource to a file, modify the finalizers and send it to the api:

   kubectl get namespace example-ns -o json > example-ns.json
   

   Edit the file, by removing the finalizers:

   "finalizers": []
   

   Assuming your modified JSON file is named example-ns.json and contains the namespace definition with the finalizers removed, you can use curl to send this file to the Kubernetes API server:

   curl -X PUT http://127.0.0.1:8001/api/v1/namespaces/example-ns/finalize -H "Content-Type: application/json" --data-binary "@example-ns.json"

3. Finally, verify that example-ns has been deleted.
   

   kubectl get namespace example-ns
   

🚀 If there's no output, the namespace has been successfully removed.

❗Don't forget to kill the proxy process that we started earlier.

 Summary

This tutorial walked you through diagnosing and resolving a namespace stuck in the Terminating state, using example-ns as a fictive example. Starting with resource identification and deletion attempts, moving to manual finalizer removal, and ultimately using direct API interaction, this guide covers an approach to force-delete a Kubernetes namespace.