kubernetes troubleshooting quick commands 1

Kubernetes troubleshooting and cluster administration center on inspecting API objects, querying runtime event streams, analyzing resource pressure, and managing node lifecycle states.

---

1. Cluster & Node Health Diagnostics

Used to evaluate control plane connectivity, compute capacity, and host-level resource pressures.

Command	Operational Purpose	Key Output Signals to Check
kubectl get nodes -o wide	Overview of node status, internal IPs, OS images, kernel versions, and container runtime versions.	STATUS: Must be Ready. Check if container runtime or kernel versions are consistent across nodes.
kubectl describe node <node>	Deep inspection of conditions, allocatable capacity, system daemons, and attached taints.	Conditions: Look for MemoryPressure, DiskPressure, or PIDPressure set to True. Check Taints preventing scheduling.
kubectl top nodes	Real-time CPU and Memory utilization per node (requires Metrics Server).	High CPU% or MEMORY% nearing 90–100%, indicating impending pod eviction or scheduling throttling.
kubectl get events -A --sort-by='.metadata.creationTimestamp'	Chronological, cluster-wide stream of warnings and state transitions.	Look for FailedScheduling, Evicted, NodeNotReady, FailedMount, or OOMKilled.

# Filter only warning events across the cluster
kubectl get events -A --field-selector type=Warning --sort-by='.metadata.creationTimestamp'

---

2. Pod & Workload Troubleshooting

Used to diagnose pods stuck in non-running states (CrashLoopBackOff, Pending, ImagePullBackOff, OOMKilled).

Command	Operational Purpose	Key Output Signals to Check
kubectl get pods -A -o wide	List all pods with IP mappings, host nodes, restart counts, and current status.	RESTARTS: Continually incrementing counts indicate crashes. STATUS: Anything other than Running or Completed.
kubectl describe pod <pod> -n <ns>	Inspect container lifecycle state, resource limits, volume mounts, and recent pod events.	Last State -> Exit Code: 137 (SIGKILL / OOMKilled), 143 (SIGTERM), 1 (Application error). Events at the bottom.
kubectl logs <pod> -n <ns> -c <container> --previous	Fetch stdout/stderr from the previously terminated container instance before it crashed.	Application stack traces, uncaught exceptions, or segmentation faults that caused the container to exit.
kubectl logs -f -l app=<label> -n <ns> --tail=100 --max-log-requests=10	Stream real-time logs across all pods matching a label selector simultaneously.	Correlating request traffic and distributed error traces across multiple replicas.

# Find all non-running pods across all namespaces
kubectl get pods -A --field-selector status.phase!=Running,status.phase!=Succeeded

# Check exit status of an OOM-killed container
kubectl get pod <pod> -n <ns> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

---

3. Container-Level & Interactive Debugging

Used when log inspection is insufficient and interactive runtime inspection is required.

# 1. Open an interactive shell inside a running container
kubectl exec -it <pod> -n <ns> -c <container> -- /bin/sh

# 2. Attach an ephemeral debugging container with networking utilities to a broken pod
kubectl debug -it pod/<pod> -n <ns> --image=nicolaka/netshoot --target=<container>

# 3. Spawn a temporary interactive pod for cluster-internal network/DNS probing
kubectl run net-tool --rm -it --image=busybox:1.28 --restart=Never -- nslookup kubernetes.default

# 4. Forward a local port directly to a pod (bypasses Service, Ingress, and NetworkPolicies)
kubectl port-forward pod/<pod> 8080:80 -n <ns>

# 5. Forward a local port directly to an internal Service
kubectl port-forward svc/<service> 5432:5432 -n <ns>

---

4. Networking & Service Resolution

Used when pods cannot discover or route traffic to internal services or external APIs.

Command	Operational Purpose	Key Output Signals to Check
kubectl get svc,endpoints,endpointslices -n <ns>	Verifies Service definition and whether backend pods are actively registered.	ENDPOINTS: If empty (<none>), the Service's selector labels do not match the target pod labels.
kubectl get netpol -n <ns>	Lists active NetworkPolicies restricting traffic.	Misconfigured ingress/egress rules silently dropping inter-namespace or DNS packets.
kubectl logs -n kube-system -l k8s-app=kube-dns	Inspect CoreDNS cluster resolver logs.	Upstream forwarding timeouts, NXDOMAIN loops, or CoreDNS pods crashing under high query volume.

# Test CoreDNS resolution from inside the cluster
kubectl run dns-test --rm -it --image=curlimages/curl --restart=Never -- curl -v http://<service-name>.<namespace>.svc.cluster.local:<port>

---

5. Storage & Volume Diagnostics

Used to resolve FailedAttachVolume, FailedMount, or stuck volume locks during pod rescheduling.

Command	Operational Purpose	Key Output Signals to Check
kubectl get pvc,pv -A	Evaluates volume binding state and capacity allocations.	STATUS: Must be Bound. If Pending, the StorageClass cannot provision the disk or capacity is exhausted.
kubectl describe pvc <pvc> -n <ns>	Inspects claims, access modes, and dynamic provisioning events.	VolumeAttributesClass, provisioner timeout errors, or quota limits preventing disk attachment.
kubectl get volumeattachment	Inspects node-to-volume CSI attachments.	ATTACHED: If false, the cloud provider or CSI driver is failing to attach the disk to the target node.

---

6. Cluster Management & Operational Control

Used during maintenance windows, node upgrades, and deployment lifecycle orchestration.

# 1. Safely take a node out of service for hardware/OS maintenance
kubectl cordon <node>                                         # Mark node unschedulable
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force # Evict workloads safely

# 2. Return node to the scheduling pool after maintenance
kubectl uncordon <node>

# 3. Trigger a zero-downtime rolling restart of all pods in a workload
kubectl rollout restart deployment/<deployment-name> -n <ns>
kubectl rollout restart statefulset/<statefulset-name> -n <ns>

# 4. Monitor rollout status or roll back a broken release
kubectl rollout status deployment/<deployment-name> -n <ns>
kubectl rollout undo deployment/<deployment-name> -n <ns>

# 5. Inspect cluster resource allocations vs ResourceQuotas
kubectl describe resourcequota -n <ns>

---

7. Advanced JSONPath & SRE One-Liners

# Top 10 pods consuming the most memory across the entire cluster
kubectl top pods -A --sort-by=memory | head -n 11

# Top 10 pods consuming the most CPU across the entire cluster
kubectl top pods -A --sort-by=cpu | head -n 11

# Print all pods sorted by their restart count
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}' | sort -k3 -n -r | head -n 20

# Identify which nodes pods are scheduled on for a specific deployment
kubectl get pods -n <ns> -l app=<app-name> -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,STATUS:.status.phase

# Extract plain-text raw logs from all terminated pods across all namespaces
kubectl get pods -A -o jsonpath='{range .items[?(@.status.containerStatuses[*].state.terminated)]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}'

---