All skills
gke-ai-troubleshooting-handle-disruption-gpu-tpu
Diagnose whether GPU/TPU node disruptions on GKE came from host maintenance and recommend mitigations.
- Part of google/skills
- 6k installs
- DevOps & Self-hosting · Containers and clusters
When to use it
You have a GKE GPU/TPU node that restarted or evicted pods and you need to tell whether scheduled host maintenance caused it.
For example
You say
Use gke-ai-troubleshooting-handle-disruption-gpu-tpu to check why my GPU nodepool in project ml-prod, cluster training-east, us-central1, restarted pods around 2025-03-10 14:00 UTC.
You get
- Maintenance cause verdict
- PromQL interruption query
- Mitigation recommendations
How it works
- Checks the scheduled-maintenance node label, interruption count metrics, and active maintenance logs before concluding.
- If all checks come back empty, rules out host maintenance and points to application-level causes instead.
More Containers and clusters skills
See all- kubernetes-specialist14.9kCreates and manages Kubernetes workloads, manifests, Helm charts, RBAC rules and pod debugging.
- dynamo-recipe-runner2.6kSelects, validates, patches, and deploys NVIDIA Dynamo Kubernetes recipes for model serving.
- dynamo-interconnect-check1.9kValidates that a Dynamo deployment's NIXL, UCX, and NCCL interconnect is ready for disaggregated serving.
- kubernetes-architect1kPlans and reviews Kubernetes setups, including GitOps workflows and enterprise container orchestration.
Also in this pack
- cloud-run-basics19.4kDeploys and manages Cloud Run services, jobs and worker pools.
- gemini-api18.8kWrites code that calls Gemini models for text, images, video, audio, embeddings and more across languages.
- google-cloud-recipe-auth17.1kExplains how to authenticate and authorize users and services to Google Cloud APIs, including IAM and ADC.
- google-cloud-recipe-onboarding16.9kWalks a new developer through creating a Google Cloud account, setting up billing and deploying a first resource.