All skills
gke-ai-troubleshooting-jobset-interruption
Diagnose GKE JobSet restart loops and preemptions, then recommend fixes like on-demand VMs or host quarantine.
- Part of google/skills
- 5.9k installs
- DevOps & Self-hosting · Containers and clusters
When to use it
Your AI/ML training JobSet keeps restarting or its pods stay pending, and you need to tell whether spot preemption, a bad host, or a worker crash is the cause.
For example
You say
Use gke-ai-troubleshooting-jobset-interruption to find out why my JobSet llama-train in cluster ml-prod keeps restarting since 30 minutes ago.
You get
- Restart and preemption findings
- Node-to-host correlation
- Recommended resolution steps
How it works
- Correlates failing nodes to their underlying GCE host to spot one bad machine causing repeated restarts.
- Reads coordinator worker logs for NCCL timeouts and collective communication hangs.
Before installing
The JobSet metrics package must be enabled in kube-state-metrics, and Cloud Logging and Cloud Monitoring must be enabled for the project.
This skill includes a script your assistant may run on your computer.
More Containers and clusters skills
See all- kubernetes-specialist14.9kCreates and manages Kubernetes workloads, manifests, Helm charts, RBAC rules and pod debugging.
- dynamo-recipe-runner2.6kSelects, validates, patches, and deploys NVIDIA Dynamo Kubernetes recipes for model serving.
- dynamo-interconnect-check1.9kValidates that a Dynamo deployment's NIXL, UCX, and NCCL interconnect is ready for disaggregated serving.
- kubernetes-architect1kPlans and reviews Kubernetes setups, including GitOps workflows and enterprise container orchestration.
Also in this pack
- cloud-run-basics19.4kDeploys and manages Cloud Run services, jobs and worker pools.
- gemini-api18.8kWrites code that calls Gemini models for text, images, video, audio, embeddings and more across languages.
- google-cloud-recipe-auth17.1kExplains how to authenticate and authorize users and services to Google Cloud APIs, including IAM and ADC.
- google-cloud-recipe-onboarding16.9kWalks a new developer through creating a Google Cloud account, setting up billing and deploying a first resource.