Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
All skills

gke-ai-troubleshooting-jobset-interruption

Diagnose GKE JobSet restart loops and preemptions, then recommend fixes like on-demand VMs or host quarantine.

  • Part of google/skills
  • 5.9k installs
  • DevOps & Self-hosting · Containers and clusters

When to use it

Your AI/ML training JobSet keeps restarting or its pods stay pending, and you need to tell whether spot preemption, a bad host, or a worker crash is the cause.

For example

You say

Use gke-ai-troubleshooting-jobset-interruption to find out why my JobSet llama-train in cluster ml-prod keeps restarting since 30 minutes ago.

You get
  • Restart and preemption findings
  • Node-to-host correlation
  • Recommended resolution steps

How it works

  • Correlates failing nodes to their underlying GCE host to spot one bad machine causing repeated restarts.
  • Reads coordinator worker logs for NCCL timeouts and collective communication hangs.

Before installing

The JobSet metrics package must be enabled in kube-state-metrics, and Cloud Logging and Cloud Monitoring must be enabled for the project.

This skill includes a script your assistant may run on your computer.