Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
All skills

gke-ai-troubleshooting-handle-disruption-gpu-tpu

Diagnose whether GPU/TPU node disruptions on GKE came from host maintenance and recommend mitigations.

  • Part of google/skills
  • 6k installs
  • DevOps & Self-hosting · Containers and clusters

When to use it

You have a GKE GPU/TPU node that restarted or evicted pods and you need to tell whether scheduled host maintenance caused it.

For example

You say

Use gke-ai-troubleshooting-handle-disruption-gpu-tpu to check why my GPU nodepool in project ml-prod, cluster training-east, us-central1, restarted pods around 2025-03-10 14:00 UTC.

You get
  • Maintenance cause verdict
  • PromQL interruption query
  • Mitigation recommendations

How it works

  • Checks the scheduled-maintenance node label, interruption count metrics, and active maintenance logs before concluding.
  • If all checks come back empty, rules out host maintenance and points to application-level causes instead.