Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
All skills

gke-ai-troubleshooting-tpu-vbar-oom

Diagnose vbar_control_agent OOMs and segfaults on GKE TPU v6e nodes from serial console and plugin logs.

  • Part of google/skills
  • 5.7k installs
  • DevOps & Self-hosting · Containers and clusters

When to use it

Your TPU v6e training job crashes or nodes go unready, and serial console logs show vbar_control_agent killed by the memory cgroup.

For example

You say

Use gke-ai-troubleshooting-tpu-vbar-oom to investigate why my TPU v6e training job on cluster tpu-cluster-prod crashed around 2026-04-14T20:00:00Z.

You get
  • OOM log findings
  • tpu-device-plugin error check
  • Custom metrics collector check

How it works

  • Searches serial console logs for the specific vbar_control_agent memory cgroup OOM pattern.
  • Checks tpu-device-plugin logs for checksum-mismatch metrics fetch errors that point to vBAR corruption.
  • Looks for custom TPU metrics collectors polling GetHostMetrics, which can trigger the race condition.

Before installing

Cloud Logging must be enabled for the project, with gcloud or equivalent access to the project and cluster.

This skill includes a script your assistant may run on your computer.