All skills
gke-ai-troubleshooting-tpu-vbar-oom
Diagnose vbar_control_agent OOMs and segfaults on GKE TPU v6e nodes from serial console and plugin logs.
- Part of google/skills
- 5.7k installs
- DevOps & Self-hosting · Containers and clusters
When to use it
Your TPU v6e training job crashes or nodes go unready, and serial console logs show vbar_control_agent killed by the memory cgroup.
For example
You say
Use gke-ai-troubleshooting-tpu-vbar-oom to investigate why my TPU v6e training job on cluster tpu-cluster-prod crashed around 2026-04-14T20:00:00Z.
You get
- OOM log findings
- tpu-device-plugin error check
- Custom metrics collector check
How it works
- Searches serial console logs for the specific vbar_control_agent memory cgroup OOM pattern.
- Checks tpu-device-plugin logs for checksum-mismatch metrics fetch errors that point to vBAR corruption.
- Looks for custom TPU metrics collectors polling GetHostMetrics, which can trigger the race condition.
Before installing
Cloud Logging must be enabled for the project, with gcloud or equivalent access to the project and cluster.
This skill includes a script your assistant may run on your computer.
More Containers and clusters skills
See all- kubernetes-specialist14.9kCreates and manages Kubernetes workloads, manifests, Helm charts, RBAC rules and pod debugging.
- dynamo-recipe-runner2.6kSelects, validates, patches, and deploys NVIDIA Dynamo Kubernetes recipes for model serving.
- dynamo-interconnect-check1.9kValidates that a Dynamo deployment's NIXL, UCX, and NCCL interconnect is ready for disaggregated serving.
- kubernetes-architect1kPlans and reviews Kubernetes setups, including GitOps workflows and enterprise container orchestration.
Also in this pack
- cloud-run-basics19.4kDeploys and manages Cloud Run services, jobs and worker pools.
- gemini-api18.8kWrites code that calls Gemini models for text, images, video, audio, embeddings and more across languages.
- google-cloud-recipe-auth17.1kExplains how to authenticate and authorize users and services to Google Cloud APIs, including IAM and ADC.
- google-cloud-recipe-onboarding16.9kWalks a new developer through creating a Google Cloud account, setting up billing and deploying a first resource.