Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
All skills

gke-inference

Generate a GKE inference manifest for a chosen model, server, and accelerator, then deploy it.

  • Part of google/skills
  • 8.5k installs
  • DevOps & Self-hosting · Containers and clusters

When to use it

You have a model like Gemma or Llama and a GKE cluster but no working inference manifest yet, and you need to pick a valid accelerator and server combination.

For example

You say

Use gke-inference to deploy gemma-2-9b-it with vLLM on an L4 GPU targeting 50ms NTPOT, and give me the manifest to apply.

You get
  • inference.yaml manifest
  • Deployment commands
  • Accelerator choice

How it works

  • Uses gcloud container ai profiles to list supported models and valid model/server/accelerator combinations before generating anything.
  • Includes a ComputeClass example for targeting GPU nodes on Autopilot.
  • Covers autoscaling with GPU duty cycle metrics and tuning tips like quantization and tensor parallelism.

Before installing

A golden path GKE Autopilot cluster, an authenticated gcloud CLI, and sufficient GPU/TPU quota in the target region.