Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
All skills

spark-engineer

Write and tune PySpark or Scala Spark jobs, including DataFrame transforms, joins, partitioning, and shuffle settings.

When to use it

You have a Spark job that runs but is slow or spilling in the Spark UI, and you need to find the shuffle or skew fix.

For example

You say

Use spark-engineer to fix my PySpark job that joins a 2 TB events table to a small product table and keeps spilling during the shuffle.

You get
  • Rewritten join code
  • Shuffle and memory config
  • Partitioning strategy notes

How it works

  • Skew handling uses salting or custom partitioning when one key dominates.
  • Small dimension tables under 200 MB are joined with a broadcast hint.
  • Caching is only recommended when a DataFrame is reused, then unpersisted.