All skills
spark-engineer
Write and tune PySpark or Scala Spark jobs, including DataFrame transforms, joins, partitioning, and shuffle settings.
- Part of Jeffallan/claude-skills
- 4.2k installs
- Developer Tools
When to use it
You have a Spark job that runs but is slow or spilling in the Spark UI, and you need to find the shuffle or skew fix.
For example
You say
Use spark-engineer to fix my PySpark job that joins a 2 TB events table to a small product table and keeps spilling during the shuffle.
You get
- Rewritten join code
- Shuffle and memory config
- Partitioning strategy notes
How it works
- Skew handling uses salting or custom partitioning when one key dominates.
- Small dimension tables under 200 MB are joined with a broadcast hint.
- Caching is only recommended when a DataFrame is reused, then unpersisted.
More Developer Tools skills
See all- lark-shared497kHandles Lark CLI login, identity, permissions and authorization status.
- lark-event489kStreams Lark real-time events like messages, approvals and meetings for bots and subscribers.
- emil-design-eng467kApplies Emil Kowalski's design-engineering philosophy to polish UI components and their interactions.
- ui-ux-pro-max463kGuides interface design with searchable styles, palettes, fonts, UX rules and chart types.
Also in this pack
- golang-pro19.8kWrites concurrent Go services with goroutines, channels, gRPC or REST, and profiles performance.
- php-pro16.4kBuilds modern PHP applications with strict typing, Laravel or Symfony, tests and REST or GraphQL APIs.
- typescript-pro15kBuilds advanced TypeScript types and end-to-end type-safe APIs with tRPC.
- flutter-expert14.9kBuilds cross-platform Flutter apps with widgets, Riverpod or Bloc state, navigation and performance tuning.