apache/dolphinscheduler

Apache DolphinScheduler is a data workflow orchestration platform: it lets teams define tasks and their dependencies as a DAG, then schedules, monitors, pauses, stops or backfills runs. It fits teams that need coordinated batch pipelines (ETL, big-data, ML preprocessing) rather than a single script runner.

How a workflow works

Users build workflows in a drag-and-drop Web UI and choose from built-in task types; the project also offers a Python SDK and an Open API for creating and managing workflows programmatically workflow UI and built-in tasks, Python SDK. The scheduler tracks complex dependencies between tasks, and the UI shows both the DAG and an abstract tree view of task relationships workflow tree view. A data-source view provides unified access to external systems such as MySQL, PostgreSQL, Hive and Trino data-source support, and a monitor page reports master, worker and database status with resource usage and load monitor page.

Operating model

The documented deployment modes are Standalone, Cluster, Docker and Kubernetes, plus a Terraform path deployment options, Kubernetes guide, Terraform. Cluster mode uses a decentralized multi-master, multi-worker architecture with native horizontal scaling, according to the project, and claims capacity for tens of millions of tasks per day README feature list. Workflows and individual workflow instances, including tasks, are versioned; runs can be paused, stopped and recovered; multi-tenancy and project/data-source permission control are available.

Requirements and caveats

The project targets enterprise-scale, distributed environments; the advertised throughput and "several times faster than other orchestration platforms" claim is the author's statement without conditions or benchmark details README. Platform and resource prerequisites differ by deployment mode, so check the mode-specific guide before committing. Monitoring, tenancy and permission features imply an operational team and external database infrastructure, but the README excerpt does not give minimum hardware, supported versions or upgrade/rollback procedures. The Apache-2.0 license permits commercial use, modification and redistribution under its terms Apache-2.0.

Evidence from users

A 2022 commenter described using it for over a year to connect a big-data platform and a machine-learning platform and found it stable; another described it as suited to enterprise scenarios with visualized tasks and DAG dependencies user comment, comment. These are unverified opinions from older versions, not current validation.