Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Clone this repo:
  1. bfb9f17 [Fix-17436][Workflow]Task timeout kill throw exception (#17437) by njnu-seafish · 31 hours ago dev
  2. 8ebc6f3 [Fix-17469]Fix threadLocal will not clean if exception occur in LoginHandlerInterceptor (#17474) by njnu-seafish · 2 days ago
  3. 4ec7c4b [Improvement-16994][TaskPlugin] support retry for every api call for serverless spark (#17476) by Evan · 3 days ago
  4. 90a8cdc fix flaky ci (#17479) by xiangzihao · 4 days ago
  5. 590d16f [Fix-17473] [Task Plugin] Fix Serverless Spark Task final state incorrect for failure (#17475) by Eric Gao · 6 days ago

Apache Dolphinscheduler

License codecov Quality Gate Status Twitter Follow Slack Status CN doc

About

Apache DolphinScheduler is a modern data orchestration platform that empowers agile, low-code development of high-performance workflows. It is dedicated to handling complex task dependencies in data pipelines, and provides a wide range of built-in job types ** out of the box**

Key features for DolphinScheduler are as follows:

  • Easy to deploy, provides four deployment modes including Standalone, Cluster, Docker and Kubernetes.
  • Easy to use, workflows can be created and managed via Web UI, Python SDK or Open API
  • Highly reliable and high availability, with a decentralized, multi-master and multi-worker architecture and native supports for horizontal scaling.
  • High performance, its performance is several times faster than other orchestration platforms, and it is capable of handling tens of millions of tasks per day
  • Cloud Native, DolphinScheduler supports orchestrating workflows cross multiple clouds and data centers, and allows custom task types
  • Workflow Versioning, provides version control for both workflows and individual workflow instances, including tasks.
  • Flexible state control of workflows and tasks, supports pause/stop/recover them in any time
  • Multi-tenancy support
  • Additional features, backfill support(Web UI native), permission control including project and data source etc.

QuickStart

User Interface Screenshots

  • Homepage: Project and workflow overview, including the latest workflow instance and task instance status statistics. home

  • Workflow Definition: Create and manage workflows by drag and drop, easy to build and maintain complex workflows, support a wide range of tasks out of box. workflow-definition

  • Workflow Tree View: Abstract tree structure could provide a clearer understanding of task relationships workflow-tree

  • Data source: Supports multiple external data sources, provides unified data access capabilities for MySQL, PostgreSQL, Hive, Trino, etc. data-source

  • Monitor: View the status of the master, worker and database in real time, including server resource usage and load, do quick health check without logging in to the server. monitor

Suggestions & Bug Reports

Follow this guide to report your suggestions or bugs.

Contributing

The community welcomes contributions from everyone. Please refer to this page to find out more details: How to contribute. Check out good first issue in here if you are new to DolphinScheduler.

Community

Welcome to join the Apache DolphinScheduler community by:

Landscapes