PyDolphinScheduler is python API for Apache DolphinScheduler, which allow you definition your workflow by python code, aka workflow-as-codes.
At Nov 7, 2022 we seperated PyDolphinScheduler from DolphinScheduler, and the version of PyDolphinScheduler 4.0.0 can match multiple versions of DolphinScheduler, for more details, please refer to version
# Install python -m pip install apache-dolphinscheduler # Verify installation is successful, it will show the version of apache-dolphinscheduler, here we use 0.1.0 as example pydolphinscheduler version # 0.1.0
NOTE: package apache-dolphinscheduler not work on above Python version 3.10(including itself) in Window operating system due to dependence py4j not work on those environments.
Here we show you how to install and run a simple example of PyDolphinScheduler
There are many ways to start DolphinScheduler, here we use docker to start and run it as a standalone server.
# Change the version of dolphinscheduler to the version you want to use, here we use 3.1.1 as example DOLPHINSCHEDULER_VERSION=3.1.1 docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -e API_PYTHON_GATEWAY_ENABLED="true" -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"
After the container is started, you can access the DolphinScheduler UI via http://localhost:12345/dolphinscheduler. For more way to start DolphinScheduler and the more detail about DolphinScheduler, please refer to DolphinScheduler
We have many examples in examples directory, we here pick up a typical one to show how to run it.
# Get the latest code of example from github curl https://raw.githubusercontent.com/apache/dolphinscheduler-sdk-python/main/src/pydolphinscheduler/examples/tutorial.py -o ./tutorial.py # Change tenant to real exists tenant in the host your DolphinScheduler running, by any editor you like # Run the example python ./tutorial.py
NOTICE: Since Apache DolphinScheduler's tenant is requests while running command, you have to change tenant value in file tutorial.py. The default value is
tenant_exists
, change it to username exists your host.
After that, a new workflow will be created by PyDolphinScheduler, and you can see it in DolphinScheduler web UI‘s Project Management page. It will trigger the workflow automatically, so you can see the workflow running in DolphinScheduler web UI’s Workflow Instance page too. For more detail about any function about DolphinScheduler Project Management, please refer to DolphinScheduler Workflow
For full documentation visit document. This documentation is generated from this repository so please raise issues or pull requests for any additions, corrections, or clarifications.
If you would like to contribute, check out the open issues on GitHub. You can also see the guide to contributing.
Follow the release guide to release a new version of PyDolphinScheduler.