Apache Ambari plugin of Apache DolphinScheduler

⚠️⚠️Important⚠️⚠️

⚠️⚠️ NOTE ⚠️⚠️: Apache Ambari plugin only support installation DolphinScheduler for version 1.3.0 - 1.3.9. The reason we split to this repo as below:

Ambari plug-in is a deployment plugin and not update for two years, and only support DolphinScheduler 1.3.x. DolphinScheduler 2.0 have quit different between 1.3.x and it seem not work in 2.0 anymore.

You could find more detail in mail thread: https://lists.apache.org/thread/5l2zy28q1l5tl87q84h8qsw59z5vg2ly

Note

This document is intended for users with a basic understanding of Ambari
This document is a description of adding the Dolphin Scheduler service to the installed Ambari service
This document is based on version 2.5.2 of Ambari

Installation preparation

Prepare the RPM packages

# Only support version 1.3.0 - 1.3.9, we here use 1.3.3 as example
DOLPHINSCHEDULER_VERSION=1.3.3

git clone --depth 1 --branch ${DOLPHINSCHEDULER_VERSION} git@github.com:apache/dolphinscheduler.git dolphinscheduler_${DOLPHINSCHEDULER_VERSION}_ambari
cd dolphinscheduler_${DOLPHINSCHEDULER_VERSION}_ambari

# It is generated by executing the command, the project root directory (In the directory: dolphinscheduler-dist/target/rpm/apache-dolphinscheduler/RPMS/noarch )
mvn -U clean install  -Prpmbuild  -Dmaven.test.skip=true -X

Create an installation for DS，who have read and write access to the installation directory (/opt/soft)

Install with rpm package

Manual installation (recommended)：
- Copy the prepared RPM packages to each node of the cluster.
- Execute with DS installation user: rpm -ivh apache-dolphinscheduler-xxx.noarch.rpm
- Mysql-connector-java packaged using the default POM file will not be included.
- The RPM package was packaged in the project with the installation path of /opt/soft. If you use mysql as the database, you need add it manually.
Automatic installation with ambari
- Each node of the cluster needs to configure the local yum source
- Copy the prepared RPM packages to each node local yum source

Copy plug-in directory

copy directory ambari_plugin/common-services/DOLPHIN to ambari-server/resources/common-services/
copy directory ambari_plugin/statcks/DOLPHIN to ambari-server/resources/stacks/HDP/2.6/services/--stack version is selected based on the actual situation

Initializes the database information

-- Create the database for the Dolphin Scheduler：dolphinscheduler
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE
utf8_general_ci;

-- Initialize the user and password for the dolphinscheduler database and assign permissions
-- Replace the {user} in the SQL statement below with the user of the dolphinscheduler database
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY
'{password}';
flush privileges;

Ambari Install Dolphin Scheduler

NOTE: You have to install zookeeper first

Install Dolphin Scheduler on ambari web interface

Select the nodes for the Dolphin Scheduler's Master installation

Configure the Dolphin Scheduler's nodes for Worker, Api, Logger, Alert installation

Set the installation users of the Dolphin Scheduler service (created in step 1) and the user groups they belong to

System Env Optimization will export some system environment config. Modify according to actual situation

Configure the database information (same as in the initialization database in step 1)

Configure additional information if needed

Perform the next steps as normal

The interface after successful installation

Add components to the node through Ambari -- for example, add a DS Worker

NOTE: DS Logger is the installation dependent component of DS Worker in Dolphin's Ambari installation (need to add installation first; Prevent the Job log on the corresponding Worker from being checked)