Apache Ambari plugin for Apache DolphinScheduler

⚠️⚠️ Important ⚠️⚠️

⚠️⚠️ NOTE ⚠️⚠️: Apache Ambari plugin only support installation DolphinScheduler for version 1.3.0 - 1.3.9. The reason we split to this repo as below:

  • Ambari plug-in is a deployment plugin and not update for two years, and only support DolphinScheduler 1.3.x. DolphinScheduler 2.0 have quit different between 1.3.x and it seem not work in 2.0 anymore.

You could find more detail in mail thread: https://lists.apache.org/thread/5l2zy28q1l5tl87q84h8qsw59z5vg2ly

Note

  • This document is intended for users with a basic understanding of Ambari
  • This document is a description of adding the Dolphin Scheduler service to the installed Ambari service
  • This document is based on version 2.5.2 of Ambari

Installation preparation

Prepare the RPM packages

# Only support version 1.3.0 - 1.3.9, we here use 1.3.3 as example
DOLPHINSCHEDULER_VERSION=1.3.3

git clone --depth 1 --branch ${DOLPHINSCHEDULER_VERSION} git@github.com:apache/dolphinscheduler.git dolphinscheduler_${DOLPHINSCHEDULER_VERSION}_ambari
cd dolphinscheduler_${DOLPHINSCHEDULER_VERSION}_ambari

# It is generated by executing the command, the project root directory (In the directory: dolphinscheduler-dist/target/rpm/apache-dolphinscheduler/RPMS/noarch )
mvn -U clean install  -Prpmbuild  -Dmaven.test.skip=true -X

Create an installation for DS,who have read and write access to the installation directory (/opt/soft)

Install with rpm package

  • Manual installation (recommended):

    • Copy the prepared RPM packages to each node of the cluster.
    • Execute with DS installation user: rpm -ivh apache-dolphinscheduler-xxx.noarch.rpm
    • Mysql-connector-java packaged using the default POM file will not be included.
    • The RPM package was packaged in the project with the installation path of /opt/soft. If you use mysql as the database, you need add it manually.
  • Automatic installation with ambari

    • Each node of the cluster needs to configure the local yum source
    • Copy the prepared RPM packages to each node local yum source

Copy plug-in directory

  • copy directory ambari_plugin/common-services/DOLPHIN to ambari-server/resources/common-services/
  • copy directory ambari_plugin/statcks/DOLPHIN to ambari-server/resources/stacks/HDP/2.6/services/--stack version is selected based on the actual situation

Initializes the database information

-- Create the database for the Dolphin Schedulerdolphinscheduler
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE
utf8_general_ci;

-- Initialize the user and password for the dolphinscheduler database and assign permissions
-- Replace the {user} in the SQL statement below with the user of the dolphinscheduler database
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY
'{password}';
flush privileges;

Ambari Install Dolphin Scheduler(You have to install zookeeper first)

  • Install Dolphin Scheduler on ambari web interface

  • Select the nodes for the Dolphin Scheduler's Master installation

  • Configure the Dolphin Scheduler's nodes for Worker, Api, Logger, Alert installation

  • Set the installation users of the Dolphin Scheduler service (created in step 1) and the user groups they belong to

  • System Env Optimization will export some system environment config. Modify according to actual situation

  • Configure the database information (same as in the initialization database in step 1)

  • Configure additional information if needed

  • Perform the next steps as normal

  • The interface after successful installation


Add components

Add components to the node through Ambari -- for example, add a DS Worker

NOTE: DS Logger is the installation dependent component of DS Worker in Dolphin's Ambari installation (need to add installation first; Prevent the Job log on the corresponding Worker from being checked)

  • Locate the component node to add -- for example, node ark3

DS2_AMBARI_011

  • Add components -- the drop-down list is all addable

DS2_AMBARI_012

  • Confirm component addition

DS2_AMBARI_013

  • After adding DS Worker and DS Logger components

DS2_AMBARI_015

  • Start the component

DS2_AMBARI_016

Remove components

Remove the component from the node with Ambari

  • Stop the component in the corresponding node

DS2_AMBARI_018

  • Remove components

DS2_AMBARI_019