[Improve] Fix ambari issue (#7)
diff --git a/README.md b/README.md
index b8c5cff..aec394a 100644
--- a/README.md
+++ b/README.md
@@ -1,148 +1,48 @@
# Apache Ambari plugin for Apache DolphinScheduler
-## ⚠️⚠️ Important ⚠️⚠️
+## Service commissioning tips
+A very important command to restart the ambira service：ambari-server restart
+ambari plugin configuration path：/var/lib/ambari-server/resources/common-services/DOLPHIN/2.0.6
+ambari cache path：/var/lib/ambari-agent/cache/common-services/
-⚠️⚠️ **NOTE** ⚠️⚠️: Apache Ambari plugin only support installation DolphinScheduler for version **1.3.0 - 1.3.9**.
-The reason we split to this repo as below:
+## Related Knowledge
-* Ambari plug-in is a deployment plugin and not update for two years, and only support DolphinScheduler 1.3.x.
- DolphinScheduler 2.0 have quit different between 1.3.x and it seem not work in 2.0 anymore.
+### Ambari python the use of dependency packages is explained in detail.
-You could find more detail in mail thread: https://lists.apache.org/thread/5l2zy28q1l5tl87q84h8qsw59z5vg2ly
+1、Dependency Package Description
+This module is located in /usr/lib/ambari- server/lib/resource_management and is also available in the /usr/lib/ambari-agent/lib entry.
+There is also a resource_management entry in the /usr/lib/ambari-agent/lib entry, which is the same as the resource_management entry in the /usr/lib/ambari-agent/lib entry.
+These two entries are the same, and ambari's custom service application environment relies on this entry.
+
-## Note
+2、How do I get the configuration information on the custom service page?
+When we open the configuration page of a custom service on the Ambari web page, we see that the configuration displayed above is the same as the contents of the xml file under configuration.
+After a custom service is deployed, the contents of the xml file will be displayed on the front-end for users to view and modify, and the modified configuration information will be saved to the database.
+How do we get the value of a particular configuration on the page? This is where we need to rely on the resource_management python package, which encapsulates a number of methods that we can directly invoke.
+For example, to get the value of the elastic_user property in elastic-env.xml, we just need to do this
+
-* This document is intended for users with a basic understanding of Ambari
-* This document is a description of adding the Dolphin Scheduler service to the installed Ambari service
-* This document is based on version 2.5.2 of Ambari
+3、Ambari cluster-related configuration information is also available at
+With Script.get_config(), we can also get some configuration information about the Ambari cluster, such as java version, hostname, etc.
+
-## Installation preparation
+4、A more friendly way to get property values than Script.get_config()
+In the dependency package resource_management, there is a better function, default(), to get property values in a more friendly way.
+If you have been using Script.get_config() for a long time, you may find that the Ambari custom service will report an error if the property obtained by Script.get_config() does not exist, and you will not be able to close the operation.
+However, the defult() method is much better, and the two methods are described below.
+from resource_management.libraries.functions.default import default
+
+Above, we know how to get the value of the attribute in the xml file in two ways: Script.get_config() and default().
-### Prepare the RPM packages
+5、It would be great if the entire custom service could be written entirely in python, but there are some scenarios where shell statements are more convenient, so the dependency package resource_management also supports the ability to execute shell statements in python, as shown below.
+from resource_management.core.resources.system import Execute
+
+The Execute() method also specifies whether the execution user and failure are to be ignored, as shown in the following figure.
+
-```shell
-# Only support version 1.3.0 - 1.3.9, we here use 1.3.3 as example
-DOLPHINSCHEDULER_VERSION=1.3.3
+6、Create/delete folders, files within py script
+6.1 Creating Folders, Files
+
+6.2 Delete Folders, Files
+
-git clone --depth 1 --branch ${DOLPHINSCHEDULER_VERSION} git@github.com:apache/dolphinscheduler.git dolphinscheduler_${DOLPHINSCHEDULER_VERSION}_ambari
-cd dolphinscheduler_${DOLPHINSCHEDULER_VERSION}_ambari
-
-# It is generated by executing the command, the project root directory (In the directory: dolphinscheduler-dist/target/rpm/apache-dolphinscheduler/RPMS/noarch )
-mvn -U clean install -Prpmbuild -Dmaven.test.skip=true -X
-```
-
-Create an installation for DS，who have read and write access to the installation directory (/opt/soft)
-
-### Install with rpm package
-
-* Manual installation (recommended)：
- * Copy the prepared RPM packages to each node of the cluster.
- * Execute with DS installation user: `rpm -ivh apache-dolphinscheduler-xxx.noarch.rpm`
- * Mysql-connector-java packaged using the default POM file will not be included.
- * The RPM package was packaged in the project with the installation path of /opt/soft.
- If you use mysql as the database, you need add it manually.
-
-* Automatic installation with ambari
- * Each node of the cluster needs to configure the local yum source
- * Copy the prepared RPM packages to each node local yum source
-
-### Copy plug-in directory
-
-* copy directory ambari_plugin/common-services/DOLPHIN to ambari-server/resources/common-services/
-* copy directory ambari_plugin/statcks/DOLPHIN to ambari-server/resources/stacks/HDP/2.6/services/--stack version is selected based on the actual situation
-
-### Initializes the database information
-
-```shell
--- Create the database for the Dolphin Scheduler：dolphinscheduler
-CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE
-utf8_general_ci;
-
--- Initialize the user and password for the dolphinscheduler database and assign permissions
--- Replace the {user} in the SQL statement below with the user of the dolphinscheduler database
-GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
-GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY
-'{password}';
-flush privileges;
-```
-
-## Ambari Install Dolphin Scheduler(You have to install zookeeper first)
-
-* Install Dolphin Scheduler on ambari web interface
-
-
-
-* Select the nodes for the Dolphin Scheduler's Master installation
-
-
-
-* Configure the Dolphin Scheduler's nodes for Worker, Api, Logger, Alert installation
-
-
-
-* Set the installation users of the Dolphin Scheduler service (created in step 1) and the user groups they belong to
-
-
-
-* System Env Optimization will export some system environment config. Modify according to actual situation
-
-
-
-* Configure the database information (same as in the initialization database in step 1)
-
-
-
-* Configure additional information if needed
-
-
-
-
-
-* Perform the next steps as normal
-
-
-
-* The interface after successful installation
-
-
-
-------
-
-## Add components
-
-Add components to the node through Ambari -- for example, add a DS Worker
-
-***NOTE***: DS Logger is the installation dependent component of DS Worker in Dolphin's Ambari installation
-(need to add installation first; Prevent the Job log on the corresponding Worker from being checked)
-
-* Locate the component node to add -- for example, node ark3
-
-
-
-* Add components -- the drop-down list is all addable
-
-
-
-* Confirm component addition
-
-
-
-* After adding DS Worker and DS Logger components
-
-
-
-* Start the component
-
-
-
-## Remove components
-
-Remove the component from the node with Ambari
-
-* Stop the component in the corresponding node
-
-
-
-* Remove components
-
-
diff --git a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-api.xml b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-api.xml
index 766c0f4..ede9abb 100644
--- a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-api.xml
+++ b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-api.xml
@@ -18,27 +18,22 @@
<property>
<name>server.port</name>
<value>12345</value>
- <description>
- server port
- </description>
- <value-attributes>
- <type>int</type>
- </value-attributes>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>server.servlet.session.timeout</name>
<value>7200</value>
- <value-attributes>
- <type>int</type>
- </value-attributes>
- <description>
- </description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>server.servlet.context-path</name>
<value>/dolphinscheduler/</value>
- <description>
- </description>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>spring.jackson.time-zone</name>
+ <value>GMT+8</value>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>spring.servlet.multipart.max-file-size</name>
@@ -47,8 +42,7 @@
<unit>MB</unit>
<type>int</type>
</value-attributes>
- <description>
- </description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>spring.servlet.multipart.max-request-size</name>
@@ -57,31 +51,36 @@
<unit>MB</unit>
<type>int</type>
</value-attributes>
- <description>
- </description>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>server.compression.enabled</name>
+ <value>true</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>server.compression.mime-types</name>
+ <value>text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,application/json,application/xml</value>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>server.jetty.max-http-post-size</name>
<value>5000000</value>
- <value-attributes>
- <type>int</type>
- </value-attributes>
- <description>
- </description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>spring.messages.encoding</name>
<value>UTF-8</value>
- <description></description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>spring.messages.basename</name>
<value>i18n/messages</value>
- <description></description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>security.authentication.type</name>
<value>PASSWORD</value>
- <description></description>
+ <on-ambari-upgrade add="true"/>
</property>
</configuration>
\ No newline at end of file
diff --git a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-mysql.xml b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-mysql.xml
index a3ba6ad..cbcb98f 100644
--- a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-mysql.xml
+++ b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-application-mysql.xml
@@ -18,20 +18,16 @@
<property>
<name>spring.datasource.url</name>
<value>jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8</value>
- <description>
- mysql url address
- </description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>spring.datasource.username</name>
- <value>ds_user</value>
- <description>
- </description>
+ <value>root</value>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>spring.datasource.password</name>
- <value>dolphinscheduler</value>
- <description>
- </description>
+ <value>123456</value>
+ <on-ambari-upgrade add="true"/>
</property>
</configuration>
\ No newline at end of file
diff --git a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-common.xml b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-common.xml
index f0dafab..8081666 100644
--- a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-common.xml
+++ b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-common.xml
@@ -17,168 +17,117 @@
<configuration>
<property>
<name>data.basedir.path</name>
- <value>/tmp/dolphinscheduler</value>
- <description>
- user data local directory path, please make sure the directory exists and have read write permissions
- </description>
+ <value>/data/dolphinscheduler</value>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>resource.storage.type</name>
- <display-name>Choose Resource Upload Startup Type</display-name>
- <description>
- resource storage type: HDFS, S3, NONE
- </description>
- <value>NONE</value>
- <value-attributes>
- <type>value-list</type>
- <entries>
- <entry>
- <value>HDFS</value>
- <label>HDFS</label>
- </entry>
- <entry>
- <value>S3</value>
- <label>S3</label>
- </entry>
- <entry>
- <value>NONE</value>
- <label>NONE</label>
- </entry>
- </entries>
- <selection-cardinality>1</selection-cardinality>
- </value-attributes>
+ <value>HDFS</value>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>resource.upload.path</name>
<value>/dolphinscheduler</value>
- <description>
- resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>hadoop.security.authentication.startup.state</name>
<value>false</value>
- <value-attributes>
- <type>value-list</type>
- <entries>
- <entry>
- <value>true</value>
- <label>Enabled</label>
- </entry>
- <entry>
- <value>false</value>
- <label>Disabled</label>
- </entry>
- </entries>
- <selection-cardinality>1</selection-cardinality>
- </value-attributes>
- <description>
- whether to startup kerberos
- </description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>java.security.krb5.conf.path</name>
<value>/opt/krb5.conf</value>
- <description>
- java.security.krb5.conf path
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>login.user.keytab.username</name>
<value>hdfs-mycluster@ESZ.COM</value>
- <description>
- login user from keytab username
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>login.user.keytab.path</name>
<value>/opt/hdfs.headless.keytab</value>
- <description>
- login user from keytab path
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>kerberos.expire.time</name>
<value>2</value>
- <description>
- kerberos expire time, the unit is hour
- </description>
- </property>
- <property>
- <name>resource.view.suffixs</name>
- <value>txt,log,sh,bat,conf,cfg,py,java,sql,xml,hql,properties,json,yml,yaml,ini,js</value>
- <description>
- resource view suffixs
- </description>
+ <on-ambari-upgrade add="true"/>
</property>
<property>
<name>hdfs.root.user</name>
<value>hdfs</value>
- <description>
- if resource.storage.type=HDFS, the user must have the permission to create directories under the HDFS root path
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>fs.defaultFS</name>
- <value>hdfs://mycluster:8020</value>
- <description>
- if resource.storage.type=S3, the value like: s3a://dolphinscheduler; if resource.storage.type=HDFS and namenode HA is enabled, you need to copy core-site.xml and hdfs-site.xml to conf dir
- </description>
+ <value>hdfs://testhdfsha:8020</value>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>fs.s3a.endpoint</name>
- <value>http://host:9010</value>
- <description>
- s3 required，s3 endpoint
- </description>
+ <value>http://192.168.xx.xx:9010</value>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>A3DXS30FO22544RE</value>
- <description>
- s3 required，s3 access key
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>OloCLq3n+8+sdPHUhJ21XrSxTC+JK</value>
- <description>
- s3 required，s3 secret key
- </description>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>resource.manager.httpaddress.port</name>
+ <value>8088</value>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>yarn.resourcemanager.ha.rm.ids</name>
- <value>192.168.xx.xx,192.168.xx.xx</value>
- <description>
- if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty
- </description>
+ <value>10.216.138.23,10.216.138.24</value>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>yarn.application.status.address</name>
<value>http://ds1:8088/ws/v1/cluster/apps/%s</value>
- <description>
- if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname
- </description>
<on-ambari-upgrade add="true"/>
</property>
<property>
- <name>dolphinscheduler.env.path</name>
- <value>env/dolphinscheduler_env.sh</value>
- <description>
- system env path
- </description>
+ <name>yarn.job.history.status.address</name>
+ <value>http://ds1:19888/ws/v1/history/mapreduce/jobs/%s</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>datasource.encryption.enable</name>
+ <value>false</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>datasource.encryption.salt</name>
+ <value>!@#$%^&*</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>support.hive.oneSession</name>
+ <value>true</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>sudo.enable</name>
+ <value>false</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>development.state</name>
+ <value>false</value>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property>
+ <name>datasource.plugin.dir</name>
+ <value>lib/plugin/datasource</value>
<on-ambari-upgrade add="true"/>
</property>
</configuration>
\ No newline at end of file
diff --git a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-env.xml b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-env.xml
index 18501c6..3f1223b 100644
--- a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-env.xml
+++ b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-env.xml
@@ -15,70 +15,17 @@
~ limitations under the License.
-->
<configuration>
- <property>
- <name>dolphin.database.type</name>
- <value>mysql</value>
- <description>Dolphin Scheduler DataBase Type Which Is Select</description>
- <display-name>Dolphin Database Type</display-name>
- <value-attributes>
- <type>value-list</type>
- <entries>
- <entry>
- <value>mysql</value>
- <label>Mysql</label>
- </entry>
- <entry>
- <value>postgresql</value>
- <label>Postgresql</label>
- </entry>
- </entries>
- <selection-cardinality>1</selection-cardinality>
- </value-attributes>
- <on-ambari-upgrade add="true"/>
- </property>
-
- <property>
- <name>dolphin.database.host</name>
- <value></value>
- <display-name>Dolphin Database Host</display-name>
- <on-ambari-upgrade add="true"/>
- </property>
-
- <property>
- <name>dolphin.database.port</name>
- <value></value>
- <display-name>Dolphin Database Port</display-name>
- <on-ambari-upgrade add="true"/>
- </property>
-
- <property>
- <name>dolphin.database.username</name>
- <value></value>
- <display-name>Dolphin Database Username</display-name>
- <on-ambari-upgrade add="true"/>
- </property>
-
- <property>
- <name>dolphin.database.password</name>
- <value></value>
- <display-name>Dolphin Database Password</display-name>
- <property-type>PASSWORD</property-type>
- <value-attributes>
- <type>password</type>
- </value-attributes>
- <on-ambari-upgrade add="true"/>
- </property>
<property>
<name>dolphin.user</name>
- <value></value>
+ <value>root</value>
<description>Which user to install and admin dolphin scheduler</description>
<display-name>Deploy User</display-name>
<on-ambari-upgrade add="true"/>
</property>
<property>
<name>dolphin.group</name>
- <value></value>
+ <value>root</value>
<description>Which user to install and admin dolphin scheduler</description>
<display-name>Deploy Group</display-name>
<on-ambari-upgrade add="true"/>
diff --git a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-registry.xml b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-registry.xml
index 7820520..345cc89 100644
--- a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-registry.xml
+++ b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/configuration/dolphin-registry.xml
@@ -31,7 +31,7 @@
</property>
<property>
<name>registry.namespace</name>
- <value>dolphinscheduler</value>
+ <value>dolphinscheduler-2.0.6</value>
<description>
</description>
<on-ambari-upgrade add="true"/>
diff --git a/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/role_command_order.json b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/role_command_order.json
new file mode 100644
index 0000000..8c07196
--- /dev/null
+++ b/ambari_plugin-2.0.6/common-services/DOLPHIN/2.0.6/role_command_order.json
@@ -0,0 +1,15 @@
+{
+ "general_deps": {
+ "DOLPHIN_MASTER_SERVICE-START": [
+ "DOLPHIN_API_SERVICE-START"
+ ],
+ "DOLPHIN_WORKER_SERVICE-START": [
+ "DOLPHIN_MASTER_SERVICE-START"
+ ],
+ "DOLPHIN_LOGGER_SERVICE-START": [
+ "DOLPHIN_MASTER_SERVICE-START",
+ "DOLPHIN_WORKER_SERVICE-START",
+ "DOLPHIN_API_SERVICE-START"
+ ]
+ }
+}
\ No newline at end of file
diff --git a/img/contents.png b/img/contents.png
new file mode 100644
index 0000000..8dc0272
--- /dev/null
+++ b/img/contents.png
Binary files differ
diff --git a/img/create folders.png b/img/create folders.png
new file mode 100644
index 0000000..1b1a2f8
--- /dev/null
+++ b/img/create folders.png
Binary files differ
diff --git a/img/delete folders.png b/img/delete folders.png
new file mode 100644
index 0000000..a93ae6a
--- /dev/null
+++ b/img/delete folders.png
Binary files differ
diff --git a/img/get cluster-related configuration information.png b/img/get cluster-related configuration information.png
new file mode 100644
index 0000000..9dbf2e5
--- /dev/null
+++ b/img/get cluster-related configuration information.png
Binary files differ
diff --git a/img/get configuration information.png b/img/get configuration information.png
new file mode 100644
index 0000000..cf57d86
--- /dev/null
+++ b/img/get configuration information.png
Binary files differ
diff --git a/img/get two ways to write the attribute value.png b/img/get two ways to write the attribute value.png
new file mode 100644
index 0000000..270ba7e
--- /dev/null
+++ b/img/get two ways to write the attribute value.png
Binary files differ
diff --git a/img/specify the execution user.png b/img/specify the execution user.png
new file mode 100644
index 0000000..90c6c3a
--- /dev/null
+++ b/img/specify the execution user.png
Binary files differ
diff --git a/img/using python to execute shell statements.png b/img/using python to execute shell statements.png
new file mode 100644
index 0000000..945745a
--- /dev/null
+++ b/img/using python to execute shell statements.png
Binary files differ
