If you want to use the resource upload function, you can select the local file directory for a single machine(this operation does not need to deploy Hadoop). Or you can also upload to a Hadoop or MinIO cluster, at this time, you need to have Hadoop (2.6+) or MinIO and other related environments
Note:
- If the resource upload function is used, the deployment user in installation and deployment must to have operation authority
- If you using Hadoop cluster with HA, you need to enable HDFS resource upload, and you need to copy the
core-site.xmlandhdfs-site.xmlunder the Hadoop cluster to/opt/dolphinscheduler/conf, otherwise Skip step
conf/common/common.properties
# Users who have permission to create directories under the HDFS root path
hdfs.root.user=hdfs
# data base dir, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions。"/escheduler" is recommended
data.store2hdfs.basepath=/dolphinscheduler
# resource upload startup type : HDFS,S3,NONE
res.upload.startup.type=HDFS
# whether kerberos starts
hadoop.security.authentication.startup.state=false
# java.security.krb5.conf path
java.security.krb5.conf.path=/opt/krb5.conf
# loginUserFromKeytab user
login.user.keytab.username=hdfs-mycluster@ESZ.COM
# loginUserFromKeytab path
login.user.keytab.path=/opt/hdfs.headless.keytab
conf/common/hadoop.properties
# ha or single namenode,If namenode ha needs to copy core-site.xml and hdfs-site.xml
# to the conf directory，support s3，for example : s3a://dolphinscheduler
fs.defaultFS=hdfs://mycluster:8020
#resourcemanager ha note this need ips , this empty if single
yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx
# If it is a single resourcemanager, you only need to configure one host name. If it is resourcemanager HA, the default configuration is fine
yarn.application.status.address=http://xxxx:8088/ws/v1/cluster/apps/%s
It is the management of various resource files, including creating basic txt/log/sh/conf/py/java and other files, uploading jar packages and other types of files, and can do edit, rename, download, delete and other operations.
The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties
Upload file: Click the “Upload File” button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name
For the file types that can be viewed, click the file name to view the file details
Click the “Download” button in the file list to download the file or click the “Download” button in the upper right corner of the file details to download the file
delete
File list -> Click the “Delete” button to delete the specified file
Re-upload file
Re-upload file: Click the “Re-upload File” button to upload a new file to replace the old file, drag the file to the re-upload area, the file name will be automatically completed with the new file name
The resource management and file management functions are similar. The difference is that the resource management is the uploaded UDF function, and the file management uploads the user program, script and configuration file. Operation function: rename, download, delete.
Same as uploading files.
Click “Create UDF Function”, enter the udf function parameters, select the udf resource, and click “Submit” to create the udf function.
Currently only supports temporary UDF functions of HIVE
The task group is mainly used to control the concurrency of task instances, and is designed to control the pressure of other resources (it can also control the pressure of the Hadoop cluster, the cluster will have queue control it). When creating a new task definition, you can configure the corresponding task group and configure the priority of the task running in the task group.
The user clicks [Resources] - [Task Group Management] - [Task Group option] - create task Group
You need to enter the information in the picture:
[Task group name]: The name displayed when the task group is used
[Project name]: The project that the task group functions, this item is optional, if not selected, all the projects in the whole system can use this task group.
[Resource pool size]: The maximum number of concurrent task instances allowed
Click the button to view task group usage information
Note: The use of task groups is applicable to tasks executed by workers, such as [switch] nodes, [condition] nodes, [sub_process] and other node types executed by the master are not controlled by the task group. Let's take the shell node as an example:
Regarding the configuration of the task group, all you need to do is to configure the part in the red box:
[Task group name] : The task group name displayed on the task group configuration page. Here you can only see the task group that the project has permission to (the project is selected when creating a task group), or the task group that acts globally (the new task group is created). when no item is selected)
[Priority] : When there is a waiting resource, the task with high priority will be distributed to the worker by the master first. The larger the value of this part, the higher the priority.
The master judges whether the task is configured with a task group when distributing the task. If the task is not configured, it is normally thrown to the worker to run; if a task group is configured, it checks whether the remaining size of the task group resource pool meets the current task operation before throwing it to the worker for execution. , if the resource pool -1 is satisfied, continue to run; if not, exit the task distribution and wait for other tasks to wake up.
When the task that has obtained the task group resource ends, the task group resource will be released. After the release, it will check whether there is a task waiting in the current task group. If there is, mark the task with the best priority to run, and create a new executable event. . The event stores the task id that is marked to obtain the resource, and then obtains the task group resource and then runs it.