The purpose of the pseudo-cluster deployment is to deploy the DolphinScheduler service on a single machine. In this mode, DolphinScheduler's master, worker, API server, are all on the same machine.
If you are a new hand and want to experience DolphinScheduler functions, we recommend you install follow Standalone deployment. If you want to experience more complete functions and schedule massive tasks, we recommend you install follow pseudo-cluster deployment. If you want to deploy DolphinScheduler in production, we recommend you follow cluster deployment or Kubernetes deployment.
Pseudo-cluster deployment of DolphinScheduler requires external software support:
JAVA_HOME to and PATH variable. You can skip this step, if it already exists in your environment.pstree for macOSpsmisc for Fedora/Red/Hat/CentOS/Ubuntu/DebianNote: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but if you need to run tasks that depend on them, you need to have the corresponding environment support.
Create a deployment user, and make sure to configure sudo without password. Here make an example to create user dolphinscheduler:
# To create a user, login as root useradd dolphinscheduler # Add password echo "dolphinscheduler" | passwd --stdin dolphinscheduler # Configure sudo without password sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers # Modify directory permissions and grant permissions for user you created above chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin
NOTICE:
- Due to DolphinScheduler's multi-tenant task switch user using command
sudo -u {linux-user}, the deployment user needs to havesudoprivileges and be password-free. If novice learners don’t understand, you can ignore this point for now.- If you find the line “Defaults requirett” in the
/etc/sudoersfile, please comment the content.
Since resources need to be sent to different machines during installation, SSH password-free login is required between each machine. The following shows the steps to configure password-free login:
su dolphinscheduler ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys
Notice: After the configuration is complete, you can run the command
ssh localhostto test works or not. If you can login with ssh without password stands for successful.
Go to the ZooKeeper installation directory, copy configure file zoo_sample.cfg to conf/zoo.cfg, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper.
# Start ZooKeeper ./bin/zkServer.sh start
After completing the preparation of the basic environment, you need to modify the configuration file according to your environment. The configuration file is in the path of conf/config/install_config.conf. Generally, you just need to modify the INSTALL MACHINE, DolphinScheduler ENV, Database, Registry Server part to complete the deployment, the following describes the parameters that must be modified:
# --------------------------------------------------------- # INSTALL MACHINE # --------------------------------------------------------- # Due to the master, worker, and API server being deployed on a single node, the IP of the server is the machine IP or localhost ips="localhost" masters="localhost" workers="localhost:default" alertServer="localhost" apiServers="localhost" # DolphinScheduler installation path, it will auto-create if not exists installPath="~/dolphinscheduler" # Deploy user, use the user you create in section **Configure machine SSH password-free login** deployUser="dolphinscheduler" # --------------------------------------------------------- # DolphinScheduler ENV # --------------------------------------------------------- # The path of JAVA_HOME, which JDK install path in section **Preparation** javaHome="/your/java/home/here" # --------------------------------------------------------- # Database # --------------------------------------------------------- # Database type, username, password, IP, port, metadata. For now `dbtype` supports `mysql` and `postgresql` dbtype="mysql" dbhost="localhost:3306" # Need to modify if you are not using `dolphinscheduler/dolphinscheduler` as your username and password username="dolphinscheduler" password="dolphinscheduler" dbname="dolphinscheduler" # --------------------------------------------------------- # Registry Server # --------------------------------------------------------- # Registration center address, the address of ZooKeeper service registryServers="localhost:2181"
DolphinScheduler metadata is stored in the relational database. Currently, supports PostgreSQL and MySQL. If you use MySQL, you need to manually download mysql-connector-java driver (8.0.16) and move it to the lib directory of DolphinScheduler. Let's take MySQL as an example for how to initialize the database:
mysql -uroot -p mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; # Change {user} and {password} by requests mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}'; mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}'; mysql> flush privileges;
After the above steps done you would create a new database for DolphinScheduler, then run Shell scripts to init database:
sh script/create-dolphinscheduler.sh
Use deployment user you created above, running the following command to complete the deployment, and the server log will be stored in the logs folder.
sh install.sh
Note: For the first time deployment, there maybe occur five times of
sh: bin/dolphinscheduler-daemon.sh: No such file or directoryin the terminal, this is non-important information that you can ignore.
Access address http://localhost:12345/dolphinscheduler and login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123
# Stop all DolphinScheduler server sh ./bin/stop-all.sh # Start all DolphinScheduler server sh ./bin/start-all.sh # Start or stop DolphinScheduler Master sh ./bin/dolphinscheduler-daemon.sh stop master-server sh ./bin/dolphinscheduler-daemon.sh start master-server # Start or stop DolphinScheduler Worker sh ./bin/dolphinscheduler-daemon.sh start worker-server sh ./bin/dolphinscheduler-daemon.sh stop worker-server # Start or stop DolphinScheduler Api sh ./bin/dolphinscheduler-daemon.sh start api-server sh ./bin/dolphinscheduler-daemon.sh stop api-server # Start or stop Alert sh ./bin/dolphinscheduler-daemon.sh start alert-server sh ./bin/dolphinscheduler-daemon.sh stop alert-server
Note:: Please refer to the section of “System Architecture Design” for service usage