Prerequisites

Build AMI

Set necessary variables by creating a file ds-ami.pkrvars.hcl and adding the following variables according to your own usage.

cat <<EOF > ds-ami.pkrvars.hcl
aws_access_key = ""
aws_secret_key = ""
aws_region     = "cn-north-1"


ds_ami_name = "my-test-ds-2"

# If you want to use the official distribution tar, just set the `ds_version` to the one you want.
ds_version  = "3.1.1"

# If you want to use a locally built distribution tar, set the `ds_tar` to the tar file location.
ds_tar      = "~/workspace/dolphinscheduler/dolphinscheduler-dist/target/apache-dolphinscheduler-3.1.3-SNAPSHOT-bin.tar.gz"
EOF

Then run the following command to initialize and build a custom AMI.

  • If you want to use the official distribution tar.
packer init --var-file=ds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl
packer build --var-file=ds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl
  • If you want to use the locally built distribution tar.
packer init --var-file=ds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl
packer build --var-file=ds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl

Create resources

Set necessary variables by creating a file terraform.tfvars and adding the following variables according to your own usage.

Make sure ds_ami_name is the same as the one in ds-ami.pkrvars.hcl above.

cat <<EOF > terraform.tfvars
aws_access_key = ""
aws_secret_key = ""
aws_region     = ""

name_prefix = "test-ds-terraform"
ds_ami_name = "my-test-ds"

ds_component_replicas = {
  master            = 1
  worker            = 1
  alert             = 1
  api               = 1
  standalone_server = 0
}
EOF

Then run the following commands to apply necessary resources.

terraform init -var-file=terraform.tfvars
terraform apply -var-file=terraform.tfvars -auto-approve

Open DolphinScheduler UI

open http://$(terraform output -json api_server_instance_public_dns | jq -r '.[0]'):12345/dolphinscheduler/ui

Inputs

NameDescriptionTypeDefaultRequired
aws_access_keyAWS access keystringn/ayes
aws_regionAWS regionstring"cn-north-1"no
aws_secret_keyAWS secret keystringn/ayes
db_instance_classDatabase instance classstring"db.t3.micro"no
db_passwordDatabase passwordstringn/ayes
db_usernameDatabase usernamestring"dolphinscheduler"no
ds_ami_nameName of DolphinScheduler AMIstring"dolphinscheduler-ami"no
ds_component_replicasReplicas of the DolphinScheduler Componentsmap(number){
“alert”: 1,
“api”: 1,
“master”: 1,
“standalone_server”: 0,
“worker”: 1
}		no
ds_versionDolphinScheduler Versionstring"3.1.1"no
name_prefixName prefix for all resourcesstring"dolphinscheduler"no
private_subnet_cidr_blocksAvailable CIDR blocks for private subnetslist(string)[
“10.0.101.0/24”,
“10.0.102.0/24”,
“10.0.103.0/24”,
“10.0.104.0/24”
]		no
public_subnet_cidr_blocksCIDR blocks for the public subnetslist(string)[
“10.0.1.0/24”,
“10.0.2.0/24”,
“10.0.3.0/24”,
“10.0.4.0/24”
]		no
s3_bucket_prefixn/astring"dolphinscheduler-test-"no
subnet_countNumber of subnetsmap(number){
“private”: 2,
“public”: 1
}		no
tagsTags to apply to all resourcesmap(string){
“Deployment”: “Test”
}		no
vm_associate_public_ip_addressAssociate a public IP address to the EC2 instancemap(bool){
“alert”: true,
“api”: true,
“master”: true,
“standalone_server”: true,
“worker”: true
}		no
vm_data_volume_sizeData volume size of the EC2 Instancemap(number){
“alert”: 10,
“api”: 10,
“master”: 10,
“standalone_server”: 10,
“worker”: 10
}		no
vm_data_volume_typeData volume type of the EC2 Instancemap(string){
“alert”: “gp2”,
“api”: “gp2”,
“master”: “gp2”,
“standalone_server”: “gp2”,
“worker”: “gp2”
}		no
vm_instance_typeEC2 instance typemap(string){
“alert”: “t2.micro”,
“api”: “t2.small”,
“master”: “t2.medium”,
“standalone_server”: “t2.small”,
“worker”: “t2.medium”
}		no
vm_root_volume_sizeRoot Volume size of the EC2 Instancemap(number){
“alert”: 30,
“api”: 30,
“master”: 30,
“standalone_server”: 30,
“worker”: 30
}		no
vm_root_volume_typeRoot volume type of the EC2 Instancemap(string){
“alert”: “gp2”,
“api”: “gp2”,
“master”: “gp2”,
“standalone_server”: “gp2”,
“worker”: “gp2”
}		no
vpc_cidrCIDR for the VPCstring"10.0.0.0/16"no
zookeeper_connect_stringZookeeper connect string, if empty, will create a single-node zookeeper for demonstration, don't use this in productionstring""no

Outputs

NameDescription
alert_server_instance_idInstance IDs of alert instances
alert_server_instance_private_ipPrivate IPs of alert instances
alert_server_instance_public_dnsPublic domain names of alert instances
alert_server_instance_public_ipPublic IPs of alert instances
api_server_instance_idInstance IDs of api instances
api_server_instance_private_ipPrivate IPs of api instances
api_server_instance_public_dnsPublic domain names of api instances
api_server_instance_public_ipPublic IPs of api instances
db_addressDatabase address
db_nameDatabase name
db_portDatabase port
master_server_instance_idInstance IDs of master instances
master_server_instance_private_ipPrivate IPs of master instances
master_server_instance_public_dnsPublic domain names of master instances
master_server_instance_public_ipPublic IPs of master instances
s3_access_keyS3 access key
s3_addressS3 address
s3_bucketS3 bucket name
s3_regional_domain_nameS3 regional domain name
s3_secretS3 access secret
vm_server_instance_idInstance IDs of standalone instances
vm_server_instance_private_ipPrivate IPs of standalone instances
vm_server_instance_public_dnsPublic domain names of standalone instances
vm_server_instance_public_ipPublic IPs of standalone instances
worker_server_instance_idInstance IDs of worker instances
worker_server_instance_private_ipPrivate IPs of worker instances
worker_server_instance_public_dnsPublic domain names of worker instances
worker_server_instance_public_ipPublic IPs of worker instances
zookeeper_server_instance_idInstance IDs of zookeeper instances
zookeeper_server_instance_private_ipPrivate IPs of zookeeper instances
zookeeper_server_instance_public_dnsPublic domain names of zookeeper instances
zookeeper_server_instance_public_ipPublic IPs of zookeeper instances