| --- |
| title: Quick Deployment Guide |
| --- |
| |
| # Quick Deployment Guide |
| |
| This guide provides a streamlined, copy-paste-friendly procedure for experienced administrators. This guide shows how to deploy a 5-node Apache Cloudberry cluster: |
| |
| - 1 Coordinator (`cdw`) |
| - 1 Standby Coordinator (`scdw`) |
| - 3 Segments (`sdw1`, `sdw2`, `sdw3`), each with 2 primary and 2 mirror instances |
| |
| :::info |
| For detailed explanations of each step, refer to the full deployment guides: |
| [Configuring Your Systems](./prepare-to-deploy.md), [Installing Using RPM/DEB Package](./install_cloudberry.md), [Creating the Data Storage Areas](./create_data_dirs.md), and [Initializing Apache Cloudberry](./init_cloudberry.md). |
| ::: |
| |
| ## Cluster layout |
| |
| | Host | Role | IP (example) | |
| |------|------|--------------| |
| | cdw | Coordinator | 192.168.1.1 | |
| | scdw | Standby Coordinator | 192.168.1.2 | |
| | sdw1 | Segment 1 | 192.168.1.3 | |
| | sdw2 | Segment 2 | 192.168.1.4 | |
| | sdw3 | Segment 3 | 192.168.1.5 | |
| |
| Data directories used: |
| |
| - Coordinator/Standby: `/data/coordinator` |
| - Segment primary: `/data/primary` (2 instances per host) |
| - Segment mirror: `/data/mirror` (2 instances per host) |
| |
| --- |
| |
| ## 1. Configuring your systems (as root on all hosts) |
| |
| ### 1.1 Deactivate SELinux |
| |
| :::note |
| This step applies to RHEL/Oracle/Rocky Linux only. On Ubuntu, SELinux is not installed by default and this step can be skipped. |
| ::: |
| |
| ```bash |
| # Check current status |
| sestatus |
| |
| # If not disabled, deactivate it |
| sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config |
| |
| # Reboot is required to take effect |
| ``` |
| |
| ### 1.2 Deactivate firewall |
| |
| **For RHEL/Rocky Linux (firewalld):** |
| |
| ```bash |
| # Check current status |
| systemctl status firewalld |
| |
| # If not disabled, deactivate it |
| systemctl stop firewalld.service |
| systemctl disable firewalld.service |
| ``` |
| |
| **For Ubuntu (ufw):** |
| |
| ```bash |
| # Check current status (disabled by default) |
| ufw status |
| |
| # If active, disable it |
| ufw disable |
| ``` |
| |
| ### 1.3 Set the hosts file |
| |
| Set the hostname on each host (takes effect immediately, no reboot required): |
| |
| ```bash |
| # Run on each host respectively |
| hostnamectl set-hostname cdw # on coordinator |
| hostnamectl set-hostname scdw # on standby coordinator |
| hostnamectl set-hostname sdw1 # on segment 1 |
| hostnamectl set-hostname sdw2 # on segment 2 |
| hostnamectl set-hostname sdw3 # on segment 3 |
| |
| # Verify |
| hostname |
| |
| # Refresh the shell prompt to show the new hostname |
| exec bash |
| ``` |
| |
| Edit `/etc/hosts` on all hosts: |
| |
| ```bash |
| cat >> /etc/hosts <<EOF |
| 192.168.1.1 cdw |
| 192.168.1.2 scdw |
| 192.168.1.3 sdw1 |
| 192.168.1.4 sdw2 |
| 192.168.1.5 sdw3 |
| EOF |
| ``` |
| |
| ### 1.4 Set sysctl.conf parameters |
| |
| ```bash |
| cat >> /etc/sysctl.conf <<EOF |
| kernel.shmall = 197951838 |
| kernel.shmmax = 810810728448 |
| kernel.shmmni = 4096 |
| vm.overcommit_memory = 2 |
| vm.overcommit_ratio = 95 |
| net.ipv4.ip_local_port_range = 10000 65535 |
| kernel.sem = 250 2048000 200 8192 |
| kernel.sysrq = 1 |
| kernel.core_uses_pid = 1 |
| kernel.msgmnb = 65536 |
| kernel.msgmax = 65536 |
| kernel.msgmni = 2048 |
| net.ipv4.tcp_syncookies = 1 |
| net.ipv4.conf.default.accept_source_route = 0 |
| net.ipv4.tcp_max_syn_backlog = 4096 |
| net.ipv4.conf.all.arp_filter = 1 |
| net.ipv4.ipfrag_high_thresh = 41943040 |
| net.ipv4.ipfrag_low_thresh = 31457280 |
| net.ipv4.ipfrag_time = 60 |
| net.core.netdev_max_backlog = 10000 |
| net.core.rmem_max = 2097152 |
| net.core.wmem_max = 2097152 |
| vm.swappiness = 10 |
| vm.zone_reclaim_mode = 0 |
| vm.dirty_expire_centisecs = 500 |
| vm.dirty_writeback_centisecs = 100 |
| vm.dirty_background_ratio = 0 |
| vm.dirty_ratio = 0 |
| vm.dirty_background_bytes = 1610612736 |
| vm.dirty_bytes = 4294967296 |
| kernel.core_pattern=/var/core/core.%h.%t |
| EOF |
| |
| mkdir -p /var/core |
| sysctl -p |
| ``` |
| |
| :::note |
| The `kernel.shmall` and `kernel.shmmax` values above are examples for a host with ~1583 GB memory. Calculate the correct values for your system: |
| |
| ```bash |
| echo $(expr $(getconf _PHYS_PAGES) / 2) # kernel.shmall |
| echo $(expr $(getconf _PHYS_PAGES) / 2 \* $(getconf PAGE_SIZE)) # kernel.shmmax |
| ``` |
| ::: |
| |
| ### 1.5 Set system resource limits |
| |
| ```bash |
| cat >> /etc/security/limits.conf <<EOF |
| * soft nofile 524288 |
| * hard nofile 524288 |
| * soft nproc 131072 |
| * hard nproc 131072 |
| * soft core unlimited |
| EOF |
| ``` |
| |
| - `nofile` — maximum number of open file descriptors |
| - `nproc` — maximum number of processes |
| - `core` — core dump file size (unlimited enables core dumps for debugging) |
| |
| ### 1.6 XFS mount options |
| |
| :::note |
| If you are using cloud VMs with only a single system disk (no additional data disk), you can skip the disk formatting and mounting steps below. Simply create the data directory on the existing filesystem: |
| |
| ```bash |
| mkdir -p /data |
| ``` |
| |
| The XFS formatting steps below apply when you have a dedicated data disk available. |
| ::: |
| |
| ```bash |
| mkdir -p /data |
| mkfs.xfs -f /dev/sdb # adjust device name as needed |
| |
| # Use device name directly |
| echo "/dev/sdb /data xfs rw,nodev,noatime,inode64 0 0" >> /etc/fstab |
| # Or use UUID for stable identification across reboots (recommended for cloud/VM environments) |
| # echo "UUID=$(blkid -s UUID -o value /dev/vdb1) /data xfs rw,nodev,noatime,inode64 0 0" >> /etc/fstab |
| |
| mount /data |
| ``` |
| |
| ### 1.7 Disk I/O settings |
| |
| ```bash |
| # Set read-ahead value |
| /sbin/blockdev --setra 16384 /dev/sdb |
| |
| # Persist read-ahead across reboots |
| echo '/sbin/blockdev --setra 16384 /dev/sdb' >> /etc/rc.d/rc.local |
| chmod +x /etc/rc.d/rc.local |
| |
| # Set I/O scheduler permanently (for non-NVMe/SSD disks) |
| # For RHEL/Rocky Linux: |
| grubby --update-kernel=ALL --args="elevator=mq-deadline" |
| # For Ubuntu: |
| sed -i 's/GRUB_CMDLINE_LINUX="\(.*\)"/GRUB_CMDLINE_LINUX="\1 elevator=mq-deadline"/' /etc/default/grub && update-grub |
| ``` |
| |
| ### 1.8 Disable Transparent Huge Pages |
| |
| ```bash |
| # Check current THP status |
| cat /sys/kernel/mm/*transparent_hugepage/enabled |
| |
| # If not [never], disable it permanently |
| # For RHEL/Rocky Linux: |
| grubby --update-kernel=ALL --args="transparent_hugepage=never" |
| # For Ubuntu: |
| sed -i 's/GRUB_CMDLINE_LINUX="\(.*\)"/GRUB_CMDLINE_LINUX="\1 transparent_hugepage=never"/' /etc/default/grub && update-grub |
| |
| # Reboot is required to take effect |
| ``` |
| |
| ### 1.9 Disable IPC object removal |
| |
| ```bash |
| sed -i 's/^#RemoveIPC=.*/RemoveIPC=no/' /etc/systemd/logind.conf |
| service systemd-logind restart |
| ``` |
| |
| ### 1.10 SSH connection threshold |
| |
| ```bash |
| sed -i 's/^#MaxStartups.*/MaxStartups 10:30:200/' /etc/ssh/sshd_config |
| sed -i 's/^#MaxSessions.*/MaxSessions 200/' /etc/ssh/sshd_config |
| service sshd restart |
| ``` |
| |
| ### 1.11 Synchronize system clocks |
| |
| ```bash |
| # For RHEL/Rocky Linux |
| systemctl enable chronyd |
| systemctl restart chronyd |
| chronyc tracking |
| |
| # For Ubuntu (enable must use 'chrony', restart works with either name) |
| systemctl enable chrony |
| systemctl restart chronyd |
| chronyc tracking |
| ``` |
| |
| ### 1.12 Create the gpadmin administrative user |
| |
| ```bash |
| groupadd gpadmin |
| useradd gpadmin -r -m -g gpadmin |
| passwd gpadmin |
| |
| # Grant passwordless sudo |
| # For RHEL/Rocky Linux: uncomment the %wheel NOPASSWD line in sudoers |
| # Or use visudo to uncomment: %wheel ALL=(ALL) NOPASSWD: ALL |
| sed -i 's/^# %wheel\tALL=(ALL)\tNOPASSWD: ALL/%wheel\tALL=(ALL)\tNOPASSWD: ALL/' /etc/sudoers |
| usermod -aG wheel gpadmin |
| |
| # For Ubuntu: add gpadmin to sudo group and create a sudoers drop-in file |
| usermod -aG sudo gpadmin |
| echo "gpadmin ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/gpadmin |
| chmod 440 /etc/sudoers.d/gpadmin |
| |
| # Set data directory ownership to gpadmin |
| chown -R gpadmin:gpadmin /data |
| ``` |
| |
| :::note |
| Make sure the `gpadmin` user has the same UID/GID across all hosts. Use `id gpadmin` to verify, or pass `-g <gid>` to `groupadd` and `-u <uid>` to `useradd` to set fixed values. |
| ::: |
| |
| --- |
| |
| ## 2. Install the Apache Cloudberry package (as root on each host) |
| |
| Download the package from [Apache Cloudberry Releases](https://cloudberry.apache.org/releases) to `/home/gpadmin/` on every host, then install: |
| |
| ```bash |
| # For RPM (Rocky Linux, RHEL, etc.) |
| dnf install /home/gpadmin/apache-cloudberry-db-incubating-*.rpm |
| |
| # For DEB (Ubuntu) |
| # apt install --fix-broken /home/gpadmin/apache-cloudberry-db-incubating-*.deb |
| |
| # Set ownership |
| chown -R gpadmin:gpadmin /usr/local/cloudberry* |
| ``` |
| |
| --- |
| |
| ## 3. Enable passwordless SSH (as gpadmin on cdw) |
| |
| ```bash |
| su - gpadmin |
| source /usr/local/cloudberry-db/cloudberry-env.sh |
| |
| # Generate SSH key (press Enter at passphrase prompts) |
| ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa |
| |
| # Enable 1-n passwordless SSH (will prompt for gpadmin password) |
| ssh-copy-id cdw |
| ssh-copy-id scdw |
| ssh-copy-id sdw1 |
| ssh-copy-id sdw2 |
| ssh-copy-id sdw3 |
| |
| # Create the host file for gpssh-exkeys |
| cat > ~/hostfile_exkeys <<EOF |
| cdw |
| scdw |
| sdw1 |
| sdw2 |
| sdw3 |
| EOF |
| |
| # Enable n-n passwordless SSH |
| gpssh-exkeys -f ~/hostfile_exkeys |
| |
| # Verify |
| gpssh -f ~/hostfile_exkeys -e 'hostname' |
| ``` |
| |
| ### Confirm the installation |
| |
| ```bash |
| gpssh -f ~/hostfile_exkeys -e 'ls -l /usr/local/cloudberry-db-*' |
| ``` |
| |
| --- |
| |
| ## 4. Create data storage areas (as gpadmin on cdw) |
| |
| ```bash |
| source /usr/local/cloudberry-db/cloudberry-env.sh |
| |
| # Coordinator data directory |
| sudo mkdir -p /data/coordinator |
| sudo chown gpadmin:gpadmin /data/coordinator |
| |
| # Standby coordinator data directory |
| gpssh -h scdw -e 'sudo mkdir -p /data/coordinator' |
| gpssh -h scdw -e 'sudo chown gpadmin:gpadmin /data/coordinator' |
| |
| # Segment host file |
| cat > ~/hostfile_gpinitsystem <<EOF |
| sdw1 |
| sdw2 |
| sdw3 |
| EOF |
| |
| # Segment primary and mirror directories |
| gpssh -f ~/hostfile_gpinitsystem -e 'sudo mkdir -p /data/primary' |
| gpssh -f ~/hostfile_gpinitsystem -e 'sudo mkdir -p /data/mirror' |
| gpssh -f ~/hostfile_gpinitsystem -e 'sudo chown -R gpadmin:gpadmin /data' |
| ``` |
| |
| --- |
| |
| ## 5. Initialize the cluster (as gpadmin on cdw) |
| |
| ### 5.1 Create the initialization host file |
| |
| ```bash |
| # Already created in step 4 as ~/hostfile_gpinitsystem |
| cat ~/hostfile_gpinitsystem |
| ``` |
| |
| ### 5.2 Create the configuration file |
| |
| ```bash |
| mkdir -p ~/gpconfigs |
| |
| cat > ~/gpconfigs/gpinitsystem_config <<EOF |
| SEG_PREFIX=gpseg |
| PORT_BASE=6000 |
| declare -a DATA_DIRECTORY=(/data/primary /data/primary) |
| COORDINATOR_HOSTNAME=cdw |
| COORDINATOR_DIRECTORY=/data/coordinator |
| COORDINATOR_PORT=5432 |
| TRUSTED_SHELL=ssh |
| CHECK_POINT_SEGMENTS=8 |
| ENCODING=UNICODE |
| MIRROR_PORT_BASE=7000 |
| declare -a MIRROR_DATA_DIRECTORY=(/data/mirror /data/mirror) |
| DATABASE_NAME=warehouse |
| EOF |
| ``` |
| |
| ### 5.3 Run the initialization utility |
| |
| ```bash |
| gpinitsystem -c ~/gpconfigs/gpinitsystem_config -h ~/hostfile_gpinitsystem \ |
| -s scdw --mirror-mode=spread |
| ``` |
| |
| When prompted, type `y` to continue. On success you should see: |
| |
| ``` |
| => Apache Cloudberry instance successfully created. |
| ``` |
| |
| :::tip |
| The `-s scdw` option initializes the standby coordinator during cluster creation. If you omit `-s` during `gpinitsystem`, you can initialize the standby separately afterwards: |
| |
| ```bash |
| gpinitstandby -s scdw |
| ``` |
| |
| Verify the standby is synchronized: |
| |
| ```bash |
| gpstate -f |
| ``` |
| ::: |
| |
| ### 5.4 Set the timezone (optional) |
| |
| ```bash |
| gpconfig -c TimeZone -v 'US/Pacific' |
| gpstop -ra |
| ``` |
| |
| ### 5.5 Set environment variables |
| |
| ```bash |
| cat >> ~/.bashrc <<EOF |
| source /usr/local/cloudberry-db/cloudberry-env.sh |
| export COORDINATOR_DATA_DIRECTORY=/data/coordinator/gpseg-1 |
| export PGPORT=5432 |
| export PGUSER=gpadmin |
| export PGDATABASE=warehouse |
| # (Optional) RHEL/Rocky Linux: ensure ps works under cloudberry-env.sh |
| export LD_PRELOAD=/lib64/libz.so.1 ps |
| EOF |
| |
| source ~/.bashrc |
| |
| # Copy to standby |
| scp ~/.bashrc scdw:~/ |
| ``` |
| |
| --- |
| |
| ## 6. Log into Apache Cloudberry |
| |
| ```bash |
| psql -d warehouse |
| ``` |
| |
| Check cluster status: |
| |
| ```sql |
| warehouse=# SELECT * FROM gp_segment_configuration ORDER BY content, role; |
| ``` |
| |
| --- |
| |
| ## Quick reference: cluster operations |
| |
| ```bash |
| # Check cluster status |
| gpstate -s |
| |
| # Stop the cluster |
| gpstop -a |
| |
| # Start the cluster |
| gpstart -a |
| |
| # Restart the cluster |
| gpstop -ra |
| ``` |