Branches

Backup Utility for Apache Cloudberry (Incubating)

gpbackup and gprestore are Go utilities for performing Greenplum database backups, which are originally developed by the Greenplum Database team. This repo is a fork of gpbackup, dedicated to supporting Cloudberry.

Pre-Requisites

The project requires the Go Programming language version 1.25 or higher. Follow the directions here for installation, usage and configuration instructions. Make sure to set the Go PATH environment variable before starting the following steps.

export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin

Download & Build

Downloading:

# Download the stable release version
go install github.com/apache/cloudberry-backup@2.1.0-incubating

# Or download the latest development version from main branch
go install github.com/apache/cloudberry-backup@main

Note: Please use the specific version @2.1.0-incubating or @main instead of @latest. The @latest tag will install an older version due to Go modules version resolution rules.

This will place the code in $GOPATH/pkg/mod/github.com/apache/cloudberry-backup.

Building and installing binaries

Make the gpbackup directory your current working directory and run:

make depend
make build

The build target will put the gpbackup and gprestore binaries in $HOME/go/bin. This will also attempt to copy gpbackup_helper to the Cloudberry segments (retrieving hostnames from gp_segment_configuration). Pay attention to the output as it will indicate whether this operation was successful.

make build_linux is for cross compiling on macOS, and the target is Linux.

make install will scp the gpbackup_helper binary (used with -single-data-file flag) to all hosts

Running the utilities

The basic command for gpbackup is

gpbackup --dbname <your_db_name>

The basic command for gprestore is

gprestore --timestamp <YYYYMMDDHHMMSS>

Run --help with either command for a complete list of options.

Standby history database synchronization

After a successful backup, gpbackup automatically copies a consistent snapshot of the coordinator's gpbackup_history.db to an up standby coordinator. Synchronization starts only after the final Success history row has been written and the local SQLite connection has been closed.

This synchronization is best effort. If no up standby exists, synchronization is skipped. If discovery, snapshot creation, transfer, or installation fails, gpbackup logs a warning but keeps the successful backup exit status. A failed or terminated backup is not synchronized. Synchronization also does not run when --no-history is used or when the final history update fails. Use --no-history-sync-standby to keep writing local history while disabling standby synchronization for one backup:

gpbackup --dbname <your_db_name> --no-history-sync-standby

Configure the sync timeout with --history-sync-standby-timeout SECONDS. The default is 300 seconds; the supported range is 1 to 86400 seconds. The timeout is one shared budget for rsync and remote install. It starts after snapshot validation. Standby discovery and SQLite snapshot creation and validation (VACUUM INTO and PRAGMA quick_check) are outside this budget. If a transport step fails, remote cleanup of the temporary file uses its own fixed 120-second timeout, independent of --history-sync-standby-timeout.

The synchronization process:

Takes a non-waiting lock next to the canonical source database.
Creates a consistent SQLite snapshot with VACUUM INTO and accepts it only when PRAGMA quick_check returns ok.
Transfers the snapshot with rsync -p -s to a unique temporary file in the standby coordinator data directory.
Preserves the existing standby file's owner, group, and mode when it exists, then atomically renames the temporary file to gpbackup_history.db.

rsync 3.0.0 or later must be installed on both the host running gpbackup and the standby coordinator. The gpbackup host must also have ssh, and the current OS user must have non-interactive SSH access to the standby host. That user must be able to create files in the standby coordinator data directory and preserve the destination file's ownership and permissions. The cluster must expose an up standby in gp_segment_configuration.

The atomic rename prevents readers from observing a partially copied database, but it is not a failover coordination mechanism. A coordinator role change during synchronization can race with discovery and installation. Processes that already have the old standby database open continue reading that old inode until they close and reopen it.

For automatic synchronization after history maintenance and for the strict manual command, see gpBackMan history synchronization.

Additional tools

This repository also includes the following tools:

gpbackup_s3_plugin — S3 storage plugin for gpbackup and gprestore.
gpBackMan — utility for managing backups created by gpbackup.
gpbackup_exporter — Prometheus exporter for collecting metrics from gpbackup history database.

Validation and code quality

Test setup

Required for Cloudberry 1.0+, several tests require the dummy_seclabel Cloudberry contrib module. This module exists only to support regression testing of the SECURITY LABEL statement. It is not intended to be used in production. Use the following commands to install the module.

pushd $(find ~/workspace/cloudberry -name dummy_seclabel)
    make install
    gpconfig -c shared_preload_libraries -v dummy_seclabel
    gpstop -ra
    gpconfig -s shared_preload_libraries | grep dummy_seclabel
popd

Test execution

NOTE: The integration and end_to_end tests require a running Cloudberry instance.

To run all tests except end-to-end (linters, unit, and integration), use make test.
To run only unit tests, use make unit.
To run only integration tests (requires a running Cloudberry instance), use make integration.
To run end to end tests (requires a running Cloudberry instance), use make end_to_end.

We provide the following targets to help developers ensure their code fits Go standard formatting guidelines:

To run a linting tool that checks for basic coding errors, use: make lint. This target runs gometalinter. Note: The lint target will fail if code is not formatted properly.
To automatically format your code and add/remove imports, use make format. This target runs goimports and gofmt. We will only accept code that has been formatted using this target or an equivalent gofmt call.

Cleaning up

To remove the compiled binaries and other generated files, run make clean.

Code Formatting

We use goimports to format go code. See https://godoc.org/golang.org/x/tools/cmd/goimports The following command formats the gpbackup codebase excluding the vendor directory and also lists the files updated.

goimports -w -l $(find . -type f -name '*.go' -not -path "./vendor/*")

Troubleshooting

Dummy Security Label module is not installed or configured

If you see errors in many integration tests (below), review the Validation and code quality [Test setup](##Test setup) section above:

SECURITY LABEL FOR dummy ON TYPE public.testtype IS 'unclassified';
      Expected
          <pgx.PgError>: {
              Severity: "ERROR",
              Code: "22023",
              Message: "security label provider \"dummy\" is not loaded",

Tablespace already exists

If you see errors indicating the test_tablespace tablespace already exists (below), execute psql postgres -c 'DROP TABLESPACE test_tablespace' to cleanup the environment and rerun the tests.

    CREATE TABLESPACE test_tablespace LOCATION '/tmp/test_dir'
    Expected
        <pgx.PgError>: {
            Severity: "ERROR",
            Code: "42710",
            Message: "tablespace \"test_tablespace\" already exists",

How to Contribute

See CONTRIBUTING.md file.

License

Licensed under Apache License Version 2.0. For more details, please refer to the LICENSE.

Acknowledgment

Thanks to all the Greenplum Backup contributors, more details in its GitHub page.