Back up and restore AlloyDB Omni RPM to S3-compatible storage using pgBackRest

1. Introduction

This codelab shows how to configure pgBackRest to back up and restore an AlloyDB Omni RPM deployment using S3-compatible object storage.

Note: In this codelab, you deploy a standalone MinIO binary on the VM to act as a local, lightweight S3-compatible storage endpoint for tutorial and testing purposes. For production workloads, always use a supported, actively maintained S3-compatible storage solution or cloud object storage.

Prerequisites

  • Access to a Google Cloud Project and Cloud Shell from Cloud Console.
  • Google Cloud SDK (gcloud) installed and configured.

What you'll learn

  • How to set up a Compute Engine VM for AlloyDB Omni RPM.
  • How to set up a local S3-compatible object storage server on the VM.
  • How to configure pgBackRest to use S3-compatible object storage as a backup repository.
  • How to perform, list, and verify backups.
  • How to restore data from a backup.

What you'll need

  • Obtain a URL link to AlloyDB Omni RPMs. The links are sent to your email which includes the repository URL server. Initialize the following environment variable to keep the URL ready to use during the codelab.
    export ALLOYDB_OMNI_REPOSITORY_URL="..."
  • A working terminal with access to Google Cloud. You can use Cloud Shell for this purpose.

2. Setup and Requirements

Project setup

Create a Google Cloud Project

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
  2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

Start Cloud Shell

Although you can operate Google Cloud remotely from your laptop, in this codelab you use the Google Cloud Shell, a command-line environment running in the cloud.

Icon to activate Cloud Shell

  1. Open the Google Cloud console and click the Cloud Shell icon on the top right toolbar.You can also open the Cloud Shell in one of the following ways:
    • If you're using the Google Cloud console, press G and then S.
    • Click this link.

It typically takes a few moments to provision and connect to the environment. When this operation completes, you see something similar to the following:

Google Cloud Shell terminal showing that the environment has connected{: width="624"}

3. Create the Compute Engine VM

In this section, you create a single Compute Engine (GCE) virtual machine (VM) using the gcloud command-line tool. You use a Rocky Linux 9 image, which is fully compatible with AlloyDB Omni RPMs.

  1. Define environment variables for your project ID, VM name, machine type, zone, and region.
    export PROJECT="your-project-id"
export INSTANCE_NAME="alloydb-omni-pgbackrest"
export MACHINE_TYPE="e2-standard-4"
export ZONE="us-central1-c"
export REGION="us-central1"
  2. Sign in with using your Google Cloud account user, and configure the default project.
    gcloud auth login
gcloud config set project $PROJECT
  3. Create a firewall rule to allow SSH connections to your VM.
    gcloud compute firewall-rules create allow-ssh-alloydb-omni \
    --project=$PROJECT \
    --direction=INGRESS \
    --action=allow \
    --rules=tcp:22 \
    --source-ranges="0.0.0.0/0"
  4. Since the VM is created without a public IP address, configure a Cloud Router and Cloud NAT gateway to let the VM access external package repositories and download required software.
    gcloud compute routers create alloydb-omni-router \
    --project=$PROJECT \
    --network=default \
    --region=$REGION

gcloud compute routers nats create alloydb-omni-nat \
    --project=$PROJECT \
    --router=alloydb-omni-router \
    --region=$REGION \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips
  5. Ensure that your VPC network has a valid default internet gateway route. In some shared or playground environments, default routes are occasionally removed by cleanup scripts. This route is required for Cloud NAT to forward outbound traffic to the internet.
    gcloud compute routes create default-internet-route \
    --project=$PROJECT \
    --network=default \
    --destination-range=0.0.0.0/0 \
    --next-hop-gateway=default-internet-gateway
  6. Create the Compute Engine VM instance with Rocky Linux 9 and a 20GB balanced persistent disk. For enhanced security, use the --no-address flag so the VM is not assigned a public IP address.
    gcloud compute instances create $INSTANCE_NAME \
    --project=$PROJECT \
    --zone=$ZONE \
    --machine-type=$MACHINE_TYPE \
    --image-family=rocky-linux-9 \
    --image-project=rocky-linux-cloud \
    --boot-disk-size=20GB \
    --boot-disk-type=pd-balanced \
    --no-address
  7. After the VM is created, verify the VM's status.
    gcloud compute instances list --filter="name=$INSTANCE_NAME" --project=$PROJECT

4. Set up S3-compatible object storage

In this section, you connect to your VM, prepare the environment, and set up a local S3-compatible object storage server (minio) along with an S3 client (mc) to host your backup bucket.

  1. Connect to the VM over SSH using Identity-Aware Proxy (IAP). Since the VM doesn't have a public IP address, gcloud will automatically tunnel the SSH connection through IAP.
    gcloud compute ssh $INSTANCE_NAME --zone=$ZONE --project=$PROJECT --tunnel-through-iap
  2. Install policycoreutils-python-utils package so that the SELinux policy can be configured later if needed.
    sudo dnf install -y policycoreutils-python-utils
  3. Download the S3-compatible server binary (minio).
    curl -L -o minio \
    https://github.com/minio/minio/releases/download/RELEASE.2024-05-10T01-41-38Z/minio.linux-amd64.RELEASE.2024-05-10T01-41-38Z
chmod +x minio
sudo mv minio /usr/local/bin/
  4. Create a data directory for the object storage server.
    sudo mkdir -p /var/lib/s3-storage/data
  5. Start the S3-compatible server in the background with example credentials. Note that the secret key must be at least 8 characters long.
    export MINIO_ROOT_USER="s3admin"
export MINIO_ROOT_PASSWORD="s3password"

nohup sudo env MINIO_ROOT_USER="$MINIO_ROOT_USER" \
    MINIO_ROOT_PASSWORD="$MINIO_ROOT_PASSWORD" \
    /usr/local/bin/minio server /var/lib/s3-storage/data \
    --console-address ":9001" \
    > s3-server.log 2>&1 &
  6. Verify that the S3-compatible server is running.
    ps aux | grep minio
  7. Download and install the S3 client (mc) to manage buckets.
    curl -L -o mc \
    https://github.com/minio/mc/releases/download/RELEASE.2024-05-09T17-04-24Z/mc.linux-amd64.RELEASE.2024-05-09T17-04-24Z
chmod +x mc
sudo mv mc /usr/local/bin/
  8. Configure the S3 client alias (s3repo) and create a bucket named alloydb-backups.
    mc alias set s3repo http://localhost:9000 s3admin s3password
mc mb s3repo/alloydb-backups

5. Install AlloyDB Omni

In this section, you install and configure AlloyDB Omni.

  1. Define environment variables on the VM for the data directory, your AlloyDB Omni repository URL, and a strong password for the PostgreSQL postgres administrator user. Replace the URL with the one you received from the signup form.
    export PGDATA="/var/lib/alloydbomni"
export ALLOYDB_OMNI_REPOSITORY_URL="..."
export PGPASSWORD="YourStrongPassword123!"
    Note: Passing passwords using environment variables on the command line (e.g., PGPASSWORD=$PGPASSWORD) is used here for simplicity in the codelab, but it can expose the password in process listings. In production, use more secure methods such as a .pgpass file or a secret manager.
  2. Create the data directory where AlloyDB Omni will store its database files.
    sudo mkdir -p $PGDATA
  3. Configure the YUM repository on your VM by creating the repository configuration file.
    sudo tee /etc/yum.repos.d/alloydbomni.repo << EOF
[alloydbomni]
name=alloydbomni
baseurl=$ALLOYDB_OMNI_REPOSITORY_URL
enabled=1
repo_gpgcheck=0
gpgcheck=1
gpgkey=https://dl.google.com/linux/linux_signing_key.pub
EOF
  4. Update the YUM cache.
    sudo yum makecache
  5. Install version 18 of the AlloyDB Omni RPM package.
    sudo dnf install -y alloydbomni18
  6. Initialize the AlloyDB Omni database instance.
    sudo PGPASSWORD=$PGPASSWORD PGDATA=$PGDATA /usr/lib/postgresql/18/bin/alloydbomni18-setup initdb
  7. Temporarily disable SELinux enforcing mode to facilitate setup, as per standard codelab practices.
    sudo setenforce 0
  8. Prepare the AlloyDB Omni database instance.
    sudo PGPASSWORD=$PGPASSWORD PGDATA=$PGDATA /usr/lib/postgresql/18/bin/alloydbomni18-setup preparedb
  9. Enable and start the AlloyDB Omni systemd service.
    sudo systemctl enable alloydbomni18
sudo systemctl start alloydbomni18
  10. Verify that the AlloyDB Omni service is active and running successfully.
    sudo systemctl status alloydbomni18

6. Configure pgBackRest

In this section, you install pgBackRest from the AlloyDB Omni Common repository, configure /etc/pgbackrest/pgbackrest.conf to use your S3-compatible object storage bucket as a backup repository, and enable WAL archiving in AlloyDB Omni.

  1. Define an environment variable on the VM for your AlloyDB Omni Common repository URL. Replace the URL with the one you received from the signup form.
    export ALLOYDB_OMNI_COMMON_REPOSITORY_URL="..."
  2. Configure the pgBackRest YUM repository and install pgbackrest-2.58.0.
    sudo tee /etc/yum.repos.d/pgbackrest.repo <<EOF
[pgbackrest]
name=Google Alloydb Omni Orchestrator
baseurl=$ALLOYDB_OMNI_COMMON_REPOSITORY_URL
enabled=1
repo_gpgcheck=0
gpgcheck=0
gpgkey=https://dl.google.com/linux/linux_signing_key.pub
EOF

sudo dnf install -y pgbackrest-2.58.0
  3. Create the spool and log directories for pgBackRest and assign ownership to the postgres user.
    sudo mkdir -p /var/log/pgbackrest
sudo mkdir -p /var/lib/pgbackrest

sudo chown -R postgres:postgres /var/log/pgbackrest
sudo chown -R postgres:postgres /var/lib/pgbackrest
  4. Create /etc/pgbackrest/pgbackrest.conf and configure strict file permissions.
    sudo mkdir -p /etc/pgbackrest/conf.d

sudo tee /etc/pgbackrest/pgbackrest.conf <<EOF
[global]
spool-path=/var/lib/pgbackrest
lock-path=/var/lib/pgbackrest

# S3 Repository details
repo1-type=s3
repo1-s3-endpoint=http://localhost:9000
repo1-s3-bucket=alloydb-backups
repo1-s3-key=s3admin
repo1-s3-key-secret=s3password
repo1-s3-region=us-east-1
repo1-s3-uri-style=path
repo1-path=/pgbackrest

# Retention policy
repo1-retention-full=3
repo1-retention-full-type=count
repo1-retention-diff=16

# Force immediate checkpoint during backup start
start-fast=y

# Logging
log-path=/var/log/pgbackrest
log-level-console=info
log-level-file=info

# Compression
compress-type=zst

# Asynchronous WAL archiving
archive-async=y
archive-push-queue-max=1024MB
archive-get-queue-max=256MB
archive-missing-retry=y

[global:archive-push]
process-max=2

[global:archive-get]
process-max=2

[omni]
pg1-user=postgres
pg1-socket-path=/tmp
pg1-path=$PGDATA
EOF

sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
  5. Initialize the pgBackRest stanza as the postgres user.
    sudo -u postgres pgbackrest --stanza=omni stanza-create
    Confirm that stanza-create command end: completed successfully is shown in the output.
  6. Configure AlloyDB Omni archiving parameters using psql.
    sudo -u postgres /usr/lib/postgresql/18/bin/psql \
    -c "ALTER SYSTEM SET archive_mode = 'on';" \
    -c "ALTER SYSTEM SET archive_command = 'pgbackrest --stanza=omni archive-push %p';" \
    -c "ALTER SYSTEM SET wal_level = 'replica';" \
    -c "ALTER SYSTEM SET max_wal_senders = 10;"
  7. Restart the AlloyDB Omni service to apply the archiving changes.
    sudo systemctl restart alloydbomni18
  8. Verify pgBackRest configuration integrity and connectivity.
    sudo -u postgres pgbackrest --stanza=omni check
    Confirm that check command end: completed successfully is shown in the output.

7. Perform Backup

In this section, you use pgBackRest to perform a full backup and a differential backup of your AlloyDB Omni database to S3-compatible object storage.

  1. Trigger a full backup using pgbackrest.
    sudo -u postgres pgbackrest \
    --stanza=omni \
    --type=full backup
  2. Trigger a differential backup using pgbackrest.
    sudo -u postgres pgbackrest \
    --stanza=omni \
    --type=diff backup
    Note: Other supported backup types include incr (incremental).

8. List and Verify Backups

In this section, you list the backups and verify that they completed successfully.

  1. List the backups using the pgbackrest info command.
    sudo -u postgres pgbackrest \
    --stanza=omni info
    Confirm that the output shows status: ok along with backup information for each backup command you executed:
    • Full backup: full backup: F (for example, 20260908-093545F)
    • Diff backup: diff backup: F_D (for example, 20260908-093545F_20260908-094016D)
  2. Verify the backup files stored in the bucket using the S3 client (mc).
    mc ls s3repo/alloydb-backups/pgbackrest/
    You should see backup, archive, and stanza.info in the output.

9. Restore Data from Backup

In this section, you stop the AlloyDB Omni database service and restore data from a specific backup using pgbackrest.

Caution: Restoring stops your database and overwrites all of your database data.

  1. Define an environment variable for the backup ID that you want to restore from. Replace the value with the backup ID shown in your pgbackrest info output.
    export BACKUPID="20260908-093545F"
  2. Stop the AlloyDB Omni service and verify that it is inactive.
    sudo systemctl stop alloydbomni18
sudo systemctl status alloydbomni18
    Confirm that the output shows Active: inactive.
  3. Run pgbackrest restore with the --delta option to restore the database from the selected backup.
    sudo -u postgres pgbackrest \
    --stanza=omni \
    --set=$BACKUPID \
    --delta restore
    Confirm that restore command end: completed successfully is shown in the output.
  4. Start the AlloyDB Omni service and verify that it is active and running.
    sudo systemctl start alloydbomni18
sudo systemctl status alloydbomni18
    Confirm that the output shows Active: active (running).

10. Cleanup and Survey

After you finish the codelab, you can clean up the Google Cloud resources to avoid incurring unwanted charges.

  1. Disconnect from your SSH session on the VM to return to your local Cloud Shell terminal.
    exit
  2. Delete the Compute Engine VM instance.
    gcloud compute instances delete $INSTANCE_NAME --zone=$ZONE --project=$PROJECT --quiet
  3. Delete the firewall rule created for SSH access.
    gcloud compute firewall-rules delete allow-ssh-alloydb-omni --project=$PROJECT --quiet
  4. Delete the Cloud NAT gateway and Cloud Router.
    gcloud compute routers nats delete alloydb-omni-nat --router=alloydb-omni-router --region=$REGION --project=$PROJECT --quiet
gcloud compute routers delete alloydb-omni-router --region=$REGION --project=$PROJECT --quiet
  5. Delete the custom route.
    gcloud compute routes delete default-internet-route --project=$PROJECT --quiet

11. Congratulations

Congratulations on completing the codelab! You successfully configured pgBackRest to back up and restore an AlloyDB Omni RPM deployment using S3-compatible object storage.

What we've covered

  • How to set up a Compute Engine VM for AlloyDB Omni RPM.
  • How to set up an S3-compatible object storage bucket.
  • How to configure pgBackRest with an S3-compatible repository.
  • How to perform, list, and verify full and differential backups.
  • How to restore data from a backup using pgBackRest.

For more information, see the AlloyDB Omni overview.

12. Survey

How will you use this tutorial?

Only read through it Read it and complete the exercises