Deploy AlloyDB Omni high availability cluster on GCE VMs using RPM Orchestrator

1. Introduction

This codelab demonstrates AlloyDB Omni high availability (HA) cluster deployment on Google Compute Engine (GCE) Virtual Machines (VMs). At the end of this codelab, you will provision a HA reference architecture consisting of 3 Database nodes and 2 HAProxy nodes, along with a control node for management operations.

Prerequisites

  • Access to a Google Cloud Project and Cloud Shell from Cloud Console.
  • Google Cloud SDK (gcloud) installed and configured. For details on installing gcloud, refer to gcloud-install
  • Terraform v1.9.8 installed.

What you'll learn

  • How to create and prepare GCE VMs for AlloyDB Omni.
  • How to install and run AlloyDB Omni RPM Orchestrator.
  • How to install and configure AlloyDB Omni RPMs for HA reference architecture.

What you'll need

  • Obtain URL link to AlloyDB Omni RPMs and RPM orchestrator. For this, you need to fill the alloydb omni signup form. The links are sent to your email which repository URL server. Initialize following environment variables to keep the URLs handy which will be used during the codelab.
    export ORCHESTRATOR_ANSIBLE_COLLECTION_PATH="..."
export ALLOYDB_OMNI_REPOSITORY_URL="..."
export ALLOYDB_OMNI_ORCHESTRATOR_REPOSITORY_URL="..."
export ALLOYDB_OMNI_COMMON_REPOSITORY_URL=....
export ETCD_REPOSITORY_URL="..."
  • A working terminal with access to the AlloyDB Omni deployment stack. You can use cloud shell for this purpose.

2. Setup and Requirements

Project setup

Create a Google Cloud Project

  1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.
  2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

Icon to activate Cloud Shell

Alternatively you can press G then S. This sequence will activate Cloud Shell if you are within the Google Cloud Console or use this link.

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

Google Cloud Shell terminal showing that the environment has connected

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You don't need to install anything.

3. Create Google Cloud Compute Engine Virtual Machines

Prepare Terraform scripts

  1. Define environment variables for your project ID and desired cluster name. You will use these throughout the codelab.
    export PROJECT="your-project-id"
export CLUSTER="your-cluster-name"
    Note: Google Cloud project ID should be between 6 to 30 characters length only. For more information, refer to https://docs.cloud.google.com/resource-manager/docs/creating-managing-projects#before_you_begin
  2. Ensure that you have logged in with google cloud account user.
    gcloud auth login
  3. Create a working directory for your deployment and copy the required Terraform configuration files from your repository source.
    mkdir -p ~/alloydb-omni/$PROJECT/$CLUSTER
cd ~/alloydb-omni/$PROJECT/$CLUSTER
gcloud storage cp gs://alloydb-omni-install/rpm-orchestrator/gce/terraform/*.tf .
  4. Create a terraform.tfvars file to specify the required parameters as per the reference architecture as per the following contents:
    cat > terraform.tfvars <<EOF
# Required instance counts for reference architecture
db_instance_count      = 3
haproxy_instance_count = 2
control_instance_count = 1

# Optional: Customize if needed with the below variables
# os_image             = "rocky-linux-cloud/rocky-linux-9"
# zone                 = "us-west4-c"
# region_1             = "us-west4"
# db_instance_type     = "c4-highmem-4"
# db_disk_type         = "hyperdisk-balanced"
# db_data_size         =  "50" # GB
# data_dir             =  "/data"
EOF

Execute Terraform scripts and Validate

  1. You can now provision the VMs.
    terraform init
  2. Before you create VMs, ensure that you have the required access to create VMs and other resources. In summary you will need the following permissions.
    roles/compute.admin
roles/iam.roleAdmin
roles/compute.osAdminLogin
roles/iam.serviceAccountCreator
roles/iam.serviceAccountUser
roles/artifactregistry.repoAdmin
roles/storage.objectUser
roles/resourcemanager.projectIamAdmin
roles/serviceusage.serviceUsageAdmin
  3. Validate the configuration and apply the configuration to provision the resources.
    terraform validate
terraform apply --auto-approve
    Note: Ensure you are using Terraform v1.9.8 as expected by the deployment configuration.
  4. Once Terraform completes, verify that the virtual machines have been successfully created by listing the created instances using gcloud.
    gcloud compute instances list --filter="name~$CLUSTER" --project=$PROJECT
    You should see instances corresponding to your 3 DB nodes, 2 HAProxy nodes, and the control VM and corresponding zones. Record the ZONE for the control node. We will need this during this codelab, later.
    export ZONE=$(gcloud compute instances list \
  --filter="name=$CLUSTER-control"          \
  --format="value(zone)" --project=$PROJECT)

4. Prepare the VMs for the deployment

You need to create a SSH session to the control node and perform steps to enable SSH access to all the VMs (also termed as nodes).

  1. Ensure that the environment variables for your project and cluster are defined.
    export PROJECT="your-project-id"
export CLUSTER="your-cluster-name"
  2. Create SSH key and add the same for logging onto the control VM.
    # Replace values with your specific GCP project, cluster and zone details
ssh-keygen -t ed25519 -f $HOME/.ssh/google_compute_engine
gcloud compute os-login ssh-keys add --key-file=$HOME/.ssh/google_compute_engine.pub --project=$PROJECT
  3. Create firewall rule to allow SSH connections and also allow VRRP traffic.
    gcloud compute config-ssh --project=$PROJECT
gcloud compute firewall-rules create $CLUSTER-allow-ssh --network=$CLUSTER --project=$PROJECT --direction=INGRESS --action=allow --rules=tcp:22 --source-ranges="0.0.0.0/0"
gcloud compute firewall-rules create $CLUSTER-allow-vrrp --network=$CLUSTER --project=$PROJECT --allow=112 --source-ranges="0.0.0.0/0"
  4. Connect to the control VM over SSH:
    gcloud compute ssh "$CLUSTER-control" --zone=$ZONE --project=$PROJECT
  5. Create Linux group on the control node with the same name as the username:
    sudo groupadd $(id -un)
sudo usermod -aG $(id -un) $(id -un)
  6. Terraform setup creates a few configuration scripts and stage them in the /tmp/ directory of the control VM which includes things like cluster name, service account, project name, etc.Create password less ssh access from control nodes to all the cluster nodes using setup scripts on the control node.
    /tmp/setup-ssh-for-cluster.sh
    Important: Note down service_account user. We will be using this as SSH_USER later in this codelab.
  7. For this codelab, we can disable SELinux on all the nodes. Terraform script adds ‘/tmp/run-all.sh' which can be used for this purpose.
    /tmp/run-all.sh sudo setenforce 0
  8. If you have signed up for AlloyDB Omni and obtained the links, it is time to add those URLS to environment variables as well.
    cat >> ~/.codelab.env <<EOF
export ORCHESTRATOR_ANSIBLE_COLLECTION_PATH="..."
export ALLOYDB_OMNI_REPOSITORY_URL="..."
export ALLOYDB_OMNI_ORCHESTRATOR_REPOSITORY_URL="..."
export ALLOYDB_OMNI_COMMON_REPOSITORY_URL=....
export ETCD_REPOSITORY_URL="..."
EOF
    Decide on a virtual IP for your environment from the cidr_range input variable in the file terraform/variables.tf such that it does not conflict with the other nodes, as shown in the following example:
    cat >> ~/.codelab.env <<EOF
export VIRTUAL_IP="10.1.0.50"
EOF

5. Install required software components on all the VMs

Next step is to install required SW components on the VMs. This can be coordinated on the control node. All the following commands need to be executed on the control node.

  1. Connect to the control VM over SSH, if not already on the control node.
    gcloud compute ssh "$CLUSTER-control" --zone=$ZONE --project=$PROJECT
    Once you are on the control VM ssh prompt, source environment file.
    source ~/.codelab.env
  2. On the control node, install Ansible and the required Python libraries.
    sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm -y
sudo dnf install -y ansible
sudo dnf install -y python3-grpcio python3-protobuf python3-googleapis-common-protos python3-grpcio-status
  3. Next, download the RPM Orchestrator Ansible collection tar file, and install the same.
    gcloud storage cp "gs://${ORCHESTRATOR_ANSIBLE_COLLECTION_PATH#https://storage.googleapis.com/}google-alloydbomni_orchestrator-*.tar.gz" .
ansible-galaxy collection install google-alloydbomni_orchestrator-0.1.0-6.tar.gz
ansible-galaxy collection list | grep alloydbomni_orchestrator
    Note: Ensure ORCHESTRATOR_ANSIBLE_COLLECTION_PATH variable has "/" in the end.
  4. The orchestrator uses a deployment specification file in Ansible inventory format to understand your cluster topology. Create a file named deployment_spec.yaml with your node details on the control node.
    cat > deployment_spec.yaml <<EOF
alloydbomni:
  vars:
    cluster_manager:
      name: "$CLUSTER"
    etcd:
      setup: true
      config_forcewrite: true
    alloydbomni:
      major_version: "18"
      repo_url: $ALLOYDB_OMNI_REPOSITORY_URL
    alloydbomni_monitor:
      repo_url: $ALLOYDB_OMNI_COMMON_REPOSITORY_URL
    alloydbomni_cluster_manager:
      repo_url: $ALLOYDB_OMNI_ORCHESTRATOR_REPOSITORY_URL
    alloydbomni_node_manager:
      repo_url: $ALLOYDB_OMNI_ORCHESTRATOR_REPOSITORY_URL
    pgbouncer:
      repo_url: $ALLOYDB_OMNI_COMMON_REPOSITORY_URL
    pgbackrest:
      repo_url: $ALLOYDB_OMNI_COMMON_REPOSITORY_URL

  children:
    primary_instance_nodes:
      hosts:
        $CLUSTER-db1:
        $CLUSTER-db2:
        $CLUSTER-db3:
    load_balancer_nodes:
      hosts:
        $CLUSTER-haproxy1:
        $CLUSTER-haproxy2:
EOF
    Note: Review the generated file to ensure it refers to valid values.
  5. Create a playbook named install.yaml that references the install role from the orchestrator collection.
    cat > install.yaml <<EOF
- name: Install AlloyDB Omni cluster components
  hosts: all
  vars:
    ansible_become: true
    ansible_user: $SSH_USER
    ansible_ssh_private_key_file: $HOME/ssh-key-cluster-sa
  roles:
    - role: google.alloydbomni_orchestrator.install
EOF
    Note: Review the generated file to ensure it refers to valid values.
  6. Execute the playbook using your inventory file to download and install the RPMs across all specified nodes.
    ansible-playbook -i deployment_spec.yaml install.yaml

6. Bootstrap AlloyDB Omni Cluster

At this stage, we have installed all the required components on all the nodes. We are ready to bootstrap AlloyDB Omni cluster.

  1. Connect to the control VM over SSH, if not already on the control node.
    gcloud compute ssh "$CLUSTER-control" --zone=$ZONE --project=$PROJECT
    Once you are on the control VM ssh prompt, source environment file.
    source ~/.codelab.env
  2. Generate a hash for your password and note down the same.
    encoded_password=$(echo -n "your unique password" | base64)
  3. To create cluster, AlloyDB Omni must know how to configure it. Create a filename dbcluster.yaml for the DB cluster specifications.
    cat > dbcluster.yaml <<EOF
Secret:
  metadata:
    name: db-pw-$CLUSTER
  spec:
    type: Opaque
    data:
      $CLUSTER: $encoded_password
---
DBCluster:
  metadata:
    name: $CLUSTER
  spec:
    databaseVersion: 18.1.0
    mode: ""
    availability:
      numberOfStandbys: 2
      enableAutoFailover: true
      enableAutoHeal: true
      autoFailoverTriggerThreshold: 2
      autoHealTriggerThreshold: 2
      healthcheckPeriodSeconds: 5
      replayReplicationSlotsOnStandbys: false
    primarySpec:
      adminUser:
        passwordRef:
          name: db-pw-$CLUSTER
      resources:
        cpu: 4
        memory: 32Gi
        disks:
        - name: DataDisk
          path: $PGDATA
      dbLoadBalancerOptions:
        gcp:
          loadBalancerIP: "$VIRTUAL_IP"
          loadBalancerType: "internal"
          loadBalancerInterface: "eth0"
EOF
  4. Create a playbook named bootstrap.yaml, which will refer to the bootstrap ansible role for creating AlloyDB Omni Cluster.
    cat > bootstrap.yaml <<EOF
- name: Create DBCluster
  hosts: localhost
  vars:
    ansible_become: true
    ansible_user: $SSH_USER
    ansible_ssh_private_key_file: $HOME/ssh-key-cluster-sa
  roles:
  - role: google.alloydbomni_orchestrator.bootstrap
EOF
  5. Run your playbook to create cluster
    ansible-playbook bootstrap.yaml -i deployment_spec.yaml -e resource_spec=dbcluster.yaml

7. (Optional) Configure PgBouncer Connection Pooler

AlloyDB Omni supports lightweight connection pooling using PgBouncer. You can configure and bootstrap PgBouncer right after your cluster is provisioned.

  1. Create a resource specification file named pgbouncer.yaml that associates the connection pooler with your database cluster:
    cat > pgbouncer.yaml <<EOF
PgBouncer:
  metadata:
    name: pgbouncer-pooler
  spec:
    dbclusterRef: $CLUSTER
    allowSuperUserAccess: true
    accessMode: "rw"
    port: 6432
EOF
  2. Bootstrap the PgBouncer pooler using the same bootstrap.yaml playbook created earlier, passing the new specification file:
    ansible-playbook bootstrap.yaml -i deployment_spec.yaml -e resource_spec=pgbouncer.yaml

8. Verify the AlloyDB Omni Cluster

To verify that your cluster is functioning correctly and accessible using the load balancer, you can connect to it using the standard PostgreSQL client from the control node.

  1. Install the PostgreSQL 18 client repository and package on the control node:
    sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-latest.noarch.rpm
sudo dnf install -y postgresql18
  2. Connect to the cluster using the virtual IP reserved earlier. You will be prompted for the password you encoded in dbcluster.yaml:
    /usr/pgsql-18/bin/psql -h $VIRTUAL_IP -U postgres -W
  3. Once connected, you can run basic SQL queries to verify the cluster state, for example, checking the database version:
    postgres=# SELECT version();
    You should see output indicating PostgreSQL 18 running with AlloyDB Omni. Type \q to exit the prompt.

9. (Optional) Back up Data using pgBackRest

AlloyDB Omni integrates with pgBackRest to manage backups directly to Cloud Storage. You can configure a backup plan and trigger an on-demand backup to the GCS bucket created by your Terraform setup.

  1. Create a backup plan specification file named backup_plan.yaml pointing to your provisioned GCS bucket:
    cat > backup_plan.yaml <<EOF
BackupPlan:
  metadata:
    name: pgb-plan
  spec:
    dbclusterRef: $CLUSTER
    backupLocation:
      type: GCS
      gcsOptions:
        bucket: $CLUSTER-gcs-backups
        key: /backups
EOF
  2. Create an Ansible playbook named backup.yaml that references the backup management role:
    cat > backup.yaml <<EOF
- name: Manage AlloyDB Omni Backups
  hosts: localhost
  vars:
    ansible_become: true
    ansible_user: $SSH_USER
    ansible_ssh_private_key_file: $HOME/ssh-key-cluster-sa
  roles:
    - role: google.alloydbomni_orchestrator.backup
EOF
  3. Apply the backup plan using the backup.yaml playbook:
    ansible-playbook backup.yaml -i deployment_spec.yaml -e resource_spec=backup_plan.yaml
  4. Once the plan is successfully established, create an on-demand backup resource file named create_backup.yaml.
    cat > create_backup.yaml <<EOF
Backup:
  metadata:
    name: on-demand-backup
  spec:
    backupPlanRef: pgb-plan
    dbclusterRef: $CLUSTER
EOF
  5. Execute the playbook to initiate the backup:
    ansible-playbook backup.yaml -i deployment_spec.yaml -e resource_spec=create_backup.yaml
  6. To verify backup's status, create a status.yaml playbook:
- name: Fetch AlloyDB Omni Resource Status
hosts: localhost
vars:
  ansible_become: true
  ansible_user: $SSH_USER
  ansible_ssh_private_key_file: $HOME/ssh-key-cluster-sa
roles:
  - role: google.alloydbomni_orchestrator.status
EOF
  1. Run the playbook to list all backups:
ansible-playbook status.yaml -i deployment_spec.yaml -e resource_type=Backup
  1. Alternatively, to get detailed info on a specific backup created earlier, pass -e resource_name=on-demand-backup:
ansible-playbook status.yaml -i deployment_spec.yaml \
  -e resource_type=Backup \
  -e resource_name=on-demand-backup

10. Cleanup Resources

When you are finished with the deployment, you can destroy the provisioned resources to avoid incurring charges.

  1. Create a playbook named teardown.yaml:
    cat > teardown.yaml <<EOF
- name: Tear down AlloyDB Omni cluster
  hosts: localhost
  vars:
    ansible_become: true
    ansible_user: $SSH_USER
    ansible_ssh_private_key_file: $HOME/ssh-key-cluster-sa
  roles:
  - role: google.alloydbomni_orchestrator.delete
EOF
  2. Execute the playbook using ansible-playbook. If you configured backups, delete the Backup and BackupPlan resources first:
    ansible-playbook teardown.yaml -i deployment_spec.yaml  -e "resource_type=Backup" -e "resource_name=on-demand-backup"
ansible-playbook teardown.yaml -i deployment_spec.yaml  -e "resource_type=BackupPlan" -e "resource_name=pgb-plan"
    If you configured PgBouncer, delete the connection pooler resource next:
    ansible-playbook teardown.yaml -i deployment_spec.yaml  -e "resource_type=PgBouncer" -e "resource_name=pgbouncer-pooler"
    Then, specify DBCluster as the resource_type and delete the database cluster itself:
    ansible-playbook teardown.yaml -i deployment_spec.yaml  -e "resource_type=DBCluster" -e "resource_name=$CLUSTER"
  3. Once the cluster is deleted, log out from the control node, switch back to your terminal and navigate to your Terraform work directory.
    Note: if you have freshly logged onto the cloud shell, then remember to set these:
    export PROJECT="your-project-id"
export CLUSTER="your-cluster-name"
  4. Delete the VRRP firewall rule and destroy the resources managed by Terraform:
    gcloud compute firewall-rules delete -q "$CLUSTER-allow-vrrp" --project="$PROJECT"
gcloud compute firewall-rules delete -q "$CLUSTER-allow-ssh" --project="$PROJECT"
terraform destroy
  5. Confirm the destruction when prompted. Since lingering network resources or routes might occasionally prevent complete teardown, run the following safe cleanup commands to ensure all associated firewall rules, NAT gateways, routers, routes, subnets, and networks are fully removed. Replace REGION with your specific deployment region, such as ‘us-central1':
    # Delete any remaining firewall rules associated with the cluster
gcloud compute firewall-rules list --project=$PROJECT 2> /dev/null | grep ^$CLUSTER- | cut -f1 -d' ' | \
  while read rule; do gcloud compute firewall-rules delete --project=$PROJECT --quiet $rule; done

# Delete the Cloud NAT gateway and router if they still exist
gcloud compute routers nats describe $CLUSTER-nat-gw --router=$CLUSTER-router --region=REGION --project=$PROJECT 2>/dev/null \
  && gcloud compute routers nats delete $CLUSTER-nat-gw --router=$CLUSTER-router --region=REGION --project=$PROJECT --quiet

gcloud compute routers describe $CLUSTER-router --region=REGION --project=$PROJECT 2>/dev/null \
  && gcloud compute routers delete $CLUSTER-router --region=REGION --project=$PROJECT --quiet

# Delete any remaining custom routes
gcloud compute routes list --project=$PROJECT --filter="network:$CLUSTER" --format="value(name)" 2>/dev/null | \
  while read route; do gcloud compute routes delete --project=$PROJECT --quiet $route 2>/dev/null || true; done

# Delete the subnet and network if they still exist
gcloud compute networks subnets describe $CLUSTER --region=REGION --project=$PROJECT 2>/dev/null \
  && gcloud compute networks subnets delete $CLUSTER --region=REGION --project=$PROJECT --quiet

gcloud compute networks describe $CLUSTER --project $PROJECT 2> /dev/null \
  && gcloud compute networks delete $CLUSTER --project $PROJECT --quiet

11. Congratulations

Congratulations for completing the codelab.

What we've covered

  • How to create and prepare GCE VMs for AlloyDB Omni.
  • How to install and run AlloyDB Omni RPM Orchestrator.
  • How to install and configure AlloyDB Omni RPMs for HA reference architecture.

You can read more about AlloyDB Omni in the documentation.

12. Survey

How will you use this tutorial?

Only read through it Read it and complete the exercises