Private Service Connect Mutability Codelab

1. Introduction

Private Service Connect is a capability of Google Cloud networking that allows consumers to access managed services privately from inside their VPC network. Similarly, it allows managed service producers to host these services in their own separate VPC networks and offer a private connection to their consumers.

Until now, once PSC services are exposed through a service attachment, changes could not be made to service load balancers without removing the service attachment and disrupting the consumer PSC endpoint connectivity. With the introduction of PSC mutability, it enables producers to update their load balancer while preserving the PSC endpoint connection. Maintaining the PSC endpoint connection to the service attachment while changing the load balancer has the benefit of not requiring any action on the consumer side.

At this time, PSC mutability will support

  • Migration to a load balancer forwarding rule of the same type (ex. Network Passthrough to Network Passthrough)
  • Migration to a load balancer forwarding rule of a different type (ex. Network Passthrough to Application load balancer)

PSC mutability does not support editing the forwarding rule in place.

In this lab, you'll create a producer web service exposed through an internal Network Passthrough load balancer, ensure that the service has connectivity through a PSC endpoint, then update the service attachment association to a new forwarding rule to an internal Application load balancer.

What you'll learn

  • Create a simple Apache web service exposed as a PSC producer service.
  • Create a PSC endpoint.
  • Create a Cloud DNS private zone for the consumer service calls.
  • Update the forwarding rule associated with the service attachment using PSC Mutability.

What you'll need

  • A Google Cloud project with owner permissions
  • Basic knowledge of Google Cloud networking

2. Test Environment

Traditionally, Producers and Consumers work in different projects. For simplicity purposes, we will perform all actions in the same project, but each of the steps will be labeled with Producer or Consumer projects in case the user prefers to work in different projects.

For this lab, we will start by creating producer-vpc with three subnets; one to host the producer service and test VM, one for the load balancer forwarding rules, and a PSC NAT subnet. We'll need a Cloud Router and Cloud NAT for internet reachability to download Linux packages. We'll expose our Apache service, configured through an unmanaged instance group, through an internal regional Network Passthrough load balancer and associate the forwarding rule with a service attachment.

On the consumer side, we'll create consumer-vpc with a single subnet to host our PSC endpoint and a test client VM. We'll configure a Cloud DNS private zone to access the service via a hostname.

To showcase the PSC Mutability feature, on the producer side, we'll add two additional subnets; one to host the new service VM, and one for the load balancer proxy-only subnet. We'll create a new VM hosting an Apache web server, again configured through an unmanaged instance group, and expose it through a new internal regional Application load balancer. We'll update the service attachment to point to the new forwarding rule and test that the connectivity on the consumer side remains in place.

683e2b440a0cc07f.png

3. Setup and Requirements

Self-paced environment setup

  1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

295004821bab6a87.png

37d264871000675d.png

96d86d3d5655cdbe.png

  • The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can always update it.
  • The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference your Project ID (typically identified as PROJECT_ID). If you don't like the generated ID, you might generate another random one. Alternatively, you can try your own, and see if it's available. It can't be changed after this step and remains for the duration of the project.
  • For your information, there is a third value, a Project Number, which some APIs use. Learn more about all three of these values in the documentation.
  1. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab won't cost much, if anything at all. To shut down resources to avoid incurring billing beyond this tutorial, you can delete the resources you created or delete the project. New Google Cloud users are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

Activate Cloud Shell

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

Screenshot of Google Cloud Shell terminal showing that the environment has connected

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.

4. Before you begin

Enable APIs

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
export project=[YOUR-PROJECT-NAME]
export region=us-central1
export zone=$region-a
echo $project
echo $region
echo $zone

Enable all necessary services

gcloud services enable compute.googleapis.com
gcloud services enable dns.googleapis.com

5. Build Producer Network [Producer Project]

Create VPC Network

From Cloud Shell

gcloud compute networks create producer-vpc \
    --subnet-mode=custom

Create Subnets

We'll start by deploying 3 subnets into the producer-vpc. One will be for deploying the producer service, one for the load balancer forwarding rules, and one that will be associated with the PSC Service Attachment for our PSC NAT subnet.

From Cloud Shell

gcloud compute networks subnets create producer-service-subnet \
    --network=producer-vpc \
    --range=10.0.0.0/28 \
    --region=$region

gcloud compute networks subnets create producer-fr-subnet \
        --network=producer-vpc \
        --range=192.168.0.0/28 \
        --region=$region

gcloud compute networks subnets create psc-nat-subnet \
    --network=producer-vpc \
    --range=10.100.0.0/28 \
    --region=$region \
    --purpose=PRIVATE_SERVICE_CONNECT

Create Cloud NAT

A Cloud NAT is required to install the proper packages for our producer services.

From Cloud Shell

gcloud compute routers create $region-cr \
    --network=producer-vpc \
    --region=$region

From Cloud Shell

gcloud compute routers nats create $region-nat \
    --router=$region-cr \
    --region=$region \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips

Create Network Firewall Policy and Rules

From Cloud Shell

gcloud compute network-firewall-policies create producer-vpc-policy --global

gcloud compute network-firewall-policies associations create \
    --firewall-policy producer-vpc-policy \
    --network producer-vpc \
    --name network-producer-vpc \
    --global-firewall-policy

To allow IAP to connect to your VM instances, create a firewall rule that:

  • Applies to all VM instances that you want to be accessible by using IAP.
  • Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.

From Cloud Shell

gcloud compute network-firewall-policies rules create 1000 \
    --action ALLOW \
    --firewall-policy producer-vpc-policy \
    --description "SSH with IAP" \
    --direction INGRESS \
    --src-ip-ranges 35.235.240.0/20 \
    --layer4-configs tcp:22  \
    --global-firewall-policy

We will start with three firewall rules. One will allow access to VMs for the load balancer health checks (2000), one to allow access to the VMs from the PSC NAT range (3000), and one to allow connectivity between VMs in the service subnet for testing purposes (4000). We will make all rules for ports 80 as we will be using this port for our service.

From Cloud Shell

gcloud compute network-firewall-policies rules create 2000 \
    --action ALLOW \
    --firewall-policy producer-vpc-policy \
    --description "LB healthchecks" \
    --direction INGRESS \
    --src-ip-ranges 130.211.0.0/22,35.191.0.0/16 \
    --layer4-configs tcp:80  \
    --global-firewall-policy

gcloud compute network-firewall-policies rules create 3000 \
    --action ALLOW \
    --firewall-policy producer-vpc-policy \
    --description "allow access from PSC NAT subnet" \
    --direction INGRESS \
    --src-ip-ranges 10.100.0.0/28 \
    --layer4-configs tcp:80  \
    --global-firewall-policy

 gcloud compute network-firewall-policies rules create 4000 \
    --action ALLOW \
    --firewall-policy producer-vpc-policy \
    --description "allow access between producer service VMs and testing client for testing purposes" \
    --direction INGRESS \
    --src-ip-ranges 10.0.0.0/28 \
    --layer4-configs tcp:80 \
    --global-firewall-policy

6. Create Producer Service [Producer Project]

We'll create a simple Apache web service using an unmanaged instance group that displays "I am a Producer Service."

Create Instance

From Cloud Shell

gcloud compute instances create producer-service-vm \
    --network producer-vpc \
    --subnet producer-service-subnet \
    --zone $zone \
    --no-address \
    --metadata startup-script='#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    a2enmod ssl
    sudo a2ensite default-ssl
    echo "I am a Producer Service." | \
    tee /var/www/html/index.html
    systemctl restart apache2'

Create Unmanaged Instance Group

From Cloud Shell

gcloud compute instance-groups unmanaged create producer-uig \
  --zone=$zone

Add producer-service-vm to the unmanaged instance group we just created.

From Cloud Shell

gcloud compute instance-groups unmanaged add-instances producer-uig \
  --zone=$zone \
  --instances=producer-service-vm

Create Load Balancer Components

For version 1 of the producer service, we'll expose the service using a Network Passthrough Load Balancer on port 80.

Deploy the health check for port 80.

From Cloud Shell

gcloud compute health-checks create http producer-hc \
        --region=$region

Deploy the backend service and add the backend to the backend service.

From Cloud Shell

gcloud compute backend-services create producer-bes \
  --load-balancing-scheme=internal \
  --protocol=tcp \
  --region=$region \
  --health-checks=producer-hc \
  --health-checks-region=$region

gcloud compute backend-services add-backend producer-bes \
  --region=$region \
  --instance-group=producer-uig \
  --instance-group-zone=$zone

Create the static IP address that will be used for the load balancer forwarding rule.

From Cloud Shell

gcloud compute addresses create producer-fr-ip\
  --region $region \
  --subnet producer-fr-subnet \
  --addresses 192.168.0.2

Create the load balancer forwarding rule exposing port 80, using the backend service and IP address we previously created.

From Cloud Shell

gcloud compute forwarding-rules create producer-fr \
  --region=$region \
  --load-balancing-scheme=internal \
  --network=producer-vpc \
  --subnet=producer-fr-subnet \
  --address=producer-fr-ip \
  --ip-protocol=TCP \
  --ports=80 \
  --backend-service=producer-bes \
  --backend-service-region=$region

7. Test the Producer Service [Producer Project]

Create a test VM in the producer-service-subnet, log into the VM and test a call to the Apache service.

From Cloud Shell

 gcloud compute instances create producer-test-vm \
    --zone=$zone \
    --subnet=producer-service-subnet \
    --no-address

From Cloud Shell

gcloud compute ssh producer-test-vm \
    --zone=$zone \
    --command="curl -s 192.168.0.2/index.html"

Expected output

I am a Producer Service.

8. Expose Producer Service Through Service Attachment [Producer Project]

Create the service attachment. Associate it with the forwarding rule we created using the psc-nat-subnet.

From Cloud Shell

gcloud compute service-attachments create producer-attachment \
    --region=$region \
    --producer-forwarding-rule=producer-fr  \
    --connection-preference=ACCEPT_AUTOMATIC \
    --nat-subnets=psc-nat-subnet

You should note down the Service Attachment URI (selfLink) as you will need it in an upcoming step for the PSC Endpoint configuration. You can obtain it by executing the following in Cloud Shell.

From Cloud Shell

gcloud compute service-attachments describe producer-attachment --region=$region

Sample output

connectionPreference: ACCEPT_AUTOMATIC
creationTimestamp: '2026-02-10T07:50:04.250-08:00'
description: ''
enableProxyProtocol: false
fingerprint: xxx
id: 'xxx'
kind: compute#serviceAttachment
name: producer-attachment
natSubnets:
- https://www.googleapis.com/compute/v1/projects/$project/regions/$region/subnetworks/psc-nat-subnet
pscServiceAttachmentId:
  high: 'xxx'
  low: 'xxx'
reconcileConnections: false
region: https://www.googleapis.com/compute/v1/projects/$project/regions/$region
selfLink: https://www.googleapis.com/compute/v1/projects/$project/regions/$region/serviceAttachments/producer-attachment
targetService: https://www.googleapis.com/compute/v1/projects/$project/regions/$region/forwardingRules/producer-fr

Copy the URI starting from projects

Example: projects/$project/regions/$region/serviceAttachments/producer-attachment

9. Build Consumer Network [Consumer Project]

Create VPC Network

From Cloud Shell

gcloud compute networks create consumer-vpc \
    --subnet-mode=custom

Create Subnet

A single subnet will be deployed where we will deploy the PSC endpoint and a test client VM.

From Cloud Shell

gcloud compute networks subnets create consumer-subnet \
    --network=consumer-vpc \
    --range=10.0.0.0/28 \
    --region=$region

Create Network Firewall Policy and Rules

From Cloud Shell

gcloud compute network-firewall-policies create consumer-vpc-policy --global

gcloud compute network-firewall-policies associations create \
    --firewall-policy consumer-vpc-policy \
    --network consumer-vpc \
    --name network-consumer-vpc \
    --global-firewall-policy

We'll also configure the consumer-vpc to use IAP for SSH.

From Cloud Shell

gcloud compute network-firewall-policies rules create 1000 \
    --action ALLOW \
    --firewall-policy consumer-vpc-policy \
    --description "SSH with IAP" \
    --direction INGRESS \
    --src-ip-ranges 35.235.240.0/20 \
    --layer4-configs tcp:22  \
    --global-firewall-policy

10. Create the Private Service Connect Endpoint [Consumer Project]

Reserve the static IP address that will be used for the PSC endpoint.

From Cloud Shell

gcloud compute addresses create psc-endpoint-ip \
    --region=$region \
    --subnet=consumer-subnet \
    --addresses 10.0.0.2

Create the PSC endpoint using the reserved static IP address and the service attachment URI that we noted earlier.

From Cloud Shell

gcloud compute forwarding-rules create psc-endpoint \
  --region=$region \
  --network=consumer-vpc \
  --address=psc-endpoint-ip \
  --target-service-attachment=projects/$project/regions/$region/serviceAttachments/producer-attachment

11. Configure a private DNS zone for the PSC Endpoint [Consumer Project]

Using Cloud DNS, we will configure a private DNS zone with the DNS name myservice.com, to be used with our PSC endpoint.

From Cloud Shell

gcloud dns managed-zones create "consumer-service" \
    --dns-name=myservice.com \
    --description="consumer service dns" \
    --visibility=private \
    --networks=consumer-vpc

gcloud dns record-sets transaction start \
   --zone="consumer-service"

Create an A record for myservice.com and point it to the PSC endpoint IP address.

From Cloud Shell

gcloud dns record-sets transaction add 10.0.0.2 \
   --name=myservice.com \
   --ttl=300 \
   --type=A \
   --zone="consumer-service"

gcloud dns record-sets transaction execute \
   --zone="consumer-service"

12. Test the PSC Endpoint [Consumer Project]

Create a client VM

From Cloud Shell

gcloud compute instances create consumer-client-vm \
    --zone=$zone \
    --subnet=consumer-subnet \
    --no-address

Test PSC endpoint connectivity

From Cloud Shell

gcloud compute ssh consumer-client-vm \
    --zone=$zone \
    --command="curl -s myservice.com/index.html"

Expected output

I am a Producer Service.

In the next part of the codelab, we will create and migrate to the new producer service, showcasing the PSC Mutability feature that allows this seamless update without requiring any updates on the consumer side configuration.

13. Create Updated Producer Service [Producer Project]

To update our producer service, we'll deploy a new VM running a similar Apache web server with the message "I am a NEW Producer service." We'll add that VM to a new unmanaged instance group and use that unmanaged instance group as a backend for our new load balancer. Instead of using a Network Passthrough load balancer, we'll update our load balancer to a regional internal Application load balancer.

Update the Network

Create a new subnet that will be used for the proxy-only subnet for the Application load balancer.

From Cloud Shell

gcloud compute networks subnets create lb-proxy-subnet \
    --network=producer-vpc \
    --range=10.200.0.0/24 \
    --region=$region \
    --purpose=REGIONAL_MANAGED_PROXY \
    --role=ACTIVE

Create a new subnet where our new service will be hosted.

From Cloud Shell

gcloud compute networks subnets create producer-service-new-subnet \
    --network=producer-vpc \
    --range=10.0.1.0/28 \
    --region=$region

Create a new firewall rule that will allow connectivity to our new producer service from the proxy-only subnet.

From Cloud Shell

 gcloud compute network-firewall-policies rules create 3001 \
    --action ALLOW \
    --firewall-policy producer-vpc-policy \
    --description "allow access from proxy only subnet" \
    --direction INGRESS \
    --src-ip-ranges 10.200.0.0/24 \
    --layer4-configs tcp:80  \
    --global-firewall-policy

Create Instance

From Cloud Shell

gcloud compute instances create new-producer-service-vm \
    --network producer-vpc \
    --subnet producer-service-new-subnet \
    --zone $zone \
    --no-address \
    --metadata startup-script='#! /bin/bash
    sudo apt-get update
    sudo apt-get install apache2 -y
    a2enmod ssl
    sudo a2ensite default-ssl
    echo "I am a NEW Producer Service." | \
    tee /var/www/html/index.html
    systemctl restart apache2'

Create Unmanaged Instance Group

From Cloud Shell

gcloud compute instance-groups unmanaged create producer-new-uig \
  --zone=$zone

gcloud compute instance-groups unmanaged add-instances producer-new-uig \
  --zone=$zone \
  --instances=new-producer-service-vm

gcloud compute instance-groups unmanaged set-named-ports producer-new-uig \
    --named-ports=http:80 \
    --zone=$zone

Create New Load Balancer Components

Create the backend service. Note that we are using the load balancing scheme INTERNAL_MANAGED because we're updating to an internal application load balancer.

From Cloud Shell

gcloud compute backend-services create producer-new-bes \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --protocol=http \
  --region=$region \
  --health-checks=producer-hc \
  --health-checks-region=$region

Add the Unmanaged Instance Group as a backend to the backend service.

From Cloud Shell

gcloud compute backend-services add-backend producer-new-bes \
  --region=$region \
  --instance-group=producer-new-uig \
  --instance-group-zone=$zone

Create the URL Map and target HTTP proxies.

From Cloud Shell

gcloud compute url-maps create producer-url-map \
  --default-service=producer-new-bes \
  --region=$region

gcloud compute target-http-proxies create http-proxy \
  --url-map=producer-url-map \
  --region=$region

Create the static IP address that we will use for our forwarding rule. This will come from the same subnet that we used for the original forwarding rule.

From Cloud Shell

gcloud compute addresses create producer-fr-new-ip\
  --region $region \
  --subnet producer-fr-subnet \
  --addresses 192.168.0.3

Create the forwarding rule.

From Cloud Shell

gcloud compute forwarding-rules create new-producer-fr \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=producer-vpc \
  --subnet=producer-fr-subnet \
  --address=producer-fr-new-ip \
  --ports=80 \
  --region=$region \
  --target-http-proxy=http-proxy \
  --target-http-proxy-region=$region

14. Test the Updated Producer Service [Producer Project]

We'll use the same test VM that we created to test the originally deployed service.

From Cloud Shell

gcloud compute ssh producer-test-vm \
    --zone=$zone \
    --command="curl -s 192.168.0.3/index.html"

Note that we are calling an updated IP address! 192.168.0.3 is the IP address for our regional internal Application load balancer.

Expected output

I am a NEW Producer Service.

15. Update the Service Attachment [Producer Project]

To better visualize the update to the Service Attachment forwarding rule, we can run a describe on the Service Attachment before and after the change.

From Cloud Shell

gcloud compute service-attachments describe producer-attachment \
    --region=$region \
    --format="value(targetService)"

Expected output

https://www.googleapis.com/compute/v1/projects/$project/regions/$region/forwardingRules/producer-fr

Update the Service Attachment.

From Cloud Shell

gcloud compute service-attachments update producer-attachment \
  --region=$region \
  --target-service=projects/$project/regions/$region/forwardingRules/new-producer-fr

Now run the describe again to see the new forwarding rule.

From Cloud Shell

gcloud compute service-attachments describe producer-attachment \
    --region=$region \
    --format="value(targetService)"

Expected output

https://www.googleapis.com/compute/v1/projects/$project/regions/$region/forwardingRules/new-producer-fr

16. Re-Test the Consumer PSC Connection [Consumer Project]

Log into the same consumer client.

From Cloud Shell

gcloud compute ssh consumer-client-vm \
    --zone=$zone \
    --command="curl -s myservice.com/index.html"

Expected output

 I am a NEW Producer Service.

Congratulations! You've successfully updated a PSC producer service load balancer and it required no configuration changes on the consumer side!

17. Cleanup steps [Consumer Project]

From a single Cloud Shell terminal delete lab components

gcloud dns record-sets delete myservice.com --zone="consumer-service" --type=A -q

gcloud dns managed-zones delete "consumer-service" -q

gcloud compute forwarding-rules delete psc-endpoint --region=$region -q

gcloud compute addresses delete psc-endpoint-ip --region=$region -q

gcloud compute instances delete consumer-client-vm --zone=$zone --project $project -q

gcloud compute network-firewall-policies rules delete 1000 --firewall-policy consumer-vpc-policy --global-firewall-policy -q

gcloud compute network-firewall-policies associations delete --firewall-policy=consumer-vpc-policy  --name=network-consumer-vpc --global-firewall-policy -q

gcloud compute network-firewall-policies delete consumer-vpc-policy --global -q

gcloud compute networks subnets delete consumer-subnet --region $region -q

gcloud compute networks delete consumer-vpc -q

18. Cleanup steps [Producer Project]

From a single Cloud Shell terminal delete lab components

gcloud compute service-attachments delete producer-attachment --region=$region -q

gcloud compute instances delete producer-test-vm --zone=$zone --project $project -q

gcloud compute forwarding-rules delete new-producer-fr --region=$region -q

gcloud compute addresses delete producer-fr-new-ip --region $region -q

gcloud compute target-http-proxies delete http-proxy --region $region -q

gcloud compute url-maps delete producer-url-map --region $region -q

gcloud compute backend-services delete producer-new-bes --region $region -q

gcloud compute instance-groups unmanaged delete producer-new-uig --zone $zone -q

gcloud compute instances delete new-producer-service-vm --zone $zone --project $project -q

gcloud compute networks subnets delete producer-service-new-subnet --region $region -q

gcloud compute networks subnets delete lb-proxy-subnet --region $region -q

gcloud compute forwarding-rules delete producer-fr --region=$region -q

gcloud compute addresses delete producer-fr-ip --region $region -q

gcloud compute backend-services delete producer-bes --region $region -q

gcloud compute health-checks delete producer-hc --region $region -q

gcloud compute instance-groups unmanaged delete producer-uig --zone $zone -q

gcloud compute instances delete producer-service-vm --zone $zone --project $project -q

gcloud compute network-firewall-policies rules delete 4000 --firewall-policy producer-vpc-policy --global-firewall-policy -q

gcloud compute network-firewall-policies rules delete 3001 --firewall-policy producer-vpc-policy --global-firewall-policy -q

gcloud compute network-firewall-policies rules delete 3000 --firewall-policy producer-vpc-policy --global-firewall-policy -q

gcloud compute network-firewall-policies rules delete 2000 --firewall-policy producer-vpc-policy --global-firewall-policy -q

gcloud compute network-firewall-policies rules delete 1000 --firewall-policy producer-vpc-policy --global-firewall-policy -q

gcloud compute network-firewall-policies associations delete --firewall-policy=producer-vpc-policy  --name=network-producer-vpc --global-firewall-policy -q

gcloud compute network-firewall-policies delete producer-vpc-policy --global -q

gcloud compute routers nats delete $region-nat --router=$region-cr --region=$region -q

gcloud compute routers delete $region-cr --region=$region -q

gcloud compute networks subnets delete psc-nat-subnet --region=$region -q

gcloud compute networks subnets delete producer-fr-subnet --region=$region -q

gcloud compute networks subnets delete producer-service-subnet --region=$region -q

gcloud compute networks delete producer-vpc -q

19. Congratulations!

Congratulations for completing the codelab.

What we've covered

  • Create a simple Apache web service exposed as a PSC producer service.
  • Create a PSC endpoint.
  • Create a Cloud DNS private zone for the consumer service calls.
  • Update the forwarding rule associated with the service attachment using PSC Mutability.