Explicit Chaining of GCP L7 Load Balancers with PSC

1. Introduction

Private Service Connect (PSC) Network Endpoint Group (NEG) supports chaining an Internal HTTPS Load Balancer with an External HTTPS Load Balancer. This provides Distributed Health-checks and Data plane traffic to On-Prem using customer defined ranges. In addition, multiple VPCs connecting to On-Prem via multiple regional InterConnects is also supported with this topology.

In this codelab, we will demonstrate how to configure this end-to-end based on the topology below. From left to right, customers on premise have a VM to simulate HTTP services, leverage hybrid connectivity (HA-VPN or InterConnect) and hybrid NEG to expose via Internal HTTPS load balancer. PSC uses Internal HTTPS LB as service attachments. PSC NEG consumes the attachments as backend service, exposed to External HTTPS LB. Internet users can use Google global network to accelerate On-Prem HTTP services access.

Figure 1. Private Service Connect uses Network Endpoint Group and service attachments to connect External HTTPS Load Balancer to Internal HTTPS Load Balancer, and extend the backend to On-Prem.

What you'll learn

Internal HTTPS Load Balancer with Hybrid NEG and Distributed Health Check
PSC Service attachment with Internal HTTPS Load Balancer
PSC Network Endpoint Group Setup
Expose PSC NEG with External HTTPS Load Balancer

What you'll need

Knowledge of Hybrid Connectivity, like HA-VPN
Knowledge of Internal/External HTTPS Load Balancing
Knowledge of Private Service Connect

2. Before you begin

Note: Codelab offers configuration and validation steps based on the illustrated topology, modify the procedure as needed to meet your organization's requirements. IAM permissions are not in scope of the codelab.

Codelab will use one project to simulate the whole process. Multiple projects are also supported.

Single Project - Update project to support producer and consumer network

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
prodproject=YOUR-PROJECT-NAME
echo $prodproject

3. Create On-Prem Resources

In the following section, we will set up an on-prem VPC and VMs to simulate customer on premise services.

VPC Network

From Cloud Shell

gcloud compute networks create vpc-demo-onprem --project=$prodproject --subnet-mode=custom

Create Subnet

From Cloud Shell

gcloud compute networks subnets create vpc-demo-onprem-asia-southeast1 --project=$prodproject --range=10.0.0.0/24 --network=vpc-demo-onprem --region=asia-southeast1

Create Firewall rules.

Internal HTTPS Load Balancer supports distributed health check, firewall rules only need to allow proxy subnet IP range. Following doc to allowlist your projects.

From Cloud Shell create a firewall rule to enable backend health checks and data plane traffic from proxy subnets.

gcloud compute firewall-rules create vpc-demo-health-checks --allow tcp:80,tcp:443 --network vpc-demo-onprem --source-ranges 10.0.3.0/24 --enable-logging

From Cloud Shell create a firewall rule to allow IAP to connect to your VM instances,

gcloud compute firewall-rules create psclab-iap-prod --network vpc-demo-onprem --allow tcp:22 --source-ranges=35.235.240.0/20 --enable-logging

4. Create On-Prem VM Instances

This VM simulates on premise services, and needs to be exposed with Internal HTTPS Load Balancer using hybrid NEG.

From Cloud Shell create instance www01

gcloud compute instances create www01 \
    --zone=asia-southeast1-b \
    --image-family=debian-11 \
    --image-project=debian-cloud \
    --network-interface=network-tier=PREMIUM,nic-type=GVNIC,stack-type=IPV4_ONLY,subnet=vpc-demo-onprem-asia-southeast1 \
    --shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --metadata=startup-script='#! /bin/bash
sudo apt-get update
sudo apt-get install nginx -y
vm_hostname="$(curl -H "Metadata-Flavor:Google" \
http://169.254.169.254/computeMetadata/v1/instance/name)"
filter="{print \$NF}"
vm_zone="$(curl -H "Metadata-Flavor:Google" \
http://169.254.169.254/computeMetadata/v1/instance/zone \
| awk -F/ "${filter}")"
echo "Page on $vm_hostname in $vm_zone" | \
tee /var/www/html/index.nginx-debian.html
sudo systemctl restart nginx'

In the following section, we will use letsencrypt to generate certificates and install on Nginx. Download the public and private key file for the next step. You need to temporarily open TCP port 80 to the Internet for certificate generation.

Make sure this VM has a publicly resolved domain name. For example, in Cloud DNS add a A record [www01.yinghli.demo.altostrat.com](http://www01.yinghli.demo.altostrat.com) and point to VM public IP address.

gcloud dns --project=$prodproject record-sets create www01.yinghli.demo.altostrat.com. --zone="yinghli-demo" --type="A" --ttl="300" --rrdatas="34.87.77.186"

From VM www01 console, follow the guidance to install certificates on Nginx and make a copy of fullchain.pem and private.pem for following steps.

sudo apt install snapd
sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot --nginx

5. Create Producers VPC network

Note: Hybrid networking configuration is NOT included in this configuration.

VPC Network

From Cloud Shell

gcloud compute networks create vpc-demo-producer --project=$prodproject --subnet-mode=custom

Create Subnet

From Cloud Shell

gcloud compute networks subnets create vpc-demo-asia-southeast1 --project=$prodproject --range=10.0.2.0/24 --network=vpc-demo-producer --region=asia-southeast1

Create Proxy Subnet

From Cloud Shell

gcloud compute networks subnets create proxy-subnet-asia-southeast1 \
  --purpose=REGIONAL_MANAGED_PROXY \
  --role=ACTIVE \
  --region=asia-southeast1 \
  --network=vpc-demo-producer \
  --range=10.0.3.0/24

Hybrid Connectivity

Follow Cloud VPN documentation to implement HA-VPN connectivity between On-Prem and Producer VPC. Keep default configuration on the Cloud Router, we don't need to add 130.211.0.0/22, 35.191.0.0/16 into BGP advertisements.

6. Create Producers Hybrid NEG

Create a hybrid network endpoint group and add on-prem VM IP:PORT into NEG.

From Cloud Shell

gcloud compute network-endpoint-groups create on-prem-service-neg \
    --network-endpoint-type=NON_GCP_PRIVATE_IP_PORT \
    --zone=asia-southeast1-b \
    --network=vpc-demo-producer

gcloud compute network-endpoint-groups update on-prem-service-neg \
    --zone=asia-southeast1-b \
    --add-endpoint="ip=10.0.0.2,port=443"

7. Create Producers Internal HTTPS Load Balancer

Currently External HTTPS Load Balancer only supports HTTPS protocol to PSC NEG( docs). When published services, we need to use Internal HTTPS Load Balancer and enable forwarding rules global access.

From Cloud Shell create the regional health check.

gcloud compute health-checks create https on-prem-service-hc \
    --region=asia-southeast1 \
    --use-serving-port

From Cloud Shell create the backend service and add Hybrid NEG.

gcloud compute backend-services create on-premise-service-backend \
   --load-balancing-scheme=INTERNAL_MANAGED \
   --protocol=HTTPS \
   --region=asia-southeast1 \
   --health-checks=on-prem-service-hc \
   --health-checks-region=asia-southeast1

gcloud compute backend-services add-backend on-premise-service-backend \
   --network-endpoint-group=on-prem-service-neg \
   --network-endpoint-group-zone=asia-southeast1-b \
   --region=asia-southeast1 \
   --balancing-mode=RATE \
   --max-rate-per-endpoint=100

From Cloud Shell create the URL map

gcloud compute url-maps create on-premise-url \
    --default-service on-premise-service-backend \
    --region=asia-southeast1

From Cloud Shell create the regional SSL certificates. Two certificate files are downloaded from the VM.

gcloud compute ssl-certificates create www01 \
    --certificate=fullchain.pem \
    --private-key=private.pem \
    --region=asia-southeast1

From Cloud Shell create https-target-proxy

gcloud compute target-https-proxies create on-premise-httpsproxy \
    --ssl-certificates=www01 \
    --url-map=on-premise-url \
    --url-map-region=asia-southeast1 \
    --region=asia-southeast1

From Cloud Shell reserve a internal static IP and create the forwarding rule

gcloud compute addresses create ilbaddress \
  --region=asia-southeast1 \
  --subnet=vpc-demo-asia-southeast1  \
  --addresses=10.0.2.100 

gcloud compute forwarding-rules create https-ilb-psc \
      --load-balancing-scheme=INTERNAL_MANAGED \
      --network=vpc-demo-producer \
      --subnet=vpc-demo-asia-southeast1  \
      --address=ilbaddress \
      --ports=443 \
      --region=asia-southeast1 \
      --target-https-proxy=on-premise-httpsproxy \
      --target-https-proxy-region=asia-southeast1
      --allow-global-access

8. Create Producer VM instance

Create a producer VM for verification.

From Cloud Shell

gcloud compute instances create test01 \
    --zone=asia-southeast1-b \
    --image-family=debian-11 \
    --image-project=debian-cloud \
    --network-interface=network-tier=PREMIUM,nic-type=GVNIC,stack-type=IPV4_ONLY,subnet=vpc-demo-asia-southeast1 \
    --shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring

To allow IAP to connect to your VM instances, create a firewall rule that:

From Cloud Shell

gcloud compute firewall-rules create psclab-iap-prod --network vpc-demo-producer --allow tcp:22 --source-ranges=35.235.240.0/20 --enable-logging

From the producer VM console, access [www01.yinghli.demo.altostrat.com](https://www01.yinghli.demo.altostrat.com) and resolve the Internal HTTPS load balancer IP address. HTTP 200 indicated that configuration worked as expected.

curl -v --resolve www01.yinghli.demo.altostrat.com:443:10.0.2.100 https://www01.yinghli.demo.altostrat.com

* Added www01.yinghli.demo.altostrat.com:443:10.0.2.100 to DNS cache
* Hostname www01.yinghli.demo.altostrat.com was found in DNS cache
*   Trying 10.0.2.100:443...
* Connected to www01.yinghli.demo.altostrat.com (10.0.2.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www01.yinghli.demo.altostrat.com
*  start date: Jun  4 10:36:43 2023 GMT
*  expire date: Sep  2 10:36:42 2023 GMT
*  subjectAltName: host "www01.yinghli.demo.altostrat.com" matched cert's "www01.yinghli.demo.altostrat.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55865ef982e0)
> GET / HTTP/2
> Host: www01.yinghli.demo.altostrat.com
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< server: nginx/1.18.0
< date: Mon, 05 Jun 2023 02:29:38 GMT
< content-type: text/html
< content-length: 35
< last-modified: Sun, 04 Jun 2023 09:02:16 GMT
< etag: "647c5318-23"
< accept-ranges: bytes
< via: 1.1 google
< 
Page on www01 in asia-southeast1-b
* Connection #0 to host www01.yinghli.demo.altostrat.com left intact

Note: You can't access VM 10.0.0.2 HTTPS services directly, because on-prem firewall only allows proxy subnet 10.0.3.0/24 to access.

9. Create PSC NAT subnet

From Cloud Shell

gcloud compute networks subnets create psc-nat-subnet \
--network=vpc-demo-producer \
--region=asia-southeast1 \
--range=10.0.5.0/24 \
--purpose=private-service-connect

10. Create HTTPs service attachment

From Cloud Shell create the HTTPs service attachment

gcloud compute service-attachments create ilbserviceattach \
--region=asia-southeast1 \
--producer-forwarding-rule=https-ilb-psc \
--connection-preference=ACCEPT_AUTOMATIC \
--nat-subnets=psc-nat-subnet

Validate the HTTPs service attachment

gcloud compute service-attachments describe ilbserviceattach --region asia-southeast1

Record service attachment name:

projects/<project>/regions/asia-southeast1/serviceAttachments/ilbserviceattach

11. Create Consumers VPC network

In the following section, the consumer VPC is configured in the same project, but different projects are also supported. Communication between the consumer and producer network is accomplished through the service attachment defined in the producer network.

VPC Network

From Cloud Shell

gcloud compute networks create vpc-demo-consumer --project=$prodproject --subnet-mode=custom

Create Subnet

From Cloud Shell

gcloud compute networks subnets create consumer-subnet --project=$prodproject  --range=10.0.6.0/24 --network=vpc-demo-consumer --region=asia-southeast1

12. Create PSC Network Endpoint Group

Create PSC NEG

Copy previous https services attachment name and paste in parameters --psc-target-service

From Cloud Shell

gcloud beta compute network-endpoint-groups create consumerpscneg \
--project=$prodproject \
--region=asia-southeast1 \
--network-endpoint-type=PRIVATE_SERVICE_CONNECT \
--psc-target-service=projects/<project>/regions/asia-southeast1/serviceAttachments/ilbserviceattach \
--network=vpc-demo-consumer \
--subnet=consumer-subnet

After PSC NEG setup successfully, from UI, following Private Service Connect -> Published Services -> Note the published ilbserviceattach connection now indicates 1 forwarding rule.

13. Create Consumer External HTTPS Load Balancer

Create an External HTTPS Load Balancer and use PSC NEG as backend services( documentation).

From Cloud Shell

gcloud compute addresses create httpspsclb \
    --ip-version=IPV4 --global

gcloud compute backend-services create consumer-bs \
    --load-balancing-scheme=EXTERNAL_MANAGED \
    --protocol=HTTPS \
    --global

gcloud compute backend-services add-backend consumer-bs \
  --network-endpoint-group=consumerpscneg \
  --network-endpoint-group-region=asia-southeast1 \
  --global

gcloud compute url-maps create consumer-url \
  --default-service=consumer-backend-service \
  --global

gcloud compute ssl-certificates create wwwglobal \
    --certificate=fullchain.pem \
    --private-key=private.pem \
    --global

gcloud compute target-https-proxies create consumer-url-target-proxy \
  --url-map=consumer-url \
  --ssl-certificates=wwwglobal

gcloud compute forwarding-rules create consumer-url-forwarding-rule \
    --load-balancing-scheme=EXTERNAL_MANAGED \
    --network-tier=PREMIUM \
    --address=httpspsclb \
    --target-https-proxy=consumer-url-target-proxy \
    --ports=443 \
    --global

Update DNS record for www01.yinghli.demo.altostrat.com and point to External HTTPS Load Balancer public IP address

gcloud dns --project=$prodproject record-sets update www01.yinghli.demo.altostrat.com. --type="A" --zone="yinghli-demo" --rrdatas="34.102.178.214" --ttl="300"

14. Validation

From your laptop, access https://www01.yinghli.demo.altostrat.com with curl.

curl -v https://www01.yinghli.demo.altostrat.com
*   Trying 34.102.178.214:443...
* Connected to www01.yinghli.demo.altostrat.com (34.102.178.214) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=www01.yinghli.demo.altostrat.com
*  start date: Jun  4 10:36:43 2023 GMT
*  expire date: Sep  2 10:36:42 2023 GMT
*  subjectAltName: host "www01.yinghli.demo.altostrat.com" matched cert's "www01.yinghli.demo.altostrat.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: www01.yinghli.demo.altostrat.com]
* h2h3 [user-agent: curl/8.0.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x149019a00)
> GET / HTTP/2
> Host: www01.yinghli.demo.altostrat.com
> user-agent: curl/8.0.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< server: nginx/1.18.0
< date: Mon, 05 Jun 2023 02:48:43 GMT
< content-type: text/html
< content-length: 35
< last-modified: Sun, 04 Jun 2023 09:02:16 GMT
< etag: "647c5318-23"
< accept-ranges: bytes
< via: 1.1 google, 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
Page on www01 in asia-southeast1-b
* Connection #0 to host www01.yinghli.demo.altostrat.com left intact

15. Cleanup steps

Producer network clean up steps

Note: Cleanup steps only show Load Balancer and PSC related configuration, VPC and Hybrid Connectivity is not included.

From a single cloud shell in the terminal delete lab components

gcloud compute forwarding-rules delete consumer-url-forwarding-rule --global
gcloud compute target-https-proxies delete consumer-url-target-proxy
gcloud compute ssl-certificates delete wwwglobal --global
gcloud compute url-maps delete consumer-url
gcloud compute backend-services delete consumer-bs --global
gcloud compute addresses delete httpspsclb --global

gcloud beta compute network-endpoint-groups delete consumerpscneg --region=asia-southeast1

gcloud compute service-attachments delete ilbserviceattach --region=asia-southeast1
gcloud compute networks subnets delete psc-nat-subnet --region=asia-southeast1

gcloud compute forwarding-rules delete https-ilb-psc --region=asia-southeast1
gcloud compute addresses delete ilbaddress --region=asia-southeast1
gcloud compute target-https-proxies delete on-premise-httpsproxy --region=asia-southeast1
gcloud compute ssl-certificates delete www01 --region=asia-southeast1
gcloud compute url-maps delete on-premise-url --region=asia-southeast1
gcloud compute backend-services delete on-premise-service-backend --region=asia-southeast1
gcloud compute health-checks delete on-prem-service-hc --region=asia-southeast1

gcloud compute network-endpoint-groups delete on-prem-service-neg --zone=asia-southeast1-b
gcloud compute networks subnets delete proxy-subnet-asia-southeast1 --region=asia-southeast1

16. Congratulations!

Congratulations for completing the codelab.

What we've covered

Internal HTTPS Load Balancer with Hybrid NEG and Distributed Health Check
PSC Service attachment with Internal HTTPS Load Balancer
PSC Network Endpoint Group Setup
Expose PSC NEG with External HTTPS Load Balancer