Private Service Connect - Using Consumer HTTP(S) Service Controls for Global XLB to Managed Services

1. Introduction

Private Service Connect lets a service producer offer services to a service consumer. A service producer VPC network can support multiple service consumers.

There are two types of Private Service Connect endpoints that can connect to a published service:

• Private Service Connect endpoint (based on a forwarding rule)

With this endpoint type, consumers connect to an internal IP address that they define. Private Service Connect performs network address translation (NAT) to route the request to the service producer.

• Private Service Connect endpoint with consumer HTTP(S) service controls (based on a global external HTTP(S) load balancer)

With this endpoint type, consumers connect to an external IP address. Private Service Connect uses a network endpoint group to route the request to the service producer.

Using a global external HTTP(S) load balancer as a policy enforcement point has the following benefits:

• You can rename services and map them to URLs of your choice.
• You can configure the load balancer to log all requests to Cloud Logging.
• You can use customer-managed TLS certificates. or Google-managed certificates.

In this codelab, you will learn about how to create a Private Service Connect endpoint Consumer HTTP(S) Service Controls using the Global XLB to privately access a service in another network. This PSC pattern can be done using a single project or separate projects. For the purposes of this lab, we will use a single project with two separate VPCs.

What you'll learn

• Create a Private Service Connect endpoint with Consumer HTTP(S) Service Controls using the Global XLB
• Configure a managed service to be exposed through a Service Attachment to accept L7 XLB connections.
• Create an SSL certificate and configure an Apache web server to terminate TLS and accept traffic on port 443.
• Create a PSC NEG.

What you'll need

• A Google Cloud project
• Knowledge of deploying instances and configuring networking components

2. Test Environment

The environment that you will create will consist of an External HTTP(S) Load Balancer and PSC NEG in a consumer VPC. The producer VPC will host a simple Apache web service configured with HTTPS. You will create a backend service from the Apache web service and front that backend service with an Internal TCP Load Balancer configured with a PSC Service Attachment.

3. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs, and you can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" after the project is created.
• There is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. If a project is deleted, that ID can never be used again.

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console in order to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, follow any "clean-up" instructions found at the end of the codelab. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

4. Before you begin

Enable APIs

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
export project=YOUR-PROJECT-NAME
export region=us-central1
echo $project
echo $region

Enable all necessary services

gcloud services enable compute.googleapis.com
gcloud services enable servicedirectory.googleapis.com

5. Producer VPC, Subnet, Firewall Rules Setup

VPC Network

From Cloud Shell

gcloud compute networks create producer-vpc --subnet-mode custom

Create Subnets

A subnet is needed on the producer side to perform the Network Address Translation (NAT) for PSC. Note that the purpose is PRIVATE_SERVICE_CONNECT. This means that this subnet cannot be used for deploying workloads.

From Cloud Shell

gcloud compute networks subnets create producer-nat-subnet \
--network=producer-vpc \
--region=$region \
--range=10.100.100.0/24 \
--purpose=PRIVATE_SERVICE_CONNECT

We will deploy two subnets in the producer VPC. The first to deploy the producer service, and an additional in another region to deploy a client-vm to test the connectivity to the service via Global Access on the TCP Internal Load Balancer.

From Cloud Shell

gcloud compute networks subnets create service-subnet \
    --network=producer-vpc \
    --range=10.0.0.0/24 \
    --region=$region

From Cloud Shell

gcloud compute networks subnets create client-subnet \
    --network=producer-vpc \
    --range=10.0.1.0/24 \
    --region=us-east4

Create Cloud NAT

A Cloud NAT is required to install the proper packages for our producer services.

From Cloud Shell

gcloud compute routers create service-cr \
--region=$region --network=producer-vpc \
--asn=65501

From Cloud Shell

gcloud compute routers nats create service-nat-gw \
--router=service-cr \
--router-region=$region \
--nat-custom-subnet-ip-ranges=service-subnet \
--auto-allocate-nat-external-ips

Create Firewall Rules

For this lab, you will be using IAP to connect to the instances you create. The following firewall rule will enable you to connect to instances through IAP. If you prefer not to use IAP, you can skip this step, and instead add public IP addresses on the instance and create a firewall rule that allows ingress on TCP port 22 from 0.0.0.0/0.

To allow IAP to connect to your VM instances, create a firewall rule that:

• Applies to all VM instances that you want to be accessible by using IAP.
• Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.

From Cloud Shell

gcloud compute firewall-rules create allow-ssh-iap \
    --network producer-vpc \
--allow tcp:22 \
--source-ranges=35.235.240.0/20

Client traffic will be originating from the Global External HTTP(S) load balancer, so a Firewall rule must be created to allow this traffic to the tagged destination servers that will be hosting our web service. We will also open the firewall rule from our client-subnet for testing purposes.

From Cloud Shell

gcloud compute firewall-rules create allow-xlb-client \
  --network=producer-vpc \
  --direction=ingress \
  --allow=tcp:443 \
  --target-tags=psc-service \
  --source-ranges=130.211.0.0/22,35.191.0.0/16,10.0.1.0/24

Create Apache Web Service

We'll create a simple Apache Web Service that displays "PSC Service"

Create Instance Template

From Cloud Shell

gcloud compute instance-templates create producer-service-template \
    --network producer-vpc \
    --subnet service-subnet \
    --region $region \
    --no-address \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --tags=psc-service \
    --metadata startup-script='#! /bin/bash
    sudo apt-get update
    apt-get install apache2 -y
    a2ensite default-ssl
    echo "PSC Service" | \
    tee /var/www/html/index.html
    systemctl restart apache2'

Create Health Check for MIG

From Cloud Shell

gcloud compute health-checks create https psc-service-mig-healthcheck \
    --port=443 \
    --global

Create Managed Instance Group

From Cloud Shell

gcloud compute instance-groups managed create psc-service-mig \
    --region $region \
    --size=2 \
    --template=producer-service-template \
    --health-check=psc-service-mig-healthcheck

Configure SSL on Apache Web Servers

Next, we'll need to configure SSL on each of the Apache Web servers. We'll do this by generating a certificate and adding that certificate to the Apache Configuration.

SSL termination must be configured on the backend services because the service must be fronted by an internal TCP/UDP (L4) load balancer for this particular PSC pattern. The internal TCP/UDP load balancer does not terminate SSL at the load balancer layer.

Start by SSHing into the first VM in your MIG. VM zone and VM name will be dynamically allocated per environment. Navigate, in the console, to Compute Engine > VM Instances to find the name and zone of your instances.

From Cloud Shell

gcloud compute ssh --zone "<YOUR_VM_ZONE>" "<YOUR_MIG_VM_1>"  --tunnel-through-iap --project $project

Next we'll create the cert through OpenSSL. You'll be asked to fill in information about your Country, State, Locality, Organization, Organizational Unit Name, Common Name, and email address. The only information you need to fill in should be the Common Name, which should be an internal FQDN of your choosing. For the purpose of this lab, you should choose example.com.

From Cloud Shell

sudo openssl genrsa -out private-key-file.pem 2048

From Cloud Shell

cat <<'EOF' >config.txt
[req]
default_bits              = 2048
req_extensions            = extension_requirements
distinguished_name        = dn_requirements

[extension_requirements]
basicConstraints          = CA:FALSE
keyUsage                  = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName            = @sans_list

[dn_requirements]
countryName               = Country Name (2 letter code)
stateOrProvinceName       = State or Province Name (full name)
localityName              = Locality Name (eg, city)
0.organizationName        = Organization Name (eg, company)
organizationalUnitName    = Organizational Unit Name (eg, section)
commonName                = Common Name (e.g. server FQDN or YOUR name)
emailAddress              = Email Address

[sans_list]
DNS.1                     = example.com

EOF

From Cloud Shell

sudo openssl req -new -key private-key-file.pem \
    -out csr.pem \
    -config config.txt

From Cloud Shell

sudo openssl x509 -req \
    -signkey private-key-file.pem \
    -in csr.pem \
    -out cert.cert \
    -extfile config.txt \
    -extensions extension_requirements \
    -days 10

Now let's update the Apache configuration information with our new certificate details.

sudo vi /etc/apache2/sites-enabled/default-ssl.conf

Add a line under ServerAdmin that reads

ServerName example.com

Update the SSLCertificateFile and SSLCertificateKeyFile the location of the cert.cert file and private-key-file.pem locations on your VM. An example is shown below. Make sure to update <profile> with your directory name.

SSLCertificateFile  /home/<profile>/cert.cert

SSLCertificateKeyFile /home/<profile>/private-key-file.pem

Close the editor and restart Apache.

sudo a2enmod ssl
sudo systemctl restart apache2

Exit from the instance and repeat the same steps on the other instance in the managed instance group.

6. Create the Producer Service

Next we'll create the load balancer components for the service.

Create the load balancer health check.

From Cloud Shell

gcloud compute health-checks create https service-lb-healthcheck \
    --port=443 \
    --region=$region

Create the backend service.

From Cloud Shell

 gcloud compute backend-services create psc-backend-service \
    --load-balancing-scheme=internal \
    --protocol=TCP \
    --region=$region \
    --health-checks=service-lb-healthcheck \
    --health-checks-region=$region

gcloud compute backend-services add-backend psc-backend-service \
--region=$region \
--instance-group=psc-service-mig

Create the Forwarding Rule. Note that the forwarding rule must be configured on port 443 and with Global Access. This is required for this PSC pattern to work.

From Cloud Shell

 gcloud compute forwarding-rules create producer-fr \
    --region=$region \
    --load-balancing-scheme=internal \
    --network=producer-vpc \
    --subnet=service-subnet \
    --address=10.0.0.100 \
    --ip-protocol=TCP \
    --ports=443 \
    --backend-service=psc-backend-service \
    --backend-service-region=$region \
    --allow-global-access

7. Test the Service

Before we create the Service Attachment, we'll create a client in a different region to test the Load Balancer configured with Global Access, as well as the Apache service configured to terminate TLS.

From Cloud Shell

gcloud compute instances create vm-client \
    --zone=us-east4-a \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --subnet=client-subnet \
    --no-address

SSH into the instance.

From Cloud Shell

gcloud compute ssh \
    --zone "us-east4-a" "vm-client" \
    --tunnel-through-iap \
    --project $project

Test the Apache Service by connecting over 443 through the load balancer.

curl https://example.com:443 -k --connect-to example.com:443:10.0.0.100:443

EXPECTED RESULT

PSC Service

8. Create the Service Attachment

From Cloud Shell

gcloud compute service-attachments create pscservice \
    --region=$region \
    --producer-forwarding-rule=producer-fr \
    --connection-preference=ACCEPT-AUTOMATIC \
    --nat-subnets=producer-nat-subnet

You should note down the Service Attachment URI as you will need it in the next step for the endpoint configuration. You can obtain it by executing the following in Cloud Shell

From Cloud Shell

gcloud compute service-attachments describe pscservice --region $region

Copy the URI starting from /projects

Example: /projects/<YOUR_PROJECT_ID>/regions/us-central1/serviceAttachments/pscservice

9. Consumer VPC and Subnet Setup

VPC Network

From Cloud Shell

gcloud compute networks create consumer-vpc --subnet-mode custom

Create Subnet

A subnet is needed on the consumer side where the Private Service Connect Network Endpoint Group (NEG) will be deployed.

From Cloud Shell

gcloud compute networks subnets create psc-neg-subnet \
--network=consumer-vpc \
--region=$region \
--range=10.100.200.0/24 \
--purpose=private

10. Create the Private Service Connect Endpoint and Test Connection

We'll follow the process of creating a PSC NEG that will be associated with the Service Attachment we just created, attach the PSC NEG to a backend service, and associate the backend service with a forwarding rule.

Have handy the Service Attachment URI we noted in the last step. Replace the URL below with your URI.

From Cloud Shell

gcloud beta compute network-endpoint-groups create xlb-psc-neg \
--network-endpoint-type=private-service-connect \
--psc-target-service=projects/<PROJECT-ID>/regions/us-central1/serviceAttachments/pscservice \
--region=$region \
--network=consumer-vpc \
--subnet=psc-neg-subnet

Create the XLB public IP address and grab the actual IP address assigned for later testing.

From Cloud Shell

gcloud compute addresses create xlb-psc-address \
--ip-version=IPv4 --global

gcloud compute addresses describe xlb-psc-address --format="get(address)" --global

Next we'll create the PSC endpoint, which in this case, in an External Load Balancer.

From Cloud Shell

gcloud beta compute backend-services create pscneg-backend-service \
--load-balancing-scheme=EXTERNAL_MANAGED \
--protocol=HTTPS \
--global

From Cloud Shell

gcloud beta compute backend-services add-backend pscneg-backend-service \
--network-endpoint-group=xlb-psc-neg \
--network-endpoint-group-region=$region \
--global

From Cloud Shell

gcloud beta compute url-maps create xlb-psc-map \
--default-service=pscneg-backend-service \
--global

From Cloud Shell

gcloud beta compute target-http-proxies create psc-http-proxy \
--url-map=xlb-psc-map

From Cloud Shell

gcloud beta compute forwarding-rules create xlb-psc-fr \
--load-balancing-scheme=EXTERNAL_MANAGED \
--network-tier=PREMIUM \
--address=xlb-psc-address \
--target-http-proxy=psc-http-proxy \
--ports=80 \
--global

Wait 5 - 7 minutes and then input the IP address associated with xlb-psc-address in the address bar of your browser.

If "PSC Service" is displayed, you have correctly configured the solution.

11. Cleanup steps

From a single Cloud Shell terminal delete lab components

gcloud beta compute forwarding-rules delete xlb-psc-fr --global --quiet

gcloud beta compute target-http-proxies delete psc-http-proxy --quiet

gcloud beta compute url-maps delete xlb-psc-map --global --quiet

gcloud beta compute backend-services delete pscneg-backend-service --global --quiet

gcloud compute addresses delete xlb-psc-address --global --quiet

gcloud beta compute network-endpoint-groups delete xlb-psc-neg --region $region --quiet

gcloud compute networks subnets delete psc-neg-subnet --region $region --quiet

gcloud compute networks delete consumer-vpc --quiet

gcloud compute service-attachments delete pscservice --region $region --quiet

gcloud compute instances delete vm-client --zone=us-east4-a --quiet

gcloud compute forwarding-rules delete producer-fr --region $region --quiet

gcloud compute backend-services delete psc-backend-service --region $region --quiet

gcloud compute health-checks delete service-lb-healthcheck --region $region --quiet

gcloud compute instance-groups managed delete psc-service-mig --region $region --quiet

gcloud compute health-checks delete psc-service-mig-healthcheck --region $region --quiet

gcloud compute instance-templates delete producer-service-template --quiet

gcloud compute firewall-rules delete allow-xlb-client --quiet

gcloud compute firewall-rules delete allow-ssh-iap --quiet

gcloud compute routers nats delete service-nat-gw –router service-cr --region $region --quiet

gcloud compute routers delete service-cr --region $region --quiet

gcloud compute networks subnets delete client-subnet --quiet

gcloud compute networks subnets delete service-subnet --quiet

gcloud compute networks subnets delete producer-nat-subnet --quiet

gcloud compute networks delete producer-vpc --quiet

12. Congratulations!

Congratulations for completing the codelab.

What we've covered

• Create a Private Service Connect endpoint with Consumer HTTP(S) Service Controls using the Global XLB
• Configure a managed service to be exposed through a Service Attachment to accept L7 XLB connections.
• Create an SSL certificate and configure an Apache web server to terminate TLS and accept traffic on port 443.
• Create a PSC NEG.