Private Service Connect - Using Consumer HTTP(S) Service Controls for Regional Google APIs

1. Introduction

With Private Service Connect, you can create private endpoints using global internal IP addresses within your VPC network to access Google APIs. Building upon that concept, you can now create a Private Service Connect endpoint with consumer HTTP(S) service controls using an internal HTTP(S) load balancer. This provides you the following features:

  • You can choose which services are available using a URL map; filtering by path lets you do more fine-grained checks.
  • You can rename services, and map them to URLs of your choice.
  • You can use customer-managed TLS certificates.
  • You can enable data residency in-transit by connecting to regional endpoints for Google APIs from workloads in that same region.

These names and IP addresses are internal to your VPC network and any on-premises networks that are connected to it using Cloud VPN tunnels or Cloud Interconnect attachments (VLANs).

What you'll learn

  • Create a Private Service Connect endpoint with Consumer HTTP(S) Service Controls
  • Create a Cloud Key Management Service (KMS) key ring and key.
  • Creating a Cloud DNS Response Policy Zone.
  • Access the KMS API (both regional and global) resolved against the public API.
  • Access the KMS API (both regional and global) against the PSC endpoint.

What you'll need

  • Knowledge of deploying instances and configuring networking components

2. Test Environment

A consumer VPC will be created with one subnet in the us-central1 region for hosting a VM and the HTTP(S) Internal Load Balancer forwarding rule and one proxy only subnet for use with the HTTP(S) Internal Load Balancer. We will start by creating a Key Ring and key in the Key Management System (KMS) and resolving to the public API endpoint. We will then create the PSC endpoint to resolve to the regional KMS endpoint in us-central1.

64f5514e7f83714f.png

3. Setup and Requirements

Self-paced environment setup

  1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

96a9c957bc475304.png

b9a10ebdf5b5a448.png

a1e3c01a38fa61c2.png

  • The Project name is the display name for this project's participants. It is a character string not used by Google APIs, and you can update it at any time.
  • The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" after the project is created.
  • There is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.
  1. Next, you'll need to enable billing in the Cloud Console in order to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, follow any "clean-up" instructions found at the end of the codelab. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

bce75f34b2c53987.png

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

f6ef2b5f13479f3a.png

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

4. Before you begin

Enable APIs

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
export project=YOUR-PROJECT-NAME
export region=us-central1
export zone=us-central1-a
echo $project
echo $region
echo $zone

Enable all necessary services

gcloud services enable compute.googleapis.com
gcloud services enable servicedirectory.googleapis.com
gcloud services enable dns.googleapis.com
gcloud services enable cloudkms.googleapis.com

5. Create VPC network, Subnets, and Firewall Rules

VPC Network

From Cloud Shell

gcloud compute networks create consumer-vpc --subnet-mode custom

Create Subnets

From Cloud Shell

gcloud compute networks subnets create consumer-subnet-1 \
--network consumer-vpc \
--range 10.0.0.0/24 \
--region $region \
--enable-private-ip-google-access

For this lab, you will be creating an internal L7 regional load balancer to point to regional API backends. The Google internal L7 load balancer is a proxy load balancer, so therefore you will need to create a proxy subnet that is dedicated to the load balancer to perform the proxy. More information about the proxy-only subnet can be found here.

From Cloud Shell

gcloud compute networks subnets create proxy-subnet1 \
--purpose=INTERNAL_HTTPS_LOAD_BALANCER \
--role=ACTIVE \
--network consumer-vpc \
--range 10.100.100.0/24 \
--region $region

Create Firewall Rules

For this lab, you will be using IAP to connect to the instances you create. The following firewall rule will enable you to connect to instances through IAP. If you prefer not to use IAP, you can skip this step, and instead add public IP addresses on the instance and create a firewall rule that allows ingress on TCP port 22 from 0.0.0.0/0.

To allow IAP to connect to your VM instances, create a firewall rule that:

  • Applies to all VM instances that you want to be accessible by using IAP.
  • Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.

From Cloud Shell

gcloud compute firewall-rules create allow-ssh-iap \
    --network consumer-vpc \
--allow tcp:22 \
--source-ranges=35.235.240.0/20

Create Cloud NAT instance

A Cloud NAT must be created to download Linux package distributions.

Create Cloud Router

From Cloud Shell

gcloud compute routers create crnat \
    --network consumer-vpc \
    --region $region

Create Cloud NAT

From Cloud Shell

gcloud compute routers nats create central-nat \
    --router=crnat \
    --auto-allocate-nat-external-ips \
    --nat-all-subnet-ip-ranges \
    --enable-logging \
    --region $region

6. Create Key Management Key Ring and Key

From Cloud Shell

gcloud kms keyrings create central-keyring \
    --location $region

gcloud kms keyrings create global-keyring \
    --location global

From Cloud Shell

gcloud kms keys create central-key \
    --location $region \
    --keyring central-keyring \
    --purpose encryption

gcloud kms keys create global-key \
    --location global \
    --keyring global-keyring \
    --purpose encryption

From Cloud Shell, confirm the key ring and key were created successfully in the us-central1 region.

gcloud kms keys list \
    --location $region \
    --keyring central-keyring

EXPECTED RESULT

NAME: projects/<PROJECT_ID>/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key
PURPOSE: ENCRYPT_DECRYPT
ALGORITHM: GOOGLE_SYMMETRIC_ENCRYPTION
PROTECTION_LEVEL: SOFTWARE
LABELS:
PRIMARY_ID: 1
PRIMARY_STATE: ENABLED

From Cloud Shell

gcloud kms keys list \
    --location global \
    --keyring global-keyring

EXPECTED RESULT

NAME: projects/<PROJECT_ID>/locations/global/keyRings/global-keyring/cryptoKeys/global-key
PURPOSE: ENCRYPT_DECRYPT
ALGORITHM: GOOGLE_SYMMETRIC_ENCRYPTION
PROTECTION_LEVEL: SOFTWARE
LABELS:
PRIMARY_ID: 1
PRIMARY_STATE: ENABLED

Note the full path names given for the keys as you will be using this to connect to later.

7. Create Client VM and Connect to KMS regional endpoint

Next you'll create a client VM, SSH into the VM, and test the resolution of the us-central1 regional KMS API endpoint.

From Cloud Shell

gcloud compute instances create client-vm \
    --network=consumer-vpc \
    --subnet=consumer-subnet-1 \
    --zone=$zone \
    --no-address \
    --scopes=https://www.googleapis.com/auth/cloud-platform \
    --image-family=debian-10 \
    --image-project=debian-cloud \
    --metadata startup-script='#! /bin/bash
    sudo apt-get update
    sudo apt-get install dnsutils -y
    sudo apt-get install tcpdump -y'

Next, you will need to update the default Compute service account to gain access to the KMS key you created. The default compute service account will be in the format <Project_Number> -compute@developer.gserviceaccount.com. To get the Project Number run the following command from the Cloud Shell and copy the number on the last line of the return results.

 gcloud projects describe $project

Update the default Compute service account to gain access to the KMS key you created.

From Cloud Shell

 gcloud kms keys add-iam-policy-binding central-key \
    --location $region \
    --keyring central-keyring \
    --member serviceAccount:<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
    --role roles/cloudkms.admin


gcloud kms keys add-iam-policy-binding global-key \
    --location global \
    --keyring global-keyring \
    --member serviceAccount:<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
    --role roles/cloudkms.admin

Create an additional cloud shell terminal by clicking + (screenshot below)

476f6ab92768752b.png

In tab 2, tunnel through IAP to SSH into client-vm. Note that environment variables will not carry and you'll need to add your project-id to the command below.

From Cloud Shell

gcloud beta compute ssh --zone us-central1-a "client-vm" \
--tunnel-through-iap \
--project <PROJECT_ID>

Connect to the KMS Regional API endpoint using the KMS key name you noted earlier.

From tab 2, client-vm

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
--resolve cloudkms.googleapis.com:443:us-central1-cloudkms.googleapis.com \
https://cloudkms.googleapis.com/v1/projects/<YOUR_PROJECT_ID>/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key

EXPECTED RESULT

{
  "name": "projects/<PROJECT_ID>/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key",
  "primary": {
    "name": "projects/<PROJECT_ID>/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key/cryptoKeyVersions/1",
    "state": "ENABLED",
    "createTime": "2021-11-12T17:41:21.543348620Z",
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION",
    "generateTime": "2021-11-12T17:41:21.543348620Z"
  },
  "purpose": "ENCRYPT_DECRYPT",
  "createTime": "2021-11-12T17:41:21.543348620Z",
  "versionTemplate": {
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION"
  },
  "destroyScheduledDuration": "86400s"
}

From tab 2, client-vm

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
--resolve cloudkms.googleapis.com:443:cloudkms.googleapis.com \
https://cloudkms.googleapis.com/v1/projects/<YOUR_PROJECT_ID>/locations/global/keyRings/global-keyring/cryptoKeys/global-key

EXPECTED RESULT

{
  "name": "projects/<PROJECT_ID>/locations/global/keyRings/global-keyring/cryptoKeys/global-key",
  "primary": {
    "name": "projects/<PROJECT_ID>/locations/global/keyRings/global-keyring/cryptoKeys/global-key/cryptoKeyVersions/1",
    "state": "ENABLED",
    "createTime": "2021-11-22T19:19:43.271238847Z",
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION",
    "generateTime": "2021-11-22T19:19:43.271238847Z"
  },  "purpose": "ENCRYPT_DECRYPT",
  "createTime": "2021-11-22T19:19:43.271238847Z",
  "versionTemplate": {
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION"
  },
  "destroyScheduledDuration": "86400s"}

Check to see where DNS resolves the us-central KMS endpoint compared to the global endpoint.

From tab2, client-vm

dig us-central1-cloudkms.googleapis.com

EXPECTED RESULT

; <<>> DiG 9.11.5-P4-5.1+deb10u6-Debian <<>> us-central1-cloudkms.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4383
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;us-central1-cloudkms.googleapis.com. IN        A

;; ANSWER SECTION:
us-central1-cloudkms.googleapis.com. 300 IN A   142.250.125.95

;; Query time: 4 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Fri Nov 12 20:40:05 UTC 2021
;; MSG SIZE  rcvd: 80

From tab 2, client-vm

dig cloudkms.googleapis.com

EXPECTED RESULT

; <<>> DiG 9.11.5-P4-5.1+deb10u6-Debian <<>> cloudkms.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49528
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cloudkms.googleapis.com.       IN      A

;; ANSWER SECTION:
cloudkms.googleapis.com. 300    IN      A       209.85.200.95

;; Query time: 8 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Mon Nov 22 15:26:17 UTC 2021
;; MSG SIZE  rcvd: 68

From the results of the dig, we can see that the DNS resolution of the us-central1 regional KMS endpoint and the Global KMS endpoint resolves to an external IP address. (Note: The IP address you see may be a different external IP address. This is normal Google APIs behavior.)

For the next section, switch back to the first tab in Cloud Shell.

8. Create Private Service Connect endpoint

You will create an Internal HTTP(S) Load Balancer with a network endpoint group pointing to the us-central1 regional KMS endpoint as a backend service. The forwarding rule of the load balancer acts as the Private Service Connect endpoint.

Create the Network Endpoint Group (NEG) with type Privet Service Connect and target service us-central1-cloudkms.googleapis.com.

From Cloud Shell

gcloud beta compute network-endpoint-groups create l7psc-kms-neg \
  --network-endpoint-type=private-service-connect \
  --psc-target-service=us-central1-cloudkms.googleapis.com \
  --region=$region

Create the backend service for the load balancer.

From Cloud Shell

gcloud compute backend-services create l7-psc-kms \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --protocol=HTTP \
  --region=$region

Add the NEG to the backend service.

From Cloud Shell

gcloud beta compute backend-services add-backend l7-psc-kms \
  --network-endpoint-group=l7psc-kms-neg \
  --region=$region

Create the URL map for the load balancer.

From Cloud Shell

gcloud compute url-maps create l7-psc-url-map \
  --default-service=l7-psc-kms \
  --region=$region

Create the target proxies for the load balancer.

From Cloud Shell

gcloud compute target-http-proxies create psc-http-proxy \
    --url-map=l7-psc-url-map \
    --region=$region

Create the forwarding rule for the load balancer which will act as the Private Service Connect Endpoint.

From Cloud Shell

gcloud beta compute forwarding-rules create l7-psc-forwarding-rule \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=consumer-vpc \
  --subnet=consumer-subnet-1 \
  --address=10.0.0.100 \
  --ports=80 \
  --region=$region \
  --target-http-proxy=psc-http-proxy \
  --target-http-proxy-region=$region

9. DNS Configuration

In this section you will create a response policy zone (RPZ) to change the DNS resolution of cloudkms.googleapis.com to the PSC endpoint you created.

From Cloud Shell

gcloud beta dns response-policies create kms-l7psc \
--networks=consumer-vpc \
--description="Cloud KMS PSC RPZ"

gcloud beta dns response-policies rules create googleapis-com \
--response-policy=kms-l7psc \
--dns-name=cloudkms.googleapis.com. \
--local-data=name="cloudkms.googleapis.com.",type="A",ttl=300,rrdatas="10.0.0.100"

10. Connect to Regional KMS Endpoint through PSC

Switch back to client-vm in tab 2 to run tcpdump to see all connection details and test the connections to both the regional and global endpoints through the PSC endpoint.

sudo tcpdump -i any net 10.0.0.100 or port 53 -n

Open a 3rd tab in Cloud Shell and SSH into client-vm.

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
http://cloudkms.googleapis.com/v1/projects/<YOUR_PROJECT_ID>/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key

EXPECTED RESULT + TCPDUMP

{
  "name": "projects/psc-2-321213/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key",
  "primary": {
    "name": "projects/psc-2-321213/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key/cryptoKeyVersions/1",
    "state": "ENABLED",
    "createTime": "2021-11-12T17:41:21.543348620Z",
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION",
    "generateTime": "2021-11-12T17:41:21.543348620Z"
  },
  "purpose": "ENCRYPT_DECRYPT",
  "createTime": "2021-11-12T17:41:21.543348620Z",
  "versionTemplate": {
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION"
  },
  "destroyScheduledDuration": "86400s"
}

---

19:54:42.924923 IP 10.0.0.3.55290 > 169.254.169.254.53: 26132+ A? cloudkms.googleapis.com. (41)
19:54:42.925137 IP 10.0.0.3.55290 > 169.254.169.254.53: 40734+ AAAA? cloudkms.googleapis.com. (41)
19:54:42.931480 IP 169.254.169.254.53 > 10.0.0.3.55290: 26132 1/0/0 A 10.0.0.100 (57)
19:54:42.931781 IP 169.254.169.254.53 > 10.0.0.3.55290: 40734 1/0/0 AAAA 2607:f8b0:4001:c0d::5f (69)
19:54:42.932285 IP 10.0.0.3.57136 > 10.0.0.100.80: Flags [S], seq 3382081170, win 65320, options [mss 1420,sackOK,TS val 1907763603 ecr 0,nop,wscale 7], length 0
19:54:42.934951 IP 10.0.0.100.80 > 10.0.0.3.57136: Flags [S.], seq 3147938658, ack 3382081171, win 65535, options [mss 1420,sackOK,TS val 3206048204 ecr 1907763603,nop,wscale 8], length 0
19:54:42.934978 IP 10.0.0.3.57136 > 10.0.0.100.80: Flags [.], ack 1, win 511, options [nop,nop,TS val 1907763606 ecr 3206048204], length 0
19:54:42.935182 IP 10.0.0.3.57136 > 10.0.0.100.80: Flags [P.], seq 1:422, ack 1, win 511, options [nop,nop,TS val 1907763606 ecr 3206048204], length 421: HTTP: GET /v1/projects/<PROJECT_ID/locations/us-central1/keyRings/central-keyring/cryptoKeys/central-key HTTP/1.1
19:54:42.935614 IP 10.0.0.100.80 > 10.0.0.3.57136: Flags [.], ack 422, win 261, options [nop,nop,TS val 3206048205 ecr 1907763606], length 0
19:54:43.010268 IP 10.0.0.100.80 > 10.0.0.3.57136: Flags [P.], seq 1:1072, ack 422, win 261, options [nop,nop,TS val 3206048280 ecr 1907763606], length 1071: HTTP: HTTP/1.1 200 OK
19:54:43.010295 IP 10.0.0.3.57136 > 10.0.0.100.80: Flags [.], ack 1072, win 503, options [nop,nop,TS val 1907763681 ecr 3206048280], length 0
19:54:43.011545 IP 10.0.0.3.57136 > 10.0.0.100.80: Flags [F.], seq 422, ack 1072, win 503, options [nop,nop,TS val 1907763683 ecr 3206048280], length 0
19:54:43.012013 IP 10.0.0.100.80 > 10.0.0.3.57136: Flags [F.], seq 1072, ack 423, win 261, options [nop,nop,TS val 3206048282 ecr 1907763683], length 0
19:54:43.012021 IP 10.0.0.3.57136 > 10.0.0.100.80: Flags [.], ack 1073, win 503, options [nop,nop,TS val 1907763683 ecr 3206048282], length 0

In tab 3, client-vm

dig cloudkms.googleapis.com

EXPECTED RESULT

; <<>> DiG 9.11.5-P4-5.1+deb10u6-Debian <<>> cloudkms.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42396
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cloudkms.googleapis.com.       IN      A

;; ANSWER SECTION:
cloudkms.googleapis.com. 300    IN      A       10.0.0.100

;; Query time: 7 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 16 18:53:51 UTC 2021
;; MSG SIZE  rcvd: 68

Check back on the tab 2 window and inspect the tcpdump information. You'll see that you were able to access the Cloud KMS regional endpoint through the PSC endpoint that you created and that the us-central1 regional endpoint resolves privately to the Managed Cloud DNS zone you created.

From tab 3, client-vm

curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \
http://cloudkms.googleapis.com/v1/projects/<YOUR_PROJECT_ID>/locations/global/keyRings/global-keyring/cryptoKeys/global-key

EXPECTED RESULT + TCPDUMP

{
  "error": {
    "code": 404,
    "message": "The request concerns location 'global' but was sent to location 'us-central1'. Either Cloud KMS is not available in 'global' or the request was misrouted. gRPC clients must ensure that 'x-goog-request-params' is specified in request metadata. See https://cloud.google.com/kms/docs/grpc for more information.",
    "status": "NOT_FOUND"
  }
}

---

20:04:32.199247 IP 10.0.0.3.57162 > 10.0.0.100.80: Flags [S], seq 2782317346, win 65320, options [mss 1420,sackOK,TS val 1908352831 ecr 0,nop,wscale 7], length 0
20:04:32.201643 IP 10.0.0.100.80 > 10.0.0.3.57162: Flags [S.], seq 1927472124, ack 2782317347, win 65535, options [mss 1420,sackOK,TS val 3731555085 ecr 1908352831,nop,wscale 8], length 0
20:04:32.201666 IP 10.0.0.3.57162 > 10.0.0.100.80: Flags [.], ack 1, win 511, options [nop,nop,TS val 1908352834 ecr 3731555085], length 0
20:04:32.201812 IP 10.0.0.3.57162 > 10.0.0.100.80: Flags [P.], seq 1:415, ack 1, win 511, options [nop,nop,TS val 1908352834 ecr 3731555085], length 414: HTTP: GET /v1/projects/<YOUR_PROJECT_ID>/locations/global/keyRings/global-keyring/cryptoKeys/global-key HTTP/1.1
20:04:32.202308 IP 10.0.0.100.80 > 10.0.0.3.57162: Flags [.], ack 415, win 261, options [nop,nop,TS val 3731555086 ecr 1908352834], length 0
20:04:32.237629 IP 10.0.0.100.80 > 10.0.0.3.57162: Flags [P.], seq 1:760, ack 415, win 261, options [nop,nop,TS val 3731555121 ecr 1908352834], length 759: HTTP: HTTP/1.1 404 Not Found
20:04:32.237656 IP 10.0.0.3.57162 > 10.0.0.100.80: Flags [.], ack 760, win 506, options [nop,nop,TS val 1908352870 ecr 3731555121], length 0
20:04:32.238283 IP 10.0.0.3.57162 > 10.0.0.100.80: Flags [F.], seq 415, ack 760, win 506, options [nop,nop,TS val 1908352870 ecr 3731555121], length 0
20:04:32.238833 IP 10.0.0.100.80 > 10.0.0.3.57162: Flags [F.], seq 760, ack 416, win 261, options [nop,nop,TS val 3731555122 ecr 1908352870], length 0
20:04:32.238851 IP 10.0.0.3.57162 > 10.0.0.100.80: Flags [.], ack 761, win 506, options [nop,nop,TS val 1908352871 ecr 3731555122], length 0

In tab 3, client-vm

dig cloudkms.googleapis.com

EXPECTED RESULT

; <<>> DiG 9.11.5-P4-5.1+deb10u6-Debian <<>> cloudkms.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55205
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cloudkms.googleapis.com.       IN      A

;; ANSWER SECTION:
cloudkms.googleapis.com. 300    IN      A       10.0.0.100

;; Query time: 6 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Thu Dec 16 18:58:06 UTC 2021
;; MSG SIZE  rcvd: 68

Trying to access a global Cloud KMS Key/Key Ring will result in an error. You will see these connection attempts in the tcpdump as well. You'll see the dig also has the same result because all traffic is now directed to the PSC endpoint.

You can now close both SSH tabs to client-vm.

11. Cleanup steps

From a single Cloud Shell terminal delete lab components

gcloud beta dns response-policies rules delete googleapis-com \
--response-policy=kms-l7psc -q

gcloud beta dns response-policies update kms-l7psc --networks=""

gcloud beta dns response-policies delete kms-l7psc -q

gcloud compute forwarding-rules delete l7-psc-forwarding-rule --region=$region --quiet

gcloud compute target-http-proxies delete psc-http-proxy --region=$region --quiet

gcloud compute url-maps delete l7-psc-url-map --region=$region --quiet

gcloud compute backend-services delete l7-psc-kms --region=$region --quiet

gcloud compute network-endpoint-groups delete l7psc-kms-neg --region=$region --quiet

gcloud compute instances delete client-vm --zone=$zone --quiet

#Replace PROJECT_NUMBER
gcloud kms keys remove-iam-policy-binding global-key --keyring=global-keyring --location=global \
--member=serviceAccount:<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
--role roles/cloudkms.admin

#Replace PROJECT_NUMBER
gcloud kms keys remove-iam-policy-binding central-key --keyring=central-keyring --location=$region \
--member=serviceAccount:<PROJECT_NUMBER>-compute@developer.gserviceaccount.com \
--role roles/cloudkms.admin

gcloud kms keys versions destroy 1 --location=global --keyring=global-keyring --key=global-key

gcloud kms keys versions destroy 1 --location=$region --keyring=central-keyring --key=central-key

gcloud compute routers nats delete central-nat --router=crnat --region=$region --quiet

gcloud compute routers delete crnat --region=$region --quiet

gcloud compute firewall-rules delete allow-ssh-iap --quiet

gcloud compute networks subnets delete proxy-subnet --region=$region --quiet

gcloud compute networks subnets delete consumer-subnet-1 --region=$region --quiet

gcloud compute networks delete consumer-vpc --quiet

12. Congratulations!

Congratulations for completing the codelab.

What we've covered

  • Creating a Private Service Connect endpoint with Consumer HTTP(S) Service Controls
  • Creating a Cloud Key Management Service (KMS) key ring and key.
  • Creating a Cloud DNS Response Policy Zone.
  • Accessing the KMS API (both regional and global) resolved against the public API.
  • Accessing the KMS API (both regional and global) against the PSC endpoint.