Private Service Connect for Google APIs

1. Introduction

With Private Service Connect, you can create private endpoints using global internal IP addresses within your VPC network to access google APIs. You can assign DNS names to these internal IP addresses with meaningful names like storage-pscendpoint.p.googleapis.com and bigtable-adsteam.p.googleapis.com. Instead of sending API requests to public service endpoints such as storage.googleapis.com, you can send the requests to the Private Service Connect endpoint, which is private and internal to your VPC network.

These names and IP addresses are internal to your VPC network and any on-premises networks that are connected to it using Cloud VPN tunnels or Cloud Interconnect attachments (VLANs).

You can control which traffic goes to which endpoint, and can demonstrate that traffic stays within Google Cloud.

What you'll learn

• Private service connect use cases
• Network requirements
• Supported APIs
• Create a Private Service Connect endpoint
• Create a cloud storage bucket
• Create and update Cloud DNS private zones
• Create a NAT GW to access public googleapis
• Create and update a BOTO configuration file
• Perform gsutil list on VM1 resolved against your PSC service endpoint
• Perform gsutil list on VM2 resolved against public googleapis.com
• Use Tcpdump to validate DNS resolution

What you'll need

• Knowledge of DNS, nano or vi editor

2. Private service connect use cases

You can create multiple Private Service Connect endpoints in the same VPC network. There is no limit on bandwidth to a particular endpoint. Because Private Service Connect endpoints use global internal IP addresses, they can be used by any resource in your VPC network.

With multiple endpoints, you can specify different network paths using Cloud Router and firewall rules.

• You can create firewall rules to prevent some VMs from accessing Google APIs through a Private Service Connect endpoint, while allowing other VMs to have access.
• You can have a firewall rule on a VM instance that disallows all traffic to the internet; traffic sent to Private Service Connect endpoints still reaches Google.
• If you have on-premises hosts that are connected to a VPC using a Cloud VPN tunnel or a Cloud Interconnect attachment (VLAN), you can send some requests through the tunnel or VLAN while sending other requests over the public internet. This configuration lets you bypass the tunnel or VLAN for services such as Google Books that are not supported by Private Google Access. To create this configuration, create a Private Service Connect endpoint, advertise the Private Service Connect endpoint IP addresses using Cloud Router custom route advertisements, and enable a Cloud DNS inbound forwarding policy. The application can send some requests through the Cloud VPN tunnel or Cloud Interconnect attachment (VLAN) by using the name of the Private Service Connect endpoint, and others over the internet by using the default DNS name.
• If you connect your on-premises network to your VPC network using multiple Cloud Interconnect attachments (VLANs), you can send some traffic from on-premises over one VLAN and the rest over others, as shown in figure 2. This lets you use your own wide-area networking instead of Google's, and to control data movement to meet geographic requirements. To create this configuration, create two Private Service Connect endpoints. Create a custom route advertisement for the first endpoint on the BGP session of the Cloud Router managing the first VLAN, and create a different custom route advertisement for the second endpoint on the BGP session of the Cloud Router managing the second VLAN. On-premises hosts that are configured to use the Private Service Connect endpoint name send traffic over the corresponding Cloud Interconnect attachment (VLAN).
• You can also use multiple Cloud Interconnect attachments (VLANs) in an active/active topology. If you advertise the same Private Service Connect endpoint IP address using custom route advertisements for the BGP sessions on the Cloud Routers managing the VLANs, packets sent from on-premises systems to the endpoints are routed across the VLANs using ECMP.

Figure 1. By configuring Private Service Connect, Cloud Router, and on-premises hosts, you can control which Cloud Interconnect attachment (VLAN) is used to send traffic to Google APIs.

3. Network requirements

To use Private Service Connect, virtual machine (VM) instances without external IP addresses must have their primary interface in a subnet with Private Google Access enabled.

A VM with an external IP address can access Google APIs and services using Private Service Connect endpoints whether or not Private Google Access is enabled for its subnet. Connectivity to the Private Service Connect endpoint stays within Google's network.

Private Service Connect endpoints are not accessible from peered VPC networks.

Supported APIs

When you create a Private Service Connect endpoint, you choose which bundle of APIs you need access to: all-apis or vpc-sc.

The API bundles give access to the same APIs that are available through the Private Google Access VIPs.

• The all-apis bundle provides access to the same APIs as private.googleapis.com.
• The vpc-sc bundle provides access to the same APIs as restricted.googleapis.com.

4. Codelab topology & use case

Figure1 - Codelab topology

Codelab use case -

Our customer requires a mix of private (interconnect) and public googleapis access for cloud storage data transfer. To meet our customers requirements, we will deploy Private Service Connect consisting of a unique /32 address, BOTO config and DNS record updates. Virtual machine 1 will utilize PSC for cloud storage bucket access; in contrast, VM2 will use public googleapis.com IP ranges via the NAT GW.

All aspects of the lab are deployed within Google Cloud Platform, yet the same use case is applicable for Hybrid Cloud deployment requiring traffic separation.

5. Setup and Requirements

Self-paced environment setup

1. Sign in to Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

Note: You can easily access Cloud Console by memorizing its URL, which is console.cloud.google.com.

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost much, if anything at all. Be sure to to follow any instructions in the "Cleaning up" section which advises you how to shut down resources so you don't incur billing beyond this tutorial. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

6. Before you begin

Enable APIs

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname

Enable all necessary services

gcloud services enable compute.googleapis.com
gcloud services enable servicedirectory.googleapis.com
gcloud services enable dns.googleapis.com

7. Create VPC network

VPC Network

From Cloud Shell

gcloud compute networks create psc-lab --subnet-mode custom

Create Subnet

From Cloud Shell

gcloud compute networks subnets create psclab-subnet \
--network psc-lab --range 10.0.0.0/24 --region us-central1 --enable-private-ip-google-access

Create Firewall Rules

To allow IAP to connect to your VM instances, create a firewall rule that:

• Applies to all VM instances that you want to be accessible by using IAP.
• Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.

From Cloud Shell

gcloud compute firewall-rules create psclab-ssh \
    --network psc-lab --allow tcp:22 --source-ranges=35.235.240.0/20

Create Cloud NAT instance

Create Cloud Router

From Cloud Shell

gcloud compute routers create crnat \
    --network psc-lab \
    --asn 65000 \
    --region us-central1

Create Cloud NAT

From Cloud Shell

gcloud compute routers nats create cloudnat \
    --router=crnat \
    --auto-allocate-nat-external-ips \
    --nat-all-subnet-ip-ranges \
    --enable-logging \
    --region us-central1

8. Create Private Service Connect endpoint

When you configure the Private Service Connect endpoint IP <pscendpointip>, you will need to provide a unique IP address not defined in your VPC.

Caution: Ensure to review PSC IP Address requirements and update the field "addresses".

From Cloud Shell

gcloud compute addresses create psc-ip \
    --global \
    --purpose=PRIVATE_SERVICE_CONNECT \
    --addresses=192.168.255.250 \
    --network=psc-lab

Store ‘pscendpointip' for duration of the lab

pscendpointip=$(gcloud compute addresses list --filter=name:psc-ip --format="value(address)")

echo $pscendpointip

Create a forwarding rule to connect the endpoint to Google APIs and services.

From Cloud Shell

gcloud compute forwarding-rules create pscendpoint \
    --global \
    --network=psc-lab \
    --address=psc-ip \
    --target-google-apis-bundle=all-apis

List the configured Private Service Connect endpoints

From Cloud Shell

gcloud compute forwarding-rules list  \
--filter target="(all-apis OR vpc-sc)" --global

Describe the configured Private Service Connect endpoints

From Cloud Shell

gcloud compute forwarding-rules describe \
    pscendpoint --global

9. Create a bucket

Caution: In order to create a bucket in a project, a user must have the storage.buckets.create project permission for the project. If you are working within a project that you did not create, you might need the project owner to give you a role that contains this permission, such as Storage Admin.

Create a Cloud Storage bucket and replace BUCKET_NAME with a globally unique name you prefer.

Note: Try another bucket name if already in use

From Cloud Shell

gsutil mb  -l us-central1 -b on gs://BUCKET_NAME

Store ‘BUCKET_NAME' for duration of the lab

BUCKET_NAME=YOUR BUCKET NAME
echo $BUCKET_NAME

10. DNS Configuration

When you create a Private Service Connect endpoint, Service Directory generates a DNS record for the APIs and services that are made available using that endpoint.

The DNS records point to your Private Service Connect endpoint IP address, and are in this format: SERVICE-ENDPOINT.p.googleapis.com.

You can use these DNS names in your API requests to direct the request to your Private Service Connect endpoint. You can also use these DNS names in the Host header of your request.

If you want to use a Private Service Connect endpoint with a client or application that accesses Google APIs and services, update your client or application to use the p.googleapis.com DNS names.

See the documentation for your client or client library for details. For example:

• Python: You can configure api_endpoint in the Client options class in the google-api-core package.
• Go: You can configure WithEndpoint in the Client options package in the api package.
• gcloud: You can configure api_endpoint_overrides using this command. gcloud config set api_endpoint_overrides/SERVICE ENDPOINT_URL

For example: gcloud config set api_endpoint_overrides/storage https://storage-xyz.p.googleapis.com/storage/v1/

If you can't configure your client or application to use a different endpoint, create DNS records to match the default DNS names. For example, storage.googleapis.com. See Creating DNS records using default DNS names.

Validate DNS record

From the cloud console validate generated dns entry located under Network Services → Cloud DNS. Note the generated DNS name ‘p.googleapis.com'.

11. Create Virtual Machines

Create Virtual Machine (psc-instance-1) used to validate Private Service Connect

From Cloud Shell

  gcloud compute instances create psc-instance-1 \
    --subnet psclab-subnet \
    --zone us-central1-a \
    --image=centos-7-v20210122 \
    --image-project=centos-cloud \
    --no-address \
    --metadata=startup-script=yum\ install\ tcpdump\ -y$'\n'yum\ install\ bind-utils\ -y$'\n'yum\ install\ nano\ -y 

Log into the VM instance (psc-instance-1)

SSH into the VM through Cloud Shell

gcloud compute ssh --zone "us-central1-a" "psc-instance-1" --project "$projectname"

Create additional cloud shell terminals by clicking + (screenshot below) three times.

Create Virtual Machine (psc-instance-2) used to validate public Googleapis

From tab 2

  gcloud compute instances create psc-instance-2 \
    --subnet psclab-subnet \
    --zone us-central1-a \
    --image=centos-7-v20210122 \
    --image-project=centos-cloud \
    --no-address \
    --metadata=startup-script=yum\ install\ tcpdump\ -y$'\n'yum\ install\ bind-utils\ -y$'\n'yum\ install\ nano\ -y 

From tab 2 SSH into the VM through Cloud Shell

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname


gcloud compute ssh --zone "us-central1-a" "psc-instance-2" --project "$projectname"

From tab 3 SSH into psc-instance-1 through Cloud Shell

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname


gcloud compute ssh --zone "us-central1-a" "psc-instance-1" --project "$projectname"

From tab 4 Shell SSH into psc-instance-2 through Cloud Shell

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname


gcloud compute ssh --zone "us-central1-a" "psc-instance-2" --project "$projectname"

12. Verify existing Gsutil behavior

From tab 4 (psc-instance-2) start tcpdump and monitor DNS traffic

sudo tcpdump -vv -i eth0 port 53

Inspect storage bucket DNS lookup from tab 2 (psc-instance-2)

BUCKET_NAME=YOUR BUCKET NAME
echo $BUCKET_NAME
gsutil -D ls gs://$BUCKET_NAME

Inspect the gsutil debug, HOST storage.googleapis.com is used for DNS resolution

<snip>
send: 'GET /storage/v1/b/$BUCKET_NAME/o?delimiter=%2F&projection=noAcl&versions=False&fields=prefixes%2CnextPageToken%2Citems%2Fname&alt=json&maxResults=1000 HTTP/1.1\r\nHost: storage.googleapis.com\r\ncontent-length: 0\r\nauthorization: Bearer ya29.c.KpkB7wfaMjfc_WXEKCeNF4Md0fEHnfDU7tqBf3cd0u43yEmYXqj8fX_X5wWdNdDVH6k1EkjeAeIJDzKGvyjPOkf1Io2kVeUqYX69sDv53huW1NslffjAHKchbZ0CP3Cg83TS3Pa55jLcuE0TLbYycVrgSbD3H90LaapUGbWD3kj4IsJLf9J8R98Bqobu8HZwwqk92hlZ4zVzRqOM\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: apitools Python/2.7.5 gsutil/4.57 (linux2) analytics/disabled interactive/True command/ls google-cloud-sdk/324.0.0\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
<snip>

From tab 4 (psc-instance-2) verify GoogleAPI.com public DNS A record(s) are used when accessing the storage bucket.

metadata.google.internal.domain > psc-instance-2.c.yourprojectname.internal.33973: [udp sum ok] 36442 q: A? storage.googleapis.com. 11/0/0 storage.googleapis.com. A 108.177.111.128, storage.googleapis.com. A 142.250.128.128, storage.googleapis.com. A 74.125.70.128, storage.googleapis.com. A 74.125.201.128, storage.googleapis.com. A 64.233.183.128, storage.googleapis.com. A 173.194.198.128, storage.googleapis.com. A 172.217.219.128, storage.googleapis.com. A 142.250.136.128, storage.googleapis.com. A 209.85.234.128, storage.googleapis.com. A 172.217.212.128, storage.googleapis.com. A 172.217.214.128

13. Modify Gsutil behavior

Recall in a previous step, you created a Private DNS Zone and A record mapped to the PSC endpoint IP address. In the following step, we will now control how gsutil behaves by updating the VM BOTO file on psc-instance-1.

View the default BOTO configuration from the VM instance terminal from tab 1 (psc-instance-1)

[psc-instance ~]$ more  /etc/boto.cfg

Output (Your project_id will differ)

[GSUtil]
default_project_id  = [your project number]
default_api_version = 2

[GoogleCompute]
service_account = default

Update BOTO configuration using nano or VI Editor, ensure to copy and paste all entries.

Example: sudo nano /etc/boto.cfg

or

Example: sudo vi /etc/boto.cfg

From the VM instance terminal tab 1(psc-instance-1)

[Credentials]
gs_host = storage-pscendpoint.p.googleapis.com
gs_host_header = storage.googleapis.com
gs_json_host = storage-pscendpoint.p.googleapis.com
gs_json_host_header = www.googleapis.com

Caution: Close attention is required when updating BOTO, improper syntax leads to failed DNS lookup, BOTO configuration is required above [GSUtil] configuration.

Validate the configuration, order of [Credentials] is critical for DNS lookup

more /etc/boto.cfg
[Credentials]
gs_host = storage-pscendpoint.p.googleapis.com
gs_host_header = storage.googleapis.com
gs_json_host = storage-pscendpoint.p.googleapis.com
gs_json_host_header = www.googleapis.com

[GSUtil]
default_project_id  = [your project number
default_api_version = 2

[GoogleCompute]
service_account = default

14. Verify updated gsutil lookup behavior

From tab 3 (psc-instance-1) start tcpdump and monitor DNS traffic

sudo tcpdump -vv -i eth0 port 53

Inspect storage bucket gsutil lookup from tab 1 (psc-instance-1)

BUCKET_NAME=YOUR BUCKET NAME
echo $BUCKET_NAME

gsutil -D ls gs://$BUCKET_NAME

Debug logs confirm the storage bucket is reachable through Private Service Connect endpoint "pscendpoint"

Output:

<snip>
INFO 0131 22:14:18.795986 base_api.py] Making http GET to https://storage-pscendpoint.p.googleapis.com/storage/v1/b/$BUCKET_NAME/o?delimiter=%2F&projection=noAcl&versions=False&fields=prefixes%2CnextPageToken%2Citems%2Fname&alt=json&maxResults=1000
INFO 0131 22:14:18.796415 base_api.py] Headers: {u'Host': 'www.googleapis.com',
 'accept': 'application/json',
 'accept-encoding': 'gzip, deflate',
 'content-length': '0',
 'user-agent': 'apitools Python/2.7.5 gsutil/4.57 (linux2) analytics/disabled interactive/True command/ls google-cloud-sdk/324.0.0'}
INFO 0131 22:14:18.796502 base_api.py] Body: (none)
connect: (storage-pscendpoint.p.googleapis.com, 443)
send: 'GET /storage/v1/b/psc-bucket/o?delimiter=%2F&projection=noAcl&versions=False&fields=prefixes%2CnextPageToken%2Citems%2Fname&alt=json&maxResults=1000 HTTP/1.1\r\ncontent-length: 0\r\naccept-encoding: gzip, deflate\r\naccept: application/json\r\nuser-agent: apitools Python/2.7.5 gsutil/4.57 (linux2) analytics/disabled interactive/True command/ls google-cloud-sdk/324.0.0\r\nhost: www.googleapis.com\r\nauthorization: Bearer ya29.c.KpkB7wd3XWiYeRyTuth5_HPlNV-hPwc2Nn7RSIeMpzrpa_j4EsMPl2m_mDGKAcGHvYIgiC5bT2UVQirAPpSbbpToa6G6lkaBbH5SZwHwgNXYfisp5Ww1UjXe4rTa69a_Wp0WesafcwPNnYzDo3xf5VGh3iGhySA04kTXuyT--MgOU8U-XLII2LJQxUWlV8KEdrvyCuqRb-jsDdk_\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
<snip>

From Tab 3 (psc-instance-1) verify your PSC endpoint IP is the DNS A record used when trying to access your storage bucket.

@psc-instance-1 ~]$ sudo tcpdump -vv -i eth0 port 53
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
05:02:33.936256 IP (tos 0x0, ttl 64, id 55416, offset 0, flags [DF], proto UDP (17), length 82)
    psc-instance-1.c.yourprojectname.internal.42296 > metadata.google.internal.domain: [bad udp cksum 0x5e4e -> 0xcceb!] 34796+ A? storage-pscendpoint.p.googleapis.com. (54)
05:02:33.936269 IP (tos 0x0, ttl 64, id 55417, offset 0, flags [DF], proto UDP (17), length 82)
    psc-instance-1.c.yourprojectname.internal.42296 > metadata.google.internal.domain: [bad udp cksum 0x5e4e -> 0x3ebd!] 5632+ AAAA? storage-pscendpoint.p.googleapis.com. (54)
05:02:33.944018 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 98)
    metadata.google.internal.domain > psc-instance-1.c.yourprojectname.42296: [udp sum ok] 34796 q: A? storage-pscendpoint.p.googleapis.com. 1/0/0 storage-pscendpoint.p.googleapis.com. A 10.10.110.10 (70)
05:02:33.946005 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 175)

Verify Private Service Connect endpoint IP is now used for DNS resolution

From tab1

nslookup storage-pscendpoint.p.googleapis.com

Output

@psc-instance ~]$ nslookup storage-pscendpoint.p.googleapis.com
Server:         169.254.169.254
Address:        169.254.169.254#53

Non-authoritative answer:
Name:   storage-pscendpoint.p.googleapis.com
Address: <pscip>

15. Cleanup steps

Exit from VM instance (all tabs)

exit

From a single Cloud Shell terminal delete lab components

gcloud compute routers nats delete cloudnat --router=crnat --region=us-central1 --quiet

gcloud compute routers delete crnat --region=us-central1 --quiet

gcloud compute forwarding-rules delete pscendpoint --global --quiet

gcloud compute addresses delete psc-ip --global --quiet

gsutil rm -r gs://$BUCKET_NAME

gcloud compute instances delete psc-instance-1 --zone=us-central1-a --quiet

gcloud compute instances delete psc-instance-2 --zone=us-central1-a --quiet

gcloud compute firewall-rules delete psclab-ssh --quiet

gcloud compute networks subnets delete psclab-subnet --region us-central1 --quiet

gcloud compute networks delete psc-lab --quiet

From Console, ensure you are viewing the correct project, then select Networking Services → Cloud DNS

16. Congratulations!

Congratulations for completing the codelab.

What we've covered

• Private service connect use cases
• Network requirements
• Supported APIs
• Created a Private Service Connect endpoint
• Created a cloud storage bucket
• Updated a BOTO configuration file
• Created a NAT GW
• Perform gsutil list on VM1 that resolves against your PSC service endpoint
• Perform gsutil list on VM2 that resolves against public googleapis.com
• Use Tcpdump to validate DNS resolution