Using Private Service Connect to publish and consume services with GKE

1. Introduction

Private Service Connect lets a service producer offer services privately to a service consumer. Private Service Connect offers the following benefits:

• A service producer VPC network can support more than one service consumer.
• Each consumer connects to an internal IP address that they define. Private Service Connect performs network address translation (NAT) to route the request to the service producer.

Figure 2. Private Service Connect uses endpoints and service attachments to let service consumers send traffic from the consumer's VPC network to services in the service producer's VPC network (click to enlarge).

What you'll learn

• Private service connect benefits
• Key concepts for service consumers
• Key concepts for service producers
• Create a producer environment
• Expose service (producer environment) through a service attachment
• Create a consumer environment
• Create a forwarding rule in the consumer network
• Validate consumer access
• Enable policy access control
• Use a egress firewall rule to block access to a consumer forwarding rule

What you'll need

• Knowledge of deploying GKE Clusters and services
• Knowledge of Internal load balancers
• Ability to create VPCs in two projects
• Ability to create GKE Cluster

2. Private service connect benefits

With PSC, you have several benefits compared to using VPC Peering:

Better control of private IP space

• As a service consumer, you can control the private IP address that is used to connect to the managed service you would like to access.
• As a service consumer, you do not need to worry about reserving private IP address ranges for backend services that are consumed in your VPC. You only need to choose an IP address from your own subnet to connect to the producer services.
• As a service producer, you can choose to deploy a multi-tenant model, where your VPC contains services that serve multiple consumer VPCs. The consumers having overlapping subnet ranges are no longer a problem.
• As a service provider, you can scale your service to as many VM instances as required, without needing to contact your consumer for more IP addresses.

Improved security and isolation

• As a service consumer, only you can initiate communication to the service producer. This uni-directional connectivity drastically simplifies firewall configuration but also reduces risk from rogue traffic coming from the service producer.
• As a service producer, you do not need to change your firewall rules based on the subnet ranges in the consumer's VPC. You can simply create firewall rules for the NAT IP address range configured for your service.

Better Scalability

• PSC enables highly-scalable design by supporting thousands of Consumers, and allows Service Producers to offer highly scalable multi-tenant or single-tenant services.
• As a service consumer using private service connect, you can create resources as required in your VPC. The scale of this is not affected by the number of such resources created in the producer VPC.

3. Key concepts for service consumers

You can use Private Service Connect endpoints to consume services that are outside of your VPC network. Service consumers create Private Service Connect endpoints that connect to a target service.

Endpoints

You use Private Service Connect endpoints to connect to a target service. Endpoints have an internal IP address in your VPC network and are based on the forwarding rule resource.

You send traffic to the endpoint, which forwards it to targets outside of your VPC network.

Targets

Private Service Connect endpoints have a target, which is the service you want to connect to:

• An API bundle:
• All APIs: most Google APIs
• VPC-SC: APIs that VPC Service Controls supports
• A published service in another VPC network. This service can be managed by your own organization or a third party.

Published service

To connect your endpoint to a service producer's service, you need the service attachment for the service. The service attachment URI has this format: projects/SERVICE_PROJECT/regions/REGION/serviceAttachments/SERVICE_NAME

4. Key concepts for service producers

To make a service available to consumers, you create one or more dedicated subnets to use for network address translation (NAT) of consumer IP addresses. You then create a service attachment which refers to those subnets.

Private Service Connect subnets

To expose a service, the service producer first creates one or more subnets with purpose Private Service Connect.

When a request is sent from a consumer VPC network, the consumer's source IP address is translated using source NAT (SNAT) to an IP address selected from one of the Private Service Connect subnets.

If you want to retain the consumer connection IP address information, see Viewing consumer connection information.

These subnets cannot be used for resources such as VM instances or forwarding rules. The subnets are used only to provide IP addresses for SNAT of incoming consumer connections.

The Private Service Connect subnet must contain at least one IP address for every 63 consumer VMs so that each consumer VM is allocated 1,024 source tuples for network address translation.

The minimum size for a Private Service Connect subnet is /24.

Service attachments

Service producers expose their service through a service attachment.

• To expose a service, a service producer creates a service attachment that refers to the service's load balancer forwarding rule.
• To access a service, a service consumer creates an endpoint that refers to the service attachment.

Connection preferences

When you create a service, you choose how to make it available. There are two options:

• Automatically accept connections for all projects - any service consumer can configure an endpoint and connect to the service automatically.
• Accept connections for selected projects - service consumers configure an endpoint to connect to the service and the service producer accepts or rejects the connection requests.

Requirements and limitations

• Limitations for Private Service Connect apply.
• You can create a service attachment in GKE versions 1.21.4-gke.300 and later.
• You cannot use the same subnet in multiple service attachment configurations.
• You must create a GKE service that uses an internal TCP/UDP load balancer.

5. Test Environment

Consumer network consists of a static IP address used to originate requests to the service producer, in addition to the target-service-attachment that maps to the producer's service attachment (published service).

Now, let's take a look at the producers network. Notice how the producers network does not have a mapping to the consumers network, instead, the producer network contains a service attachment (published service) that is used by the consumer for services. The producer's service attachment is exposed by a GKE ingress L4 ILB (published service) enabling communication to the GKE Pods & associated applications.

The NAT subnet is used when a request is sent from a consumer VPC network, the consumer's source IP address is translated using source NAT (SNAT) to an IP address selected from one of the Private Service Connect subnets.

These subnets cannot be used for resources such as VM instances or forwarding rules. The subnets are used only to provide IP addresses for SNAT of incoming consumer connections.

To find out more about L4ILB for GKE Private Service Connect and obtain direct access to content used to make this lab reference the following.

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs, and you can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" after the project is created.
• There is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. If a project is deleted, that ID can never be used again.

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console in order to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, follow any "clean-up" instructions found at the end of the codelab. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

6. Before you begin

Codelab requires two projects, although not a requirement for PSC. Note the references to support single or multiple projects.

Single Project - Update project to support producer and consumer network

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
prodproject=YOUR-PROJECT-NAME
consumerproject=YOUR-PROJECT-NAME
echo $prodproject
echo $consumerproject

Multiple projects - Update project to support producer network

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
prodproject=YOUR-PROJECT-NAME
echo $prodproject

Note the following coloring code convention:

7. Create Producers VPC network

VPC Network

From Cloud Shell

gcloud compute networks create gke-producer-l4-vpc --project=$prodproject --subnet-mode=custom 

Create GKE Cluster subnet

From Cloud Shell

gcloud compute networks subnets create node-subnet1 --project=$prodproject --range=192.168.10.0/24 --network=gke-producer-l4-vpc --region=us-central1 --secondary-range=pod=10.10.10.0/24,service=10.10.20.0/24 --enable-private-ip-google-access

Create GKE Cluster

From Cloud Shell

gcloud container clusters create gke-psc-l4 \
    --release-channel=rapid \
    --enable-ip-alias \
    --zone=us-central1-a \
    --network gke-producer-l4-vpc \
    --num-nodes 1 \
    --subnetwork node-subnet1 \
    --cluster-secondary-range-name pod \
    --services-secondary-range-name service

Create a subnet for Private Service Connect (NAT Subnet)

You must create one or more dedicated subnets for use with Private Service Connect. If you're using the Google Cloud Console to publish a service, you can create the subnets during that procedure.

For information about Private Service Connect subnets, see Private Service Connect subnets.

From Cloud Shell

gcloud beta compute networks subnets create gke-nat-subnet \
    --project $prodproject \
    --network gke-producer-l4-vpc \
    --region us-central1 \
    --range 100.100.10.0/24 \
    --purpose PRIVATE_SERVICE_CONNECT

8. Deploy a workload and services

The following manifest describes a Deployment that runs a sample web application container image. Save the manifest as my-deployment.yaml from cloud shell

apiVersion: apps/v1
kind: Deployment
metadata:
  name: psc-ilb
spec:
  replicas: 3
  selector:
    matchLabels:
      app: psc-ilb
  template:
    metadata:
      labels:
        app: psc-ilb
    spec:
      containers:
      - name: whereami
        image: gcr.io/google-samples/whereami:v1.2.1
        ports:
          - name: http
            containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          timeoutSeconds: 1

Apply the manifest to your cluster from cloud shell

kubectl apply -f my-deployment.yaml

Create a service

The following manifest describes a service that creates an internal TCP/UDP load balancer on TCP port 8080. Save the manifest as my-service.yaml from cloud shell.

apiVersion: v1
kind: Service
metadata:
  name: gke-l4-psc
  annotations:
    networking.gke.io/load-balancer-type: "Internal"
spec:
  type: LoadBalancer
  selector:
    app: psc-ilb
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

Apply the manifest to your cluster from cloud shell

kubectl apply -f my-service.yaml

Create ServiceAttachment

The following manifest describes a ServiceAttachment that exposes the service that you created to service consumers. Save the manifest as my-psc.yaml from cloud shell.

apiVersion: networking.gke.io/v1beta1
kind: ServiceAttachment
metadata:
 name: emoji-sa
 namespace: default
spec:
 connectionPreference: ACCEPT_AUTOMATIC
 natSubnets:
 - gke-nat-subnet
 proxyProtocol: false
 resourceRef:
   kind: Service
   name: gke-l4-psc

Apply the manifest to your cluster from cloud shell

kubectl apply -f my-psc.yaml

The ServiceAttachment has the following fields:

• connectionPreference: the connection preference that determines how customers connect to the service. You can either use automatic project approval using ACCEPT_AUTOMATIC or explicit project approval using ACCEPT_MANUAL. For more information, see Publishing services using Private Service Connect.
• natSubnets: a list of subnetwork resource names to use for the service attachment.
• proxyProtocol: when set to true, the consumer source IP and Private Service Connect connection ID are available in the requests. This field is optional and defaults to false if not provided.
• consumerAllowList: the list of consumer projects that are allowed to connect to the ServiceAttachment. This field can only be used when connectionPreference is ACCEPT_MANUAL. For more information about this field and other options see Publishing services using Private Service Connect.

Producer Validation

Viewing the Service Attachment details

You can view the details of a ServiceAttachment using the following command from cloud shell

kubectl describe serviceattachment emoji-sa

View the GKE L4 ILB

From cloud console, navigate to Network Services → Load Balancing → Frontends

Identify the frontend IP address that aligns into the previously defined node subnet 192.168.10.0/24, note screenshot below, your IP Address may differ.

View the Published Service

From cloud console, navigate to Network Services → Private Service Connect → Published Services

Identify the service with the network used in the lab, gke-producer-l4-vpc, note screenshot below, although your Service and Target values may differ

Click on the service name that advances you to the screen below, note the service attachment details populated in the Basic Info. Also, note ‘Connected Projects' is empty since the consumer has yet to register with the service. ACCEPT and REJECT will remain greyed out since Connection preference is set to "ACCEPT_AUTOMATICALLY"', this option can be changed at any time to "ACCEPT_MANUAL" by modifying the the service attachment yaml (my-psc.yaml).

9. Create Consumers VPC network

In the following section the consumer VPC is configured in a separate project. Communication between the consumer and producer network is accomplished through the service attachment defined in the consumers network.

Codelab requires two projects, although not a requirement for PSC. Note the references to support single or multiple projects.

Single Project - Update project to support producer and consumer network

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
consumerproject=YOUR-PROJECT-NAME
prodproject=YOUR-PROJECT-NAME
echo $prodproject
echo $consumerproject

Multiple projects - Update project to support consumer a network

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
consumerproject=YOUR-PROJECT-NAME
echo $consumerproject

VPC Network

From Cloud Shell

gcloud compute networks create vpc-demo-consumer --project=$consumerproject --subnet-mode=custom

Create a subnet for PSC

From Cloud Shell

gcloud compute networks subnets create consumer-subnet --project=$consumerproject  --range=10.0.60.0/24 --network=vpc-demo-consumer --region=us-central1

Create a subnet for VM instances

From Cloud Shell

gcloud compute networks subnets create consumer-subnet-vm --project=$consumerproject  --range=10.0.70.0/24 --network=vpc-demo-consumer --region=us-central1

Create a static IP address to access the published service

From Cloud Shell

gcloud compute addresses create vpc-consumer-psc --region=us-central1 --subnet=consumer-subnet --addresses 10.0.60.100

Create Firewall Rules

To allow IAP to connect to your VM instances, create a firewall rule that:

• Applies to all VM instances that you want to be accessible by using IAP.
• Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding

From Cloud Shell

gcloud compute firewall-rules create psclab-iap-consumer --network vpc-demo-consumer --allow tcp:22 --source-ranges=35.235.240.0/20 --enable-logging

Although not required for PSC create a egress firewall rule to monitor consumer PSC traffic to the producers service attachment

gcloud compute --project=$consumerproject firewall-rules create vpc-consumer-psc --direction=EGRESS --priority=1000 --network=vpc-demo-consumer --action=ALLOW --rules=all --destination-ranges=10.0.60.0/24 --enable-logging

10. Create consumer test instance 1

From Cloud Shell

gcloud compute instances create consumer-instance-1 --zone=us-central1-a --machine-type=e2-micro --private-network-ip=10.0.70.10 --no-address --subnet=consumer-subnet-vm --tags=google1 --image-family=debian-10 --image-project=debian-cloud

11. Create consumer test instance 2

From Cloud Shell

gcloud compute instances create consumer-instance-2 --zone=us-central1-a --machine-type=e2-micro --private-network-ip=10.0.70.20 --no-address --subnet=consumer-subnet-vm --tags=google2 --image-family=debian-10 --image-project=debian-cloud

12. Create a service attachment

In a previous step, you copied the Producer Service Attachment string in a safe place, let's insert the stored value into the "target-service-attachment" field.

From Cloud Shell

gcloud compute forwarding-rules create vpc-consumer-psc-fr --region=us-central1 --network=vpc-demo-consumer --address=vpc-consumer-psc --target-service-attachment=yoursavedproducerserviceattachment

13. Validation - Consumer

We will use CURL & firewall logs to validate consumer and producer communication.

Within the Consumer's project the static IP addresses are used to originate communication to the Producer. This mapping of static IP address to Consumer forwarding rule is validated by performing the following syntax.

From the Consumer VPCs Cloud use shell identify the forwarding rule and static IP

gcloud compute forwarding-rules describe vpc-consumer-psc-fr --region us-central1

Output below, we will use 10.0.60.100 to reach the producer in a later step

IPAddress: 10.0.60.100
creationTimestamp: '2021-09-30T21:13:54.124-07:00'
id: '3564572805904938477'
kind: compute#forwardingRule
labelFingerprint: 42WmSpB8rSM=
name: vpc-consumer-psc-fr
network: https://www.googleapis.com/compute/v1/projects/deepakmichaelstage/global/networks/vpc-demo-consumer
networkTier: PREMIUM
pscConnectionId: '36583161500548196'
pscConnectionStatus: ACCEPTED

View the Connected Service

From cloud console, navigate to Network Services → Private Service Connect → Connected Endpoints & view the newly created endpoint

Lets log into consumer-instance-1 and test access to the Producer Published Service

From cloud shell open a new tab by clicking the +

From cloud shell perform the following:

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname

gcloud compute ssh --zone "us-central1-a" "consumer-instance-1" --project "$projectname"

Once logged into consumer-instance-1 instance perform a curl against the forwarding rule IP Address 10.0.60.100

From cloud shell perform the following:

user@consumer-instance-1:~$ curl 10.0.60.100

Example output

user@consumer-instance-1:~$ curl 10.0.60.100
{
  "cluster_name": "gke-psc-l4",
  "host_header": "10.0.60.100",
  "node_name": "gke-gke-psc-l4-default-pool-f2c6e301-vnlz.c.prodprojectid.internal",
  "pod_name": "psc-ilb-588887dfdb-w7tbr",
  "pod_name_emoji": "🤷",
  "project_id": "prodorijectid",
  "timestamp": "2021-10-01T17:43:37",
  "zone": "us-central1-a"

Lets log into consumer-instance-2 and test access to the Producer Published Service

From cloud shell open a new tab by clicking the +

From cloud shell perform the following:

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname

gcloud compute ssh --zone "us-central1-a" "consumer-instance-2" --project "$projectname"

From cloud shell perform the following:

user@consumer-instance-2:~$ curl 10.0.60.100

Example output

deepakmichael@consumer-instance-2:~$ curl 10.0.60.100
{
  "cluster_name": "gke-psc-l4",
  "host_header": "10.0.60.100",
  "node_name": "gke-gke-psc-l4-default-pool-f2c6e301-vnlz.c.prodproject.internal",
  "pod_name": "psc-ilb-588887dfdb-4jdql",
  "pod_name_emoji": "🧑🏿",
  "project_id": "prodproject",
  "timestamp": "2021-10-01T17:49:51",
  "zone": "us-central1-a"

14. Firewall logging - Allocated Validation

Using Logs Explorer validate firewall rule "vpc-consumner-psc" is capturing the flow between the VM's instance and static IP

1. From Cloud Console, Identify Operations Logging → Log Explorer
2. In the Query field update the entry below with yourconsumerproject and select "Run Query"

logName:(projects/yourconsumerprojectID/logs/compute.googleapis.com%2Ffirewall) AND jsonPayload.rule_details.reference:("network:vpc-demo-consumer/firewall:vpc-consumer-psc")

3. Query results provide the following per screenshot provided

4. Expand the log (jsonPayload → Connection) and identify the output provided below. Take note of the dest_ip: 10.0.60.100 is the STATIC TCP IP used to access the Producer Service and src_ip: 10.0.70.10 or 10.0.70.20 are the VM instance IP address(s). Disposition is Allowed.

15. Validation - Producer

From the Producers project verify the service attachment is connected successfully. Navigate to Network Services → Private Service Connect → Published Services

Click on the service reveals your connected consumer project and status as illustrated below

16. Restrict access to a Published Service

Thus far we have confirmed both instances have access to the Published Services, let's create an Egress Firewall Rule to deny consumer-instance-2 access to the Published Service.

By default GCP allows all Egress yet denies all Ingress traffic. In the following steps we will create a firewall rule based on a previously defined networking tag ‘google2' used when creating consumer-instance-2 to deny access to the Published Service.

Open a new cloud shell tab by clicking on the + execute the following firewall rule in cloud shell

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname

gcloud compute --project=$projectname firewall-rules create psc-endpoint-deny-egress --direction=EGRESS --priority=999 --network=vpc-demo-consumer --action=DENY --rules=all --destination-ranges=10.0.60.100/32 --target-tags=google2 --enable-logging

Now, let's test if consumer-instance-2 is able to access the Published Service. If your session timed out you will need to open a new cloud shell + and log into the VM as detailed below.

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
projectname=YOUR-PROJECT-NAME
echo $projectname

gcloud compute ssh --zone "us-central1-a" "consumer-instance-2" --project "$projectname"

From cloud shell perform the following:

user@consumer-instance-2:~$ curl 10.0.60.100

Example output

user@consumer-instance-2:~$ curl 10.0.60.100
curl: (7) Failed to connect to 10.0.60.100 port 80: Connection timed out

Firewall logging - Denied Validation

Using Logs Explorer validate firewall rule "psc-endpoint-deny-egress" is capturing the flow between the VM instance and static IP

1. From Cloud Console, Identify Operations Logging → Log Explorer
2. In the Query field update the entry below with yourconsumerproject and select "Run Query"

logName:(projects/yourconsumerprojectID/logs/compute.googleapis.com%2Ffirewall) AND jsonPayload.rule_details.reference:("network:vpc-demo-consumer/firewall:psc-endpoint-deny-egress")

3. Query results provide the following per screenshot provided

5. Expand the log and identify the output provided below. Take note of the dest_ip: 10.0.60.100 is the STATIC TCP IP and src_ip: 10.0.70.10 or 10.0.70.20 are the VM instance IP address(s). Disposition is Denied.

17. Cleanup steps

Producer network clean up steps

From a single cloud shell in the Producer project terminal delete lab components

gcloud container clusters delete gke-psc-l4 --region us-central1-a --quiet

gcloud compute networks subnets delete gke-nat-subnet --region=us-central1 --quiet

gcloud compute networks subnets delete node-subnet1 --region=us-central1 --quiet

gcloud compute networks delete gke-producer-l4-vpc --quiet

Consumer network clean up steps

From a single cloud shell in the Consumer project terminal delete lab components

gcloud compute instances delete consumer-instance-1 --zone=us-central1-a --quiet

gcloud compute instances delete consumer-instance-2 --zone=us-central1-a --quiet

gcloud compute forwarding-rules delete vpc-consumer-psc-fr --region=us-central1 --quiet

gcloud compute addresses delete vpc-consumer-psc --region=us-central1 --quiet

gcloud compute firewall-rules delete psclab-iap-consumer --quiet

gcloud compute networks subnets delete consumer-subnet --region=us-central1 --quiet

gcloud compute networks subnets delete consumer-subnet-vm --region=us-central1 --quiet

gcloud compute firewall-rules delete vpc-consumer-psc --quiet

gcloud compute firewall-rules delete psc-endpoint-deny-egress --quiet

gcloud compute networks delete vpc-demo-consumer --quiet

18. Congratulations!

Congratulations for completing the codelab.

What we've covered

• Private service connect benefits
• Key concepts for service consumers
• Key concepts for service producers
• Create a producer environment
• Expose service (producer environment) through a service attachment
• Create a consumer environment
• Create a forwarding rule in the consumer network
• Validate consumer access
• Enable policy access control
• Used an egress firewall rule to block access to a consumer forwarding rule