Looker PSC Southbound HTTPS Internet NEG SMTP

1. Introduction

NOTE: Looker supports Southbound Producers in any region. For Northbound access to Looker, the consumer endpoint or backend must be in the same region as the Looker PSC instance.

In this codelab you will create a southbound connection to Lookers SMTP service using an internal tcp proxy load balancer and internet network endpoint group (NEG) with the FQDN notifications-pa.googleapis.com invoked from Looker PSC as a Service Consumer.

Private Service Connect is a capability of Google Cloud networking that allows consumers to access managed services privately from inside their VPC network. Similarly, it allows managed service producers to host these services in their own separate VPC networks and offer a private connection to their consumers. For example, when you use Private Service Connect to access Looker, you are the service consumer, and Google is the service producer, as highlighted in Figure 1.

Figure 1.

Southbound access, also known as reverse PSC, enables the Consumer to create a Published Service as a Producer to allow Looker access to endpoints on-premises, in a VPC, to managed services and the Internet. Southbound connections can be deployed in any region, irrespective of where Looker PSC is deployed, as highlighted in Figure 2.

Figure 2.

What you'll learn

• Network requirements
• Create a Private Service Connect producer service
• Create a Private Service Connect endpoint in Looker
• Establish connectivity to Lookers SMTP service

What you'll need

• Google Cloud Project with Owner permissions
• Existing Looker PSC Instance

2. What you'll build

You'll establish a Producer network, looker-psc-demo, to deploy internal tcp proxy load balancer and Internet NEG published as a service via Private Service Connect (PSC). Once published, you'll perform the following actions to validation access to the Producer service:

• Create a PSC Endpoint in Looker associated with the Producer Service Attachment
• Use the Looker Console to create a SMTP mail settings

3. Network requirements

Below is the breakdown of network requirements for the Producer network, the consumer in this codelab is the Looker PSC instance.

Components	Description
VPC (looker-psc-demo)	Custom mode VPC
PSC NAT Subnet	Packets from the consumer VPC network are translated using source NAT (SNAT) so that their original source IP addresses are converted to source IP addresses from the NAT subnet in the producer's VPC network.
PSC forwarding rule subnet	Used to allocate an IP address for the Regional Internal TCP Proxy Load Balancer
PSC NEG Subnet	Used to allocate an IP address for the Network Endpoint Group
Proxy Only Subnet	Each of the load balancer's proxies is assigned an internal IP address. Packets sent from a proxy to a backend VM or endpoint has a source IP address from the proxy-only subnet.
Internet NEG	A resource used to define an external backend for the load balancer. The endpoint cannot be reachable only over Cloud VPN or Cloud Interconnect.
Backend Service	A backend service acts as a bridge between your load balancer and your backend resources. In the tutorial, the backend service is associated with the Internet NEG.
Cloud Router	Cloud NAT relies on Cloud Routers for control plane capabilities, but not for BGP session management.
Cloud NAT	The regional internet NEG leverages Cloud NAT for internet egress.

4. Codelab topology

5. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can always update it.
• The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference your Project ID (typically identified as PROJECT_ID). If you don't like the generated ID, you might generate another random one. Alternatively, you can try your own, and see if it's available. It can't be changed after this step and remains for the duration of the project.
• For your information, there is a third value, a Project Number, which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID is globally unique and can't be used by anyone else after you've selected it. You are the only user of that ID. Even if a project is deleted, the ID can't be used again

Note: If you use a Gmail account, you can leave the default location set to No organization. If you use a Google Workspace account, choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab won't cost much, if anything at all. To shut down resources to avoid incurring billing beyond this tutorial, you can delete the resources you created or delete the project. New Google Cloud users are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.

6. Before you begin

Enable APIs

Inside Cloud Shell, make sure that your project id is set up:

gcloud config list project
gcloud config set project [YOUR-PROJECT-ID]
project=[YOUR-PROJECT-ID]
region=[YOUR-REGION]
echo $project
echo $region

Enable all necessary services:

gcloud services enable compute.googleapis.com

7. Create Producer VPC Network

VPC Network

Inside Cloud Shell, perform the following:

gcloud compute networks create looker-psc-demo --subnet-mode custom

Create Subnets

The PSC subnet will be associated with the PSC Service Attachment for the purpose of Network Address Translation.

Inside Cloud Shell, create the PSC NAT Subnet:

gcloud compute networks subnets create producer-psc-nat-subnet --network looker-psc-demo --range 172.16.10.0/28 --region $region --purpose=PRIVATE_SERVICE_CONNECT

Inside Cloud Shell, create the producer forwarding rule subnet:

gcloud compute networks subnets create producer-psc-fr-subnet --network looker-psc-demo --range 172.16.20.0/28 --region $region --enable-private-ip-google-access

Inside Cloud Shell, create the producer regional proxy only subnet:

gcloud compute networks subnets create $region-proxy-only-subnet \
  --purpose=REGIONAL_MANAGED_PROXY \
  --role=ACTIVE \
  --region=$region \
  --network=looker-psc-demo \
  --range=10.10.10.0/24

Create the Public NAT gateway

The NAT Gateway is used by the regional internal tcp proxy load balancer for internet egress with the configuration option, –endpoint-types=ENDPOINT_TYPE_MANAGED_PROXY_LB, therefore the same NATGW will not support GCE/GKE internet egress. Deploy an additional NAT GW with the –endpoint-types=ENDPOINT_TYPE_VM for GCE/GKE internet egress.

Inside Cloud Shell, create the Cloud Router:

gcloud compute routers create looker-psc-demo-cloud-router --network looker-psc-demo --region $region

Inside Cloud Shell, create the Cloud NAT gateway enabling internet egress for the tcp proxy load balancer:

gcloud compute routers nats create looker-psc-demo-natgw \
  --router=looker-psc-demo-cloud-router \
  --endpoint-types=ENDPOINT_TYPE_MANAGED_PROXY_LB \
  --nat-custom-subnet-ip-ranges=$region-proxy-only-subnet \
  --auto-allocate-nat-external-ips \
  --region=$region

Reserve the load balancer's IP address

Inside Cloud Shell, reserve an internal IP address for the load balancer:

gcloud compute addresses create internet-neg-lb-ip \
  --region=$region \
  --subnet=producer-psc-fr-subnet

Inside Cloud Shell, view the reserved IP Address.

gcloud compute addresses describe internet-neg-lb-ip \
  --region=$region | grep -i address:

Example output:

user@cloudshell$ gcloud compute addresses describe internet-neg-lb-ip   --region=$region | grep -i address:
address: 172.16.20.2

Set up the Internet NEG

Create an Internet NEG, and set the –network-endpoint-type to internet-fqdn-port (the hostname and port where your external backend can be reached).

Inside Cloud Shell, create a Internet NEG used for github.com

gcloud compute network-endpoint-groups create smtp-internet-neg \
    --network-endpoint-type=INTERNET_FQDN_PORT \
    --network=looker-psc-demo \
    --region=$region

NOTE: fqdn=notifications-pa.googleapis.com,port=443 is required to communicate with Lookers SMTP service

Inside Cloud Shell, update the Internet NEG smtp-internet-neg with the FQDN notifications-pa.googleapis.com and port 443

gcloud compute network-endpoint-groups update smtp-internet-neg \
    --add-endpoint="fqdn=notifications-pa.googleapis.com,port=443" \
    --region=$region

Create Network Firewall Policy and Firewall Rules

Inside Cloud Shell, perform the following:

gcloud compute network-firewall-policies create looker-psc-demo-policy --global

gcloud compute network-firewall-policies associations create --firewall-policy looker-psc-demo-policy --network looker-psc-demo --name looker-psc-demo --global-firewall-policy

The following firewall rule allows traffic from the PSC NAT Subnet range to all instances in the network.

Inside Cloud Shell, perform the following:

gcloud compute network-firewall-policies rules create 2001 --action ALLOW --firewall-policy looker-psc-demo-policy --description "allow traffic from PSC NAT subnet" --direction INGRESS --src-ip-ranges 172.16.10.0/28 --global-firewall-policy --layer4-configs=tcp

8. Create Producer Service

Create Load Balancer Components

Inside Cloud Shell, perform the following:

gcloud compute backend-services create producer-backend-svc  --protocol=tcp --region=$region --load-balancing-scheme=INTERNAL_MANAGED

gcloud compute backend-services add-backend producer-backend-svc --network-endpoint-group=smtp-internet-neg --network-endpoint-group-region=$region --region=$region

In Cloud Shell, Create a target TCP proxy to route requests to your backend service:

gcloud compute target-tcp-proxies create producer-lb-tcp-proxy \
      --backend-service=producer-backend-svc  \
      --region=$region

In the following syntax, create a forwarding rule (internal tcp proxy load balancer).

In Cloud Shell, perform the following:

gcloud compute forwarding-rules create producer-smtp-fr \
     --load-balancing-scheme=INTERNAL_MANAGED \
     --network-tier=PREMIUM \
     --network=looker-psc-demo \
     --subnet=producer-psc-fr-subnet \
     --address=internet-neg-lb-ip \
     --target-tcp-proxy=producer-lb-tcp-proxy \
     --target-tcp-proxy-region=$region \
     --region=$region \
     --ports=443

Create Service Attachment

Inside Cloud Shell, create the Service Attachment, smtp-svc-attachment:

gcloud compute service-attachments create smtp-svc-attachment --region=$region --producer-forwarding-rule=producer-smtp-fr --connection-preference=ACCEPT_AUTOMATIC --nat-subnets=producer-psc-nat-subnet

Next, obtain and note the Service Attachment listed in the selfLink URI starting with projects to configure the PSC endpoint in Looker.

selfLink: projects/<your-project-id>/regions/<your-region>/serviceAttachments/github-svc-attachment-https

Inside Cloud Shell, perform the following:

gcloud compute service-attachments describe smtp-svc-attachment --region=$region

Example:

connectionPreference: ACCEPT_AUTOMATIC
creationTimestamp: '2024-10-04T14:56:50.409-07:00'
description: ''
enableProxyProtocol: false
fingerprint: KUPXTZjrGkw=
id: '8947818105173563981'
kind: compute#serviceAttachment
name: smtp-svc-attachment
natSubnets:
- https://www.googleapis.com/compute/v1/projects/$project/regions/$region/subnetworks/producer-psc-nat-subnet
pscServiceAttachmentId:
  high: '23100082169578472'
  low: '8947818105173563981'
reconcileConnections: false
region: https://www.googleapis.com/compute/v1/projects/$project/regions/$region
selfLink: https://www.googleapis.com/compute/v1/projects/$project/regions/$region/serviceAttachments/smtp-svc-attachment
targetService: https://www.googleapis.com/compute/v1/projects/$project/regions/$region/forwardingRules/producer-smtp-fr

In Cloud Console, navigate to:

Network Services → Private Service Connect → Published Services

9. Establish a PSC Endpoint Connection in Looker

In the following section, you will associate the Producers Service Attachment with Looker Core PSC through the use –psc-service-attachment flags in Cloud Shell for a single domain.

NOTE: Each time you run an update command with –psc-service-attachment flags, you must include every connection that you want to be enabled, including connections that were already enabled previously. Reference Looker PSC docs for an example.

NOTE: The flag domain=notifications-pa.googleapis.com is required to establish for SMTP

NOTE: Your Looker PSC Instance needs to be enabled before proceeding.

Inside Cloud Shell, create the psc association by updating the following parameters to match your environment:

• INSTANCE_NAME: The name of your Looker (Google Cloud core) instance.
• DOMAIN_1: notifications-pa.googleapis.com
• SERVICE_ATTACHMENT_1: URI captured when describing the Service Attachment, smtp-svc-attachment
• REGION: The region in which your Looker (Google Cloud core) instance is hosted.

Inside Cloud Shell, perform the following:

gcloud looker instances update INSTANCE_NAME \
--psc-service-attachment  domain=DOMAIN_1,attachment=SERVICE_ATTACHMENT_URI_1 \
--region=REGION

Example:

gcloud looker instances update looker-psc-instance \
--psc-service-attachment  domain=notifications-pa.googleapis.com,attachment=projects/$project/regions/$region/serviceAttachments/smtp-svc-attachment \
--region=$region

Inside Cloud Shell, validate the serviceAttachments connectionStatus is "ACCEPTED", update with your Looker PSC INSTANCE_NAME

gcloud looker instances describe [INSTANCE_NAME] --region=$region --format=json

Example:

gcloud looker instances describe looker-psc-instance --region=$region --format=json

Example:

{
...........................
    "serviceAttachments": [
      {
        "connectionStatus": "ACCEPTED",
        "localFqdn": "notifications-pa.googleapis.com",
        "targetServiceAttachmentUri": "projects/$project/regions/$region/serviceAttachments/smtp-svc-attachment"
      }
    ]
  },
  "pscEnabled": true,
  "state": "ACTIVE",
  "updateTime": "2024-10-04T22:02:31.445761128Z"
}

Validate the PSC endpoint in Cloud Console

From Cloud Console you can validate the PSC Connection

In Cloud Console, navigate to:

Looker → Looker Instance → Details

10. Test SMTP Connectivity

In the following steps, you'll use Looker Console to validate SMTP connectivity by generating a SMTP Test & report.

Test email

Looker sends email using the default SMTP server notifications-pa.googleapis.com. No additional steps are needed in this case. If you wish to use a different SMTP server for notifications, please refer to the section custom SMTP configuration.

Open the Looker Console and navigate to the following:

Admin → Platform → SMTP → Select Send Test Email

Below is the test email obtained from Looker:

Generated report email

Navigate to the following:

Explore → System Activity → API Usage → API Query Type → Run

Example output:

To send the results to your email select the gear icon

Select Send that opens a new window.

Insert your email address and select send

Below is the test email obtained from Looker:

11. Custom SMTP configuration

In this tutorial the Looker SMTP server is used to send email notifications thru the FQDN notifications-pa.googleapis.com and Cloud NAT performing internet egress. To use your own SMTP server create a new service attachment with the following modifications:

Update the internet NEG with your SMTP Server FQDN:

gcloud compute network-endpoint-groups update smtp-internet-neg \
    --add-endpoint="fqdn=<your SMTP FQDN>,port=443" \
    --region=$region

Specify your SMTP domain when updating the Looker PSC Instance:

• INSTANCE_NAME: The name of your Looker (Google Cloud core) instance.
• DOMAIN_1: <your SMTP server FQDN>
• SERVICE_ATTACHMENT_1: URI captured when describing the Service Attachment, smtp-svc-attachment
• REGION: The region in which your Looker (Google Cloud core) instance is hosted.

gcloud looker instances update INSTANCE_NAME \
--psc-service-attachment  domain=DOMAIN_1,attachment=SERVICE_ATTACHMENT_URI_1 \
--region=REGION

To use a different email service, select Use custom mail settings.

12. Clean up

From a single Cloud Shell terminal delete lab components

gcloud compute service-attachments delete smtp-svc-attachment --region=$region -q

gcloud compute forwarding-rules delete producer-smtp-fr --region=$region -q

gcloud compute target-tcp-proxies delete producer-lb-tcp-proxy --region=$region -q

gcloud compute backend-services delete producer-backend-svc --region=$region -q

gcloud compute network-firewall-policies rules delete 2001 --firewall-policy looker-psc-demo-policy --global-firewall-policy -q

gcloud compute network-firewall-policies associations delete --firewall-policy=looker-psc-demo-policy  --name=looker-psc-demo --global-firewall-policy -q

gcloud compute network-firewall-policies delete looker-psc-demo-policy --global -q

gcloud compute routers nats delete looker-psc-demo-natgw --router=looker-psc-demo-cloud-router --router-region=$region -q

gcloud compute routers delete looker-psc-demo-cloud-router --region=$region -q

gcloud compute network-endpoint-groups delete smtp-internet-neg --region=$region -q

gcloud compute addresses delete internet-neg-lb-ip --region=$region -q

gcloud compute networks subnets delete producer-psc-fr-subnet producer-psc-nat-subnet $region-proxy-only-subnet --region=$region -q

gcloud compute networks delete looker-psc-demo -q

13. Congratulations

Congratulations, you've successfully configured and validated connectivity to Lookers SMTP Service using Looker Console powered by Private Service Connect.

You created the producer infrastructure, learned how to create an Internet NEG, Producer Service and Looker PSC endpoint that allowed connectivity to the Producer service.

Cosmopup thinks codelabs are awesome!!

What's next?

Check out some of these codelabs...

• Using Private Service Connect to publish and consume services
• Connect to on-prem services over Hybrid Networking using Private Service Connect and an internal TCP Proxy load balancer
• Access to all published Private Service Connect codelabs

Further reading & Videos

• Private Service Connect overview

Reference docs

• Set up a regional internal proxy Network Load Balancer with an external backend
• Cloud NAT overview
• Create a Looker (Google Cloud core) Private Service Connect instance
• How to publish a service using Private Service Connect
• Admin settings - SMTP | Looker | Google Cloud