Policy Based Routes (PBR) Codelab

1. Introduction

Policy Based Routes

Policy-based routes let you choose a next hop based on more than a packet's destination IP address. You can match traffic by protocol and source IP address as well. Matching traffic is redirected to an Internal Network load balancer. This can help you insert appliances such as firewalls into the path of network traffic.

When you create a policy-based route, you select which resources can have their traffic processed by the route. The route can apply to the following:

  • The entire network: all virtual machine (VM) instances, VPN gateways, and Interconnects
  • Using network tags: select VM instances in the VPC
  • Interconnect region: All traffic entering the VPC network by way of VLAN attachments for the region

The next hop of a policy-based route must be a valid Internal Network load balancer that is in the same VPC network as the policy-based route.

Internal Network load balancers use symmetric hashing by default, so traffic can reach the same appliance on the outgoing and return paths without configuring source NAT.

Policy-based routes have a higher priority than other route types except for special return paths.

If two policy-based routes have the same priority, Google Cloud uses a deterministic, internal algorithm to select a single policy-based route, ignoring other routes with the same priority. Policy-based routes do not use longest-prefix matching and only select the highest priority route.

You can create a single rule for one-way traffic or multiple rules to handle bidirectional traffic.

To use policy-based routes with Cloud Interconnect, the route must be applied to all Cloud Interconnect connections in an entire region. Policy-based routes cannot be applied to an individual Cloud Interconnect connection only.

The VM instances that receive traffic from a policy-based route must have IP forwarding enabled.

Considerations with PBR

Special configuration is necessary to use policy-based routes in the following ways.

For example using PBR with GKE, PSC, or with PGA/PSA.

More details on PBR with GKE can be found here and the general PBR limitations section here.

What you'll learn

  • How to configure a policy based routes

What you'll need

  • Knowledge of deploying instances and configuring networking components
  • VPC Firewall configuration knowledge

2. Test Environment

This Codelab will leverage a single VPC. There will be two compute resources, clienta and clientb, in this environment that will send packets to another server resource. Using PBR and filters, we will force traffic from clienta through another compute resource for firewall enforcement while clientb traffic goes directly to the server. The diagram below illustrates the path.

bff32b01ada8e9ad.png

In the diagram above, there should technically be an ILB (network internal load balancer) for PBR paths. This has been omitted for diagram simplicity.

3. Before you begin

Codelab requires a single project.

From cloudshell:

export project_id=`gcloud config list --format="value(core.project)"`
export region=us-central1
export zone=us-central1-a
export prefix=codelab-pbr

4. Enable APIs

If not already done, enable the APIs to use the products

From cloudshell:

gcloud services enable compute.googleapis.com
gcloud services enable networkconnectivity.googleapis.com

5. Create VPC network and subnet

VPC Network

Create codelab-pbr-vpc VPC:

From cloudshell:

gcloud compute networks create $prefix-vpc --subnet-mode=custom 

Subnet

Create the respective subnets in the selected region:

From cloudshell:

gcloud compute networks subnets create $prefix-vpc-subnet \
   --range=10.10.10.0/24 --network=$prefix-vpc --region=${region}

6. Create Firewall Rules

To allow IAP to connect to your VM instances, create a firewall rule that:

  • Applies to all VM instances that you want to be accessible by using IAP.
  • Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.

From cloudshell:

gcloud compute firewall-rules create $prefix-allow-iap-proxy \
--direction=INGRESS \
--priority=1000 \
--network=$prefix-vpc \
--action=ALLOW \
--rules=tcp:22 \
--source-ranges=35.235.240.0/20

To allow the standard HTTP port (TCP 80) and ICMP protocol to the server:

  • Applies to resources with network tag "server"
  • Allows ingress from all sources

From cloudshell:

gcloud compute firewall-rules create $prefix-allow-http-icmp \
--direction=INGRESS \
--priority=1000 \
--network=$prefix-vpc \
--action=ALLOW \
--rules=tcp:80,icmp \
--source-ranges=0.0.0.0/0 \
--target-tags=server

To allow the FW to receive packets, allow an ingress on all protocols and ports.

  • Applies to resources with network tag "fw"
  • Allows ingress from 10.10.10.0/24 sources

From cloudshell:

gcloud compute firewall-rules create $prefix-fw-allow-ingress \
--direction=INGRESS \
--priority=1000 \
--network=$prefix-vpc \
--action=ALLOW \
--rules=all \
--source-ranges=10.10.10.0/24 \
--target-tags=fw

To allow the health checks probes

  • Applies to resources with the network tag "fw"
  • Allows ingress from health check ranges

From cloudshell:

gcloud compute firewall-rules create $prefix-allow-hc-ingress \
--direction=INGRESS \
--priority=1000 \
--network=$prefix-vpc \
--action=ALLOW \
--rules=tcp:80 \
--source-ranges=130.211.0.0/22,35.191.0.0/16 \
--target-tags=fw

7. Create Cloud Router & Cloud NAT

The purpose of this section is so that the private virtual machines can download the appropriate software packages from the internet.

Create Cloud Router

From cloudshell:

gcloud compute routers create ${prefix}-cr \
--region=${region} \
--network=${prefix}-vpc

Create Cloud NAT Gateway

From cloudshell:

gcloud compute routers nats create ${prefix}-nat-gw-${region} \
--router=${prefix}-cr \
--router-region=${region} \
--auto-allocate-nat-external-ips \
--nat-all-subnet-ip-ranges

8. Create Compute Instances

Create the compute instances ClientA, ClientB, FW, and Server:

From cloudshell:

gcloud compute instances create clienta \
   --subnet=$prefix-vpc-subnet \
   --no-address \
   --private-network-ip=10.10.10.10 \
   --zone $zone \
   --tags client \
   --metadata startup-script='#! /bin/bash
apt-get update'

From cloudshell:

gcloud compute instances create clientb \
   --subnet=$prefix-vpc-subnet \
   --no-address \
   --private-network-ip=10.10.10.11 \
   --zone $zone \
   --tags client \
   --metadata startup-script='#! /bin/bash
apt-get update'

From cloudshell:

gcloud compute instances create server \
   --subnet=$prefix-vpc-subnet \
   --no-address \
   --private-network-ip=10.10.10.200 \
   --zone $zone \
   --tags server \
   --metadata startup-script='#! /bin/bash
sudo su
apt-get update
apt-get -y install tcpdump
apt-get -y install nginx
cat > /var/www/html/index.html << EOF
<html><body><p>Server</p></body></html>
EOF'

From cloudshell:

gcloud compute instances create fw \
   --subnet=$prefix-vpc-subnet \
   --can-ip-forward \
   --no-address \
   --private-network-ip=10.10.10.75 \
   --zone $zone \
   --tags fw \
   --metadata startup-script='#! /bin/bash
apt-get update
sudo apt-get -y install tcpdump
sudo apt-get -y install nginx
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -I FORWARD -d 10.10.10.200 -j REJECT'

9. Test Connectivity without PBR

SSH into client compute VMs we recently created and verify connectivity from both clients to server.

From cloudshell1 log in to clienta:

gcloud compute ssh clienta --zone=$zone --tunnel-through-iap

Run the following commands:

ping 10.10.10.200 -c 5
curl 10.10.10.200/index.html

The pings and curl requests should be successful.

Output:

root@clienta:~$ ping 10.10.10.200 -c 5
PING 10.10.10.200 (10.10.10.200) 56(84) bytes of data.
64 bytes from 10.10.10.200: icmp_seq=1 ttl=64 time=1.346 ms
64 bytes from 10.10.10.200: icmp_seq=2 ttl=64 time=0.249 ms
64 bytes from 10.10.10.200: icmp_seq=3 ttl=64 time=0.305 ms
64 bytes from 10.10.10.200: icmp_seq=4 ttl=64 time=0.329 ms
64 bytes from 10.10.10.200: icmp_seq=5 ttl=64 time=0.240 ms
root@clienta:~$ curl 10.10.10.200/index.html
<html><body><p>Server</p></body></html>

Open an additional cloudshell tab by clicking the +.

3722d8574c3812b1.png

From cloudshell2 set variables for use:

export project_id=`gcloud config list --format="value(core.project)"`
export region=us-central1
export zone=us-central1-a
export prefix=codelab-pbr

From cloudshell2 SSH to clientb:

gcloud compute ssh clientb --zone=$zone --tunnel-through-iap

Run the following commands:

ping 10.10.10.200 -c 5
curl 10.10.10.200/index.html

The pings and curl requests should be successful.

Output:

root@clientb:~$ ping 10.10.10.200 -c 5
PING 10.10.10.200 (10.10.10.200) 56(84) bytes of data.
64 bytes from 10.10.10.200: icmp_seq=1 ttl=64 time=1.346 ms
64 bytes from 10.10.10.200: icmp_seq=2 ttl=64 time=0.249 ms
64 bytes from 10.10.10.200: icmp_seq=3 ttl=64 time=0.305 ms
64 bytes from 10.10.10.200: icmp_seq=4 ttl=64 time=0.329 ms
64 bytes from 10.10.10.200: icmp_seq=5 ttl=64 time=0.240 ms
root@clientb:~$ curl 10.10.10.200/index.html
<html><body><p>Server</p></body></html>

Now exit the VM terminal and head back to cloudshell.

10. Create an Instance Group

Create an unmanaged instance group for your fw VM.

From cloudshell:

gcloud compute instance-groups unmanaged create pbr-uig --zone=$zone

Add the fw instance to the unmanaged instance group.

From cloudshell:

gcloud compute instance-groups unmanaged add-instances pbr-uig --instances=fw --zone=$zone

11. Create a health check

Create a health check for backend service. We will do a simple TCP port 80 health check.

From cloudshell:

gcloud compute health-checks create tcp $prefix-hc-tcp-80 --region=$region --port 80

12. Create a backend service

Create a backend service to attach to forwarding rule.

From cloudshell:

gcloud compute backend-services create be-pbr --load-balancing-scheme=internal --protocol=tcp --region=$region --health-checks=$prefix-hc-tcp-80 --health-checks-region=$region

Now add the instance group to the backend service.

From cloudshell:

gcloud compute backend-services add-backend be-pbr --region=$region --instance-group=pbr-uig --instance-group-zone=$zone

13. Create a forwarding rule

From cloudshell:

gcloud compute forwarding-rules create fr-pbr --region=$region --load-balancing-scheme=internal --network=$prefix-vpc --subnet=$prefix-vpc-subnet --ip-protocol=TCP --ports=ALL --backend-service=be-pbr --backend-service-region=$region --address=10.10.10.25 --network-tier=PREMIUM

14. Create PBR Rule

This PBR rule applies to clients. It will route all IPv4 traffic to the forwarding rule 10.10.10.25 if the source IP is 10.10.10.10/32 (clienta's address) and the destination IP is 10.10.10.0/24.

This means clienta will match PBR and not clientb.

From cloudshell:

gcloud network-connectivity policy-based-routes create pbr-client \
   --network=projects/$project_id/global/networks/$prefix-vpc \
   --next-hop-ilb-ip=10.10.10.25 \
   --source-range=10.10.10.10/32 \
   --destination-range=10.10.10.0/24 \
   --protocol-version=IPv4 \
   --priority=1000 \
   --tags=client

This PBR rule applies to the server. It will route all IPv4 traffic to the forwarding rule 10.10.10.25 if the source IP is 10.10.10.200/32 and the destination IP is 10.10.10.10/32.

From cloudshell:

gcloud network-connectivity policy-based-routes create pbr-server \
   --network=projects/$project_id/global/networks/$prefix-vpc \
   --next-hop-ilb-ip=10.10.10.25 \
   --source-range=10.10.10.200/32 \
   --destination-range=10.10.10.10/32 \
   --protocol-version=IPv4 \
   --priority=2000 \
   --tags=server

15. Test Connectivity with PBR

We will now verify PBR functionality. The "fw" instance is configured with iptables to reject requests destined for the server. If PBR is functional, the requests that worked previously on clienta will now fail, while clientb is still successful.

SSH into clienta compute VM and run the same tests.

From cloudshell1:

gcloud compute ssh clienta --zone=$zone --tunnel-through-iap

Run the following commands:

ping 10.10.10.200 -c 5
curl 10.10.10.200/index.html

Output:

root@clienta:~$ ping 10.10.10.200 -c 5
PING 10.10.10.200 (10.10.10.200) 56(84) bytes of data.
From 10.10.10.75 icmp_seq=1 Destination Port Unreachable
From 10.10.10.75 icmp_seq=2 Destination Port Unreachable
From 10.10.10.75 icmp_seq=3 Destination Port Unreachable
From 10.10.10.75 icmp_seq=4 Destination Port Unreachable
From 10.10.10.75 icmp_seq=5 Destination Port Unreachable
root@clienta:~$ curl 10.10.10.200/index.html
curl: (7) Failed to connect to 10.10.10.200 port 80: Connection refused

Since the requests failed, we can confirm that PBR is actively routing traffic for clienta to the fw instance which was configured to block this traffic.

SSH into clientb and run the same connectivity test.

From cloudshell2:

gcloud compute ssh clientb --zone=$zone --tunnel-through-iap

Run the following commands:

ping 10.10.10.200 -c 5
curl 10.10.10.200/index.html

Output:

root@clientb:~$ ping 10.10.10.200 -c 5
PING 10.10.10.200 (10.10.10.200) 56(84) bytes of data.
64 bytes from 10.10.10.200: icmp_seq=1 ttl=63 time=0.361 ms
64 bytes from 10.10.10.200: icmp_seq=2 ttl=63 time=0.475 ms
64 bytes from 10.10.10.200: icmp_seq=3 ttl=63 time=0.379 ms
root@clientb:~$ curl 10.10.10.200
<html><body><p>Server</p></body></html>

As you can see, requests from clientb to server are successful. This is because the requests do not match a PBR rule for the source IP.

16. [Optional] Validating with captures on firewall

In this optional section, you have the opportunity to validate PBR by taking packet captures on the firewall VM.

You should still have an SSH connection in cloudshell1 and cloudshell2 to clienta and clientb.

Open an additional cloudshell tab by clicking the +.

5eb3a9956f7e41a2.png

From cloudshell3, set variables:

export project_id=`gcloud config list --format="value(core.project)"`
export region=us-central1
export zone=us-central1-a
export prefix=codelab-pbr

SSH into fw:

gcloud compute ssh fw --zone=$zone --tunnel-through-iap

Run the following command on fw (cloudshell3):

sudo tcpdump -i any icmp -nn

From clienta (cloudshell1) run the ping test:

ping 10.10.10.200 -c 5

From clientb (cloudshell2) run the ping test:

ping 10.10.10.200 -c 5

Output on fw (cloudshell 3):

root@fw:~$ sudo tcpdump -i any icmp -nn
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
17:07:42.215297 ens4  In  IP 10.10.10.10 > 10.10.10.200: ICMP echo request, id 25362, seq 1, length 64
17:07:42.215338 ens4  Out IP 10.10.10.75 > 10.10.10.10: ICMP 10.10.10.200 protocol 1 port 51064 unreachable, length 92
17:07:43.216122 ens4  In  IP 10.10.10.10 > 10.10.10.200: ICMP echo request, id 25362, seq 2, length 64
17:07:43.216158 ens4  Out IP 10.10.10.75 > 10.10.10.10: ICMP 10.10.10.200 protocol 1 port 30835 unreachable, length 92
17:07:44.219064 ens4  In  IP 10.10.10.10 > 10.10.10.200: ICMP echo request, id 25362, seq 3, length 64
17:07:44.219101 ens4  Out IP 10.10.10.75 > 10.10.10.10: ICMP 10.10.10.200 protocol 1 port 2407 unreachable, length 92

You will not see any packets on the tcpdump from clientb (10.10.10.11) since PBR is not applicable.

Exit back to cloudshell for cleaning up resources.

17. Cleanup steps

From Cloud Shell, remove the PBR rule, forwarding rule, backend service, health check, instance group, compute instances, NAT, Cloud Router, and firewall rules.

gcloud -q network-connectivity policy-based-routes delete pbr-client

gcloud -q network-connectivity policy-based-routes delete pbr-server

gcloud -q compute forwarding-rules delete fr-pbr --region=$region

gcloud -q compute backend-services delete be-pbr --region=$region

gcloud -q compute health-checks delete $prefix-hc-tcp-80 --region=$region

gcloud -q compute instance-groups unmanaged delete pbr-uig --zone=$zone

gcloud -q compute instances delete clienta --zone=$zone
gcloud -q compute instances delete clientb --zone=$zone
gcloud -q compute instances delete server --zone=$zone
gcloud -q compute instances delete fw --zone=$zone


gcloud -q compute routers nats delete ${prefix}-nat-gw-${region} \
--router=$prefix-cr --router-region=$region

gcloud -q compute routers delete $prefix-cr --region=$region

gcloud -q compute firewall-rules delete $prefix-allow-iap-proxy
gcloud -q compute firewall-rules delete $prefix-allow-http-icmp
gcloud -q compute firewall-rules delete $prefix-fw-allow-ingress
gcloud -q compute firewall-rules delete $prefix-allow-hc-ingress

Remove the subnet and VPCs:

gcloud -q compute networks subnets delete $prefix-vpc-subnet \
    --region $region

gcloud -q compute networks delete $prefix-vpc

18. Congratulations!

Congratulations for completing the codelab.

What we've covered

  • Policy Based Routes