Rate Limiting with Cloud Armor

1. Introduction

Google Cloud HTTP(S) load balancing is deployed at the edge of Google's network in Google points of presence (POP) around the world. User traffic directed to an HTTP(S) load balancer enters the POP closest to the user and is then load balanced over Google's global network to the closest backend that has sufficient capacity available.

Cloud Armor is Google's distributed denial of service and web application firewall (WAF) detection system. Cloud Armor is tightly coupled with the Google Cloud HTTP Load Balancer and allows you to interrogate incoming traffic for unwanted requests. The rate limiting feature of this service allows you to curtail traffic to backend resources based on request volume and prevents unwelcome traffic from consuming resources on your Virtual Private Cloud (VPC) network.

In this lab, you configure an HTTP Load Balancer with global backends, as shown in the diagram below. Then, you'll stress test the Load Balancer and add a Cloud Armor rate limiting policy to limit the traffic hitting your backend resources.

What you'll learn

• How to set up a HTTP Load Balancer with appropriate health checks.
• How to create a Cloud Armor rate limiting policy.
• How to validate that the rate limiting policy is blocking traffic when running a stress test from a VM.

What you'll need

• Basic Networking and knowledge of HTTP
• Basic Unix/Linux command line knowledge

2. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs, and you can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" after the project is created.
• There is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. If a project is deleted, that ID can never be used again.

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console in order to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, follow any "clean-up" instructions found at the end of the codelab. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

Before you begin

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
PROJECT_ID=[YOUR-PROJECT-NAME]
echo $PROJECT_ID

Enable APIs

Enable all necessary services

gcloud services enable compute.googleapis.com
gcloud services enable logging.googleapis.com
gcloud services enable monitoring.googleapis.com

3. Configure firewall rules to allow HTTP traffic to backends

Configure firewall rules to allow HTTP traffic to the backends from the Google Cloud health checks and the Load Balancer.

We will be using the default VPC network created in your project. Create a firewall rule to allow HTTP traffic to the backends. Health checks determine which instances of a load balancer can receive new connections. For HTTP load balancing, the health check probes to your load balanced instances come from addresses in the ranges 130.211.0.0/22 and 35.191.0.0/16. Your VPC firewall rules must allow these connections. Also, the load balancers talk to the backend on the same IP range.

1. In the Cloud Console, navigate to Navigation menu ( ) > VPC network > Firewall.

2. Notice the existing ICMP, internal, RDP, and SSH firewall rules.Each Google Cloud project starts with the default network and these firewall rules.
3. Click Create Firewall Rule.
4. Set the following values, leave all other values at their defaults:

Property	Value (type value or select option as specified)
Name	default-allow-health-check
Network	default
Targets	Specified target tags
Target tags	http-server
Source filter	IP Ranges
Source IP ranges	130.211.0.0/22, 35.191.0.0/16
Protocols and ports	Specified protocols and ports, and then check tcp

Make sure to enter the two Source IP ranges one-by-one and press SPACE in between them.

5. Click Create.

Alternatively, if you are using the gcloud command line. Below is the command -

gcloud compute firewall-rules create default-allow-health-check --direction=INGRESS --priority=1000 --network=default --action=ALLOW --rules=tcp --source-ranges=130.211.0.0/22,35.191.0.0/16 --target-tags=http-server

4. Configure instance templates and create managed instance groups

A managed instance group uses an instance template to create a group of identical instances. Use these to create the backends of the HTTP Load Balancer.

Configure the instance templates

An instance template is an API resource that you use to create VM instances and managed instance groups. Instance templates define the machine type, boot disk image, subnet, labels, and other instance properties. Create one instance template for us-east1 and one for europe-west1.

1. In the Cloud Console, navigate to Navigation menu ( ) > Compute Engine > Instance templates, and then click Create instance template.
2. For Name, type us-east1-template.
3. For Series, select N1.
4. Click Networking, Disks, Security, Management , Sole-Tenancy.

5. Go to the Management section -

6. Under Metadata, click Add Item and specify the following:

Key	Value
startup-script-url	gs://cloud-training/gcpnet/httplb/startup.sh

The startup-script-url specifies a script that executes when instances are started. This script installs Apache and changes the welcome page to include the client IP and the name, region, and zone of the VM instance. Feel free to explore this script here.

7. Click Networking.
8. Set the following values and leave all other values at their defaults -

Property	Value (type value or select option as specified)
Network (Under Network Interfaces)	default
Subnet (Under Network Interfaces)	default (us-east1)
Network tags	http-server

The network tag http-server ensures that the HTTP and Health Check firewall rules apply to these instances.

9. Click Create.
10. Wait for the instance template to be created.

Now create another instance template for subnet-b by copying us-east1-template:

1. Click on us-east1-template and then click on the Copy option from the top.
2. For Name, type europe-west1-template.
3. Click Networking, Disks, Security, Management , Sole-Tenancy.
4. Click Networking.
5. Under Network Interfaces, edit the default interface. For Subnet, select default (europe-west1).
6. Click Create.

Create the managed instance groups

Create a managed instance group in us-east1 and one in europe-west1.

1. Still in Compute Engine, click Instance groups in the left menu.

2. Click Create instance group. Select New managed instance group (stateless).
3. Set the following values, leave all other values at their defaults:

Property	Value (type value or select option as specified)
Name	us-east1-mig
Location	Multiple zones
Region	us-east1
Instance template	us-east1-template
Autoscaling > Autoscaling Policy > Click Pencil icon > Metric type	CPU utilization
Target CPU utilization	80, click Done.
Cool-down period	45
Minimum number of instances	1
Maximum number of instances	5

Managed instance groups offer autoscaling capabilities that allow you to automatically add or remove instances from a managed instance group based on increases or decreases in load. Autoscaling helps your applications gracefully handle increases in traffic and reduces cost when the need for resources is lower. You just define the autoscaling policy and the autoscaler performs automatic scaling based on the measured load.

4. Click Create.

Now repeat the same procedure to create a second instance group for europe-west1-mig in europe-west1:

1. Click Create Instance group.
2. Set the following values, leave all other values at their defaults:

Property	Value (type value or select option as specified)
Name	europe-west1-mig
Location	Multiple zones
Region	europe-west1
Instance template	europe-west1-template
Autoscaling > Autoscaling Policy > Click Pencil icon > Metric type	CPU utilization
Target CPU utilization	80, click Done.
Cool-down period	45
Minimum number of instances	1
Maximum number of instances	5

3. Click Create.

5. Configure the HTTP Load Balancer

Configure the HTTP Load Balancer to balance traffic between the two backends (us-east1-mig in us-east1 and europe-west1-mig in europe-west1), as illustrated in the network diagram:

Start the configuration

1. In the Cloud Console, click Navigation menu ( ) > click Network Services > Load balancing, and then click Create load balancer.
2. Under HTTP(S) Load Balancing, click on Start configuration.

3. Select From Internet to my VMs, Classic HTTP(S) Load Balancer and click Continue.
4. Set the Name to http-lb.

Configure the backend

Backend services direct incoming traffic to one or more attached backends. Each backend is composed of an instance group and additional serving capacity metadata.

1. Click on Backend configuration.
2. For Backend services & backend buckets, click Create a backend service.
3. Set the following values, leave all other values at their defaults:

Property	Value (select option as specified)
Name	http-backend
Instance group	us-east1-mig
Port numbers	80
Balancing mode	Rate
Maximum RPS	50 (per instance)
Capacity	100

This configuration means that the load balancer attempts to keep each instance of us-east1-mig at or below 50 requests per second (RPS).

4. Click Done.
5. Click Add backend.
6. Set the following values, leave all other values at their defaults:

Property	Value (select option as specified)
Instance group	europe-west1-mig
Port numbers	80
Balancing mode	Utilization
Maximum backend utilization	80
Capacity	100

This configuration means that the load balancer attempts to keep each instance of europe-west1-mig at or below 80% CPU utilization.

7. Click Done.
8. For Health Check, select Create a health check.

9. Set the following values, leave all other values at their defaults:

Property	Value (select option as specified)
Name	http-health-check
Protocol	TCP
Port	80

Health checks determine which instances receive new connections. This HTTP health check polls instances every 5 seconds, waits up to 5 seconds for a response and treats 2 successful or 2 failed attempts as healthy or unhealthy, respectively.

10. Click Save.
11. Check the Enable Logging box.
12. Set the Sample Rate to 1:

13. Click Create to create the backend service.

Configure the frontend

The host and path rules determine how your traffic will be directed. For example, you could direct video traffic to one backend and static traffic to another backend. However, you are not configuring the Host and path rules in this lab.

1. Click on Frontend configuration.
2. Specify the following, leaving all other values at their defaults:

Property	Value (type value or select option as specified)
Protocol	HTTP
IP version	IPv4
IP address	Ephemeral
Port	80

3. Click Done.
4. Click Add Frontend IP and port.
5. Specify the following, leaving all other values at their defaults:

Property	Value (type value or select option as specified)
Protocol	HTTP
IP version	IPv6
IP address	Ephemeral
Port	80

6. Click Done.

HTTP(S) load balancing supports both IPv4 and IPv6 addresses for client traffic. Client IPv6 requests are terminated at the global load balancing layer, then proxied over IPv4 to your backends.

Review and create the HTTP Load Balancer

1. Click on Review and finalize.

2. Review the Backend services and Frontend.

3. Click on Create.
4. Wait for the load balancer to be created.
5. Click on the name of the load balancer (http-lb).
6. Note the IPv4 and IPv6 addresses of the load balancer for the next task. They will be referred to as [LB_IP_v4] and [LB_IP_v6], respectively.

6. Test the HTTP Load Balancer

Now that you created the HTTP Load Balancer for your backends, verify that traffic is forwarded to the backend service.

Access the HTTP Load Balancer

To test IPv4 access to the HTTP Load Balancer, open a new tab in your browser and navigate to http://[LB_IP_v4]. Make sure to replace [LB_IP_v4] with the IPv4 address of the load balancer.

It might take up to 5 minutes to access the HTTP Load Balancer. In the meantime, you might get a 404 or 502 error. Keep trying until you see the page of one of the backends.

Depending on your proximity to us-east1 and europe-west1, your traffic is either forwarded to a us-east1-mig or europe-west1-mig instance.

If you have a local IPv6 address, try the IPv6 address of the HTTP Load Balancer by navigating to http://[LB_IP_v6]. Make sure to replace [LB_IP_v6] with the IPv6 address of the load balancer.

The Hostname and Server Location identifies which backend receives the request from the HTTP Load Balancer.

Stress test the HTTP Load Balancer

Create a new VM to simulate a load on the HTTP Load Balancer using siege. Then, determine if traffic is balanced across both backends when the load is high.

1. In the Console, navigate to Navigation menu ( ) > Compute Engine > VM instances.
2. Click Create instance.
3. Set the following values, leave all other values at their defaults:

Property	Value (type value or select option as specified)
Name	siege-vm
Region	us-west1
Zone	us-west1-c
Series	N1

Given that us-west1 is closer to us-east1 than to europe-west1, traffic should be forwarded only to us-east1-mig (unless the load is too high).

4. Click Create.
5. Wait for the siege-vm instance to be created.
6. For siege-vm, click SSH to launch a terminal and connect.
7. Run the following command, to install siege:

sudo apt-get -y install siege

8. To store the IPv4 address of the HTTP Load Balancer in an environment variable, run the following command, replacing [LB_IP_v4] with the IPv4 address:

export LB_IP=[LB_IP_v4]

9. To simulate a load, run the following command:

siege -c 250 http://$LB_IP

The output should look like this (do not copy; this is example output):

New configuration template added to /home/student/.siege
Run siege -C to view the current settings in that file
** SIEGE 4.0.4
** Preparing 250 concurrent users for battle.
The server is now under siege...

10. In the Cloud Console, click on the Navigation menu ( ), click Network Services > Load balancing.
11. Click http-lb.
12. Click on the Monitoring tab. Monitor the traffic between North America and the two backends for 2 to 3 minutes.

At first, traffic should just be directed to us-east1-mig but as the RPS increases, traffic is also directed to europe-west1-mig.​​

This demonstrates that by default, traffic is forwarded to the closest backend, but if the load is very high, traffic can be distributed across the backends.

13. Return to the SSH terminal of siege-vm.
14. Press CTRL+C to stop siege.

7. Create Cloud Armor Rate Limiting Policy

In this section, you will use Cloud Armor to deny the siege-vm from accessing the HTTP Load Balancer by setting a rate limiting policy.

1. In Cloud Shell(refer to "Start Cloud Shell" under "Setup and Requirements" for instructions on how to use Cloud Shell), create security policy via gcloud:

gcloud compute security-policies create rate-limit-siege \
    --description "policy for rate limiting"

2. Next, add a rate limiting rule:

gcloud beta compute security-policies rules create 100 \
    --security-policy=rate-limit-siege     \
    --expression="true" \
    --action=rate-based-ban                   \
    --rate-limit-threshold-count=50           \
    --rate-limit-threshold-interval-sec=120   \
    --ban-duration-sec=300           \
    --conform-action=allow           \
    --exceed-action=deny-404         \
    --enforce-on-key=IP

3. Attach the security policy to the backend service http-backend:

gcloud compute backend-services update http-backend \
    --security-policy rate-limit-siege –-global

4. In the Console, navigate to Navigation menu > Network Security > Cloud Armor.
5. Click rate-limit-siege. Your policy should resemble the following:

Verify the Security Policy

1. Return to the SSH terminal of siege-vm.
2. Run a curl against the LB IP to verify you can still connect to it, should receive a 200 response.

curl http://$LB_IP

3. In the SSH terminal of siege-vm, to simulate a load, run the following command:

siege -c 250 http://$LB_IP

The output should look like this (do not copy; this is example output):

** SIEGE 4.0.4
** Preparing 250 concurrent users for battle.
The server is now under siege...

4. Explore the security policy logs to determine if this traffic is also blocked.
5. In the Console, navigate to Navigation menu > Network Security > Cloud Armor.
6. Click rate-limit-siege.
7. Click Logs.

8. Click View policy logs.
9. On the Logging page, make sure to clear all the text in the Query preview.
10. Select resource as Cloud HTTP Load Balancer > http-lb-forwarding-rule > http-lb then click Add. Alternatively, below is the MQL(monitoring query language) query, you can copy and paste into the query editer -

resource.type="http_load_balancer" resource.labels.forwarding_rule_name="http-lb-forwarding-rule" resource.labels.url_map_name="http-lb"

11. Now click Run Query.
12. Expand a log entry in Query results.
13. Expand httpRequest. The request should be from the siege-vm IP address. If not, expand another log entry.
14. Expand jsonPayload.
15. Expand enforcedSecurityPolicy.

Notice that the configuredAction is set to RATE_BASED_BAN with the name rate-limit-siege.

Cloud Armor security policies create logs that can be explored to determine when traffic is denied and when it is allowed, along with the source of the traffic.

16. As an additional check, go to on the Navigation menu ( ), click Network Services > Load balancing. Click http-lb. Click on the Monitoring tab.

You can see the siege traffic in the graphs. You will also note that the rate limited traffic does not reach the backend and is blocked by the Cloud Armor policy.

Congratulations! You have completed this lab on Rate Limiting with Cloud Armor

©2020 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.

8. Lab Clean up

1. Navigate to Network Security >> Cloud Armor >> %POLICY NAME% and select delete -

2. Navigate to Networking >> Network services >> Load Balancing. Select the load balancer you created and click delete.

Select the backend service and health check as additional resources to be deleted -

3. Navigate to Navigation menu ( ) > Compute Engine > Instance Groups. Select both the managed instances groups and click delete -

Confirm deletion by typing "delete" into the textbox.

Wait for the managed instance groups to be deleted. This also deletes the instances in the group. You can delete the templates only after the instance group has been deleted.

4. Navigate to Instance templates from the left hand side pane**.** Select both the instances templates and click delete -

5. Navigate to VM instances from the left hand side pane**.** Select the ellipsis next to the siege-vm instance and click delete.

6. Navigate to Navigation menu ( ) > VPC network > Firewall. Select the default-allow-health-check and click delete -

9. Congratulations!

You successfully implemented rate limiting with Cloud Armor. You configured an HTTP Load Balancer with backends in us-east1 and europe-west1. Then, you stress tested the Load Balancer with a VM and denylisted the IP address via rate limiting with Cloud Armor. You were able to explore the security policy logs to identify why the traffic was blocked.

What we've covered

• How to set up instance templates and create managed instance groups.
• How to set up a HTTP Load Balancer.
• How to create a Cloud Armor rate limiting policy.
• How to validate that the Rate Limiting Policy is working as intended.

Next steps

• Try setting up a Rate Limiting Policy based on a source IP range. Sample command below -

gcloud alpha compute security-policies rules create 105 \
    --security-policy sec-policy     \
    --src-ip-ranges "1.2.3.0/24"     \
    --action throttle                \
    --rate-limit-threshold-count 100 \
    --rate-limit-threshold-interval-sec 60 \
    --conform-action allow           \
    --exceed-action deny-429         \
    --enforce-on-key IP

• Try setting up a Rate Limiting Policy based on region code. Sample command below -

gcloud alpha compute security-policies rules create 101 \
    --security-policy sec-policy     \
    --expression "origin.region_code == 'US'" \
    --action rate-based-ban                 \
    --rate-limit-threshold-count 10         \
    --rate-limit-threshold-interval-sec 60  \
    --ban-duration-sec 300           \
    --ban-threshold-count 1000       \
    --ban-threshold-interval-sec 600 \
    --conform-action allow           \
    --exceed-action deny-403         \
    --enforce-on-key IP