Defending Edge Cache with Cloud Armor

1. Introduction

Google Cloud Armor Edge Security Policies allow you to restrict access to cached objects on Google Cloud CDN and Google Cloud Storage. Examples of reasons to do this include ensuring that your users do not access objects in storage buckets from restricted geographies or ensuring that your media distribution is filtering on the geographies that you have a license to do so.

In this lab we will create a GCS bucket, upload an image to it, bind it to a load balancer and then enable Cloud CDN and Edge Security policies on it.

What you'll learn

• How to set up a Cloud Storage Bucket with cacheable content
• How to create an Edge Security Policy to protect the content
• How to validate that the Edge Security Policy is working as expected

What you'll need

• Basic Networking and knowledge of HTTP
• Basic Unix/Linux command line knowledge

2. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs, and you can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" after the project is created.
• There is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. If a project is deleted, that ID can never be used again.

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console in order to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, follow any "clean-up" instructions found at the end of the codelab. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

Before you begin

Inside Cloud Shell, make sure that your project id is set up

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
PROJECT_ID=[YOUR-PROJECT-NAME]
echo $PROJECT_ID

Enable APIs

Enable all necessary services

gcloud services enable compute.googleapis.com
gcloud services enable logging.googleapis.com
gcloud services enable monitoring.googleapis.com

3. Create a Cloud Storage Bucket

In the Cloud Console, go to Navigation menu > Cloud Storage > Browser. Click CREATE BUCKET:

Bucket naming rules:

• Do not include sensitive information in the bucket name, because the bucket namespace is global and publicly visible.
• Bucket names must contain only lowercase letters, numbers, dashes (-), underscores (_), and dots (.). Names containing dots require verification.
• Bucket names must start and end with a number or letter.
• Bucket names must contain 3 to 63 characters. Names containing dots can contain up to 222 characters, but each dot-separated component can be no longer than 63 characters.
• Bucket names cannot be represented as an IP address in dotted-decimal notation (for example, 192.168.5.4).
• Bucket names cannot begin with the "goog" prefix.
• Bucket names cannot contain "google" or close misspellings of "google".
• Also, for DNS compliance and future compatibility, you should not use underscores (_) or have a period adjacent to another period or dash. For example, ".." or "-." or ".-" are not valid in DNS names.

Click CONTINUE.

Location type: region

Location: choose a region far away from you

Click CONTINUE.

Default Storage class: Standard

Click CONTINUE.

Uncheck Enforce public access prevention on this bucket checkbox under Prevent public access.

Choose Fine Grained under Access Control.

Click CONTINUE.

Once you've gotten your bucket configured, click CREATE:

That's it — you've just created a Cloud Storage bucket!

4. Upload an Object to your bucket

Now upload an object into a bucket.

First, download this image to a temporary instance in Cloud Shell. In the example below the Google image from the Google homepage is used.

From Cloud Shell

wget --output-document google.png https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png

Use the gsutil cp command to upload the image from the location where you saved it to the bucket you created:

From Cloud Shell

gsutil cp google.png gs://YOUR-BUCKET-NAME

Now remove the downloaded image:

From Cloud Shell

rm google.png

Locate your Object and click on the three dots on the right side and edit the permissions to be public.

5. Create a Load Balancer

Next we will create an HTTP Load balancer.

Navigate to Networking >> Network services >> Load Balancing >> Create Load Balancer >> HTTP Load Balancer >> Internet to my VMs >> Classic HTTP(S) Load Balancer

First, create a name for the load balancer you will create.

Create backend bucket

Create a new backend bucket that is your bucket you just created and select Enabled CDN and Cache Static Content. Click Create.

Create host and path rules

Navigate to Host and path rules on the left hand side. We will use a simple host/path rule and send any request to the bucket.

Create frontend configuration

Select the frontend configuration. For the frontend configuration we will use HTTP (though HTTPS will also work if you have a certificate) and an ephemeral IP address and ensure that you have selected the premium tier network.

Click Create

Get Load Balancer IP

Get the Load Balancer IP from the console by clicking on the load balancer name in the list of load balancers for your project.

Query the Load Balancer

After a couple of minutes, attempt to query the load balancer for the object you uploaded. You will need the load balancer IP address and the name of the image. The command is structure as so:

From Terminal

curl -svo /dev/null http://LOAD_BALANCER_IP/google.png

Output

armanrye-macbookpro% curl -svo /dev/null http://34.98.81.123/google.png
*   Trying 34.98.81.123...
* TCP_NODELAY set
* Connected to 34.98.81.123 (34.98.81.123) port 80 (#0)
> GET /google.png HTTP/1.1
> Host: YOUR_IP
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-GUploader-UploadID: ADPycdtoILI76KVsvBvdVGvSfzaxys1m3zYqCepBrmJxAI48ni24cWCRIdNu-53PX3DS6iycxp6xwFbMpwtcHHZQUQmEBxAgng
< Expires: Mon, 13 Dec 2021 22:58:26 GMT
< Date: Mon, 13 Dec 2021 21:58:26 GMT
< Cache-Control: public, max-age=3600
< Last-Modified: Mon, 13 Dec 2021 21:45:57 GMT
< ETag: "8f9327db2597fa57d2f42b4a6c5a9855"
< x-goog-generation: 1639431957957903
< x-goog-metageneration: 2
< x-goog-stored-content-encoding: identity
< x-goog-stored-content-length: 5969
< Content-Type: image/png
< x-goog-hash: crc32c=TeiHTA==
< x-goog-hash: md5=j5Mn2yWX+lfS9CtKbFqYVQ==
< x-goog-storage-class: STANDARD
< Accept-Ranges: bytes
< Content-Length: 5969
< Server: UploadServer

Validate that your content is being served from the CDN via CDN or Load Balancing Monitoring. You should be able to get close to a 100% hit ratio. If you need to run a few queries you can do the following

From Terminal

#for i in `seq 1 50`; do curl http://%loadbalncer-IP%/google.png; done

Confirm content served by Cloud CDN

Validate that you are serving traffic from CDN by navigating to Network Services >> CDN

6. Delete the Object from GCS

Now that the cache is populated let's delete the object from the bucket, reinforcing that we are applying the policy to the cache and not the backend.

Navigate to Cloud Storage >> %bucket name% >> objects

Select the image and delete it.

7. Create Edge Security Policy for your CDN Cache

Navigate to Network Security >> Cloud Armor and click create policy

Select the following

Policy Type: Edge Security Policy

Default Action: Deny

Apply the policy to your backend bucket.

8. Validate Edge Security Policy

Now that we have created an Edge Security Policy in front of our back-end bucket, lets validate that it is working as expected.

Check the security policy

After a few minutes have passed, you will be able to check that the Cloud Armor Policy is running. From the command line, running the following command will give you a 403

From Terminal

curl -svo /dev/null http://LOAD_BALANCER_IP/google.png

Output

curl -svo /dev/null http://34.98.81.123/google.png
*   Trying 34.98.81.123...
* TCP_NODELAY set
* Connected to 34.98.81.123 (34.98.81.123) port 80 (#0)
> GET /google.png HTTP/1.1
> Host: YOUR_IP
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< X-GUploader-UploadID: ADPycdtS6FtJOGIsiWYDrAAE8VFeQuNutcvbGoQe2t8EZxsuspVtmCjyiTv_P3CNktroHMOGFXkTCfG-Jj-rUO60ZGPpEbpqcw
< Content-Type: application/xml; charset=UTF-8
< Content-Length: 111
< Date: Mon, 13 Dec 2021 23:09:35 GMT
< Expires: Mon, 13 Dec 2021 23:09:35 GMT
< Cache-Control: private, max-age=0
< Server: UploadServer
<

Investigate the logs

Next, you can check the logs to see the enforced edge security policy. Navigate to Operations >> Logging >> Logs Explorer

Enter the below snippet into the query and click run

resource.type:(http_load_balancer) AND jsonPayload.enforcedEdgeSecurityPolicy.name:(%SECURITY_POLICY_NAME%)

Note the 403 response and the enforced security policy

Remove the security policy

Remove the security policy and query the object from cache.

Navigate to Network Security >> Cloud Armor >> %POLICY NAME% >> targets and remove the target bucket.

Confirm policy removed

After a few minutes have passed, send another curl to the resource in the storage bucket. You will get a 200 response this time.

From Terminal

curl -svo /dev/null http://LOAD_BALANCER_IP/google.png

Output

armanrye-macbookpro% curl -svo /dev/null http://34.98.81.123/google.png

*   Trying 34.98.81.123...
* TCP_NODELAY set
* Connected to 34.98.81.123 (34.98.81.123) port 80 (#0)
> GET /google.png HTTP/1.1
> Host: YOUR_IP
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< X-GUploader-UploadID: ADPycdtI7f49P3MSuZSZ8vl6RwfwmnIDJ59EeSKp7UPvLPawdaiRHXiNWLtseQTxUxceWOvSLvpYmT3pWVkV4qeIP7M
< Date: Mon, 13 Dec 2021 23:06:46 GMT
< Last-Modified: Mon, 13 Dec 2021 21:45:57 GMT
< ETag: "8f9327db2597fa57d2f42b4a6c5a9855"
< x-goog-generation: 1639431957957903
< x-goog-metageneration: 2
< x-goog-stored-content-encoding: identity
< x-goog-stored-content-length: 5969
< Content-Type: image/png
< x-goog-hash: crc32c=TeiHTA==
< x-goog-hash: md5=j5Mn2yWX+lfS9CtKbFqYVQ==
< x-goog-storage-class: STANDARD
< Accept-Ranges: bytes
< Content-Length: 5969
< Server: UploadServer
< Age: 1621
< Cache-Control: public,max-age=3600
<
{ [775 bytes data]
* Connection #0 to host 34.98.81.123 left intact
* Closing connection 0

Congratulations! You have completed this lab on Cloud Armor Edge Security Policies.

©2020 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.

9. Lab Clean up

Navigate to Network Security >> Cloud Armor >> %POLICY NAME% and select delete

Navigate to Cloud Storage select the bucket you created and click delete

Navigate to Networking >> Network services >> Load Balancing. Select the load balancer you created and click delete.

10. Congratulations!

You have completed the Defending Edge Cache with Cloud Armor codelab!

What we've covered

• How to set up a cloud storage bucket and associated cloud load balancer
• How to create a Cloud Armor Edge Security Policy
• How to validate that the Edge Security Policy is working as intended.

Next steps

• Try setting up an Edge Security Policy with a cache hit from a compute engine resource.