Lab: Service Extensions on Media CDN

Lab:
Service Extensions on Media CDN

About this codelab

subjectLast updated Jul 8, 2024
account_circleWritten by Xiaozang Li

1. Introduction

Last Updated: 2024-05-01

Content Delivery Networks (CDNs) improve user performance by caching frequently accessed content closer to the end users, terminating connections closer to the clients, re-using connections to the origin, and through adoption of modern networking protocols and customizations.

Media CDN, GCP's global edge network for streaming media, provides many built-in or "core" capabilities.The core capabilities are intended to address the most common use cases, but you may also have requirements that aren't addressed by this core feature set.

Service Extensions for Media CDN, sometimes also referred as Edge Programmability, allows you to run your own code at the edge to customize the behavior of Media CDN. This unlocks additional use cases ranging from normalizing cache key, custom token authentication and token revocation, additional custom log fields, A/B testing, and custom error page.

What you'll build

In this code lab, we will walk through the steps to deploy an Edge Compute-enabled CDN delivery environment with Media CDN (CDN) + Service Extensions (Edge Programmability) + Cloud Storage (source of CDN).

1f19151bdd96acb0.png

What you'll learn

  • How to set up Media CDN with a Cloud Storage Bucket set as Origin
  • How to create a Service Extension plugin with custom HTTP authentication and associate it with Media CDN
  • How to validate that the Service Extension plugin is working as expected
  • (optional) How to manage a Service Extension plugin like updating, referencing, rolling back, and deleting a specific plugin version

What you'll need

  • Basic Networking and knowledge of HTTP
  • Basic Unix/Linux command line knowledge

2. Before you begin

Request for Media CDN Allowlist & Service Extensions Allowlist

Before getting started, you will need to ensure your project has been added to the private preview allowlist for both Media CDN and Service Extensions for Media CDN.

  • To request access to both Media CDN and Service Extensions for Media CDN, please contact your Google Account Team to create an access request on your behalf for Media CDN and Service Extensions

3. Setup and Requirements

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

1dec6f9683153af0.png

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

de496bb88f9a0b10.png

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

Before you begin

IAM Roles & Access

The Identity and Access Management (IAM) permissions required to create Media CDN and Artifact Registry resources are the following:

  • roles/networkservices.edgeCacheAdmin
  • roles/networkservices.edgeCacheUser
  • roles/networkservices.edgeCacheViewer
  • roles/artifactregistry.repoAdmin

Inside Cloud Shell, make sure that your project_id, project_num, location, and repository environment variables are set up.

gcloud config list project
gcloud config set project [YOUR-PROJECT-NAME]
PROJECT_ID=[YOUR-PROJECT-NAME]
PROJECT_NUM=[YOUR-PROJECT-NUMBER]
LOCATION=us-central1
REPOSITORY=service-extension-$PROJECT_ID

Enable APIs

Enable Media CDN & Service Extensions APIs through the commands below

gcloud services enable networkservices.googleapis.com
gcloud services enable networkactions.googleapis.com
gcloud services enable edgecache.googleapis.com
gcloud services enable artifactregistry.googleapis.com

4. Create a Cloud Storage Bucket

Media CDN content can originate from locations such as a Cloud Storage bucket, a third-party storage location, or any public accessible HTTP(HTTPS) endpoint.

In this codelab, we'll store content in a Cloud Storage bucket.

We will use the gsutil mb command to create the bucket

gsutil mb gs://mediacdn-bucket-$PROJECT_ID

Optionally, you can create a Cloud Storage bucket using the GUI as the following:

  1. In the Google Cloud console, go to the Cloud Storage page.
  2. Click the CREATE button.
  3. Enter a name for the bucket. - i.e."mediacdn-bucket-$PROJECT_ID".
  4. Leave the rest settings as default.
  5. Click the CREATE Button.

50475e01c5a3adbe.png

5. Upload a test object to the Cloud Storage Bucket

Now we will upload an object into the Cloud Storage bucket.

  1. Create a file in cloud shell then upload it into the bucket using gsutil
echo media-cdn-service-extensions-test > file.txt

gsutil cp file.txt gs://mediacdn-bucket-$PROJECT_ID
  1. Grant Media CDN access to the bucket
gsutil iam ch \
serviceAccount:service-$PROJECT_NUM@gcp-sa-mediaedgefill.iam.gserviceaccount.com:objectViewer gs://mediacdn-bucket-$PROJECT_ID

6. Configure Media CDN

Next we will create a Media CDN configuration.

Each Media CDN configuration consists of two main resources:

  • EdgeCacheService, responsible for client-facing configuration (TLS, IP addressing), routing, CDN configuration (cache modes, TTLs, signing), and security policies.
  • EdgeCacheOrigin, responsible for per-origin configuration for any HTTP-based origin, as well as retry conditions when content is not available or reachable.

Configure an Edge Cache Origin

Now Let's create an origin that points to the Cloud Storage bucket you just created.

  1. In the Google Cloud console, go to the Media CDN page.
  2. Click the ORIGINS tab.
  3. Click CREATE ORIGIN.
  4. Enter ‘cloud-storage-origin' as the name for the edge cache origin.
  5. Under Origin address:
  6. choose ‘Select a Google Cloud Storage bucket'.
  7. BROWSE to the Cloud Storage bucket named ‘mediacdn-bucket-$PROJECT_ID'.
  8. click SELECT.
  9. Leave the rest settings as default.
  10. Click CREATE ORIGIN.

e6eb0faa94838c4.png

The newly created EdgeCacheOrigin resource appears in the list of origins in your project on the Origins page.

Configure an Edge Cache Service

  1. In the Google Cloud console, go to the Media CDN page.
  2. Click the SERVICES tab.
  3. Click CREATE SERVICE.
  4. Enter a unique name for your service - i.e. ‘media-cdn' - and then click Next.

d2f9ac837bc5d45a.png

  1. In the Routing section, click ADD HOST RULE.
  2. Enter wildcard - "*" in the Hosts field.

25d3e25000934e59.png

  1. Click ADD ROUTE RULE.
  2. For Priority, specify "1".
  3. Click ADD A MATCH CONDITION, for Path match, select "Prefix match" as Match type, specify "/" in Path match field, and then click Done.
  4. Select Fetch from an Origin under Primary action, and then select the origin that you configured in the drop down list.

d1975f366233521a.png

  1. Click ADVANCED CONFIGURATIONS to extend more configuration options.
  2. In Route action, click ADD AN ITEM. Then, do the following:
  3. For Type, select "CDN policy".
  4. For Cache mode, select "Force cache all".
  5. Leave the rest as default
  6. Click Done.
  7. Click Save.

b7e77d059db84ab6.png

  1. Click CREATE SERVICE.

The newly created EdgeCacheService resource appears on the Services page in the list of services in your project.

Retrieve the MediaCDN IP address and Testing

  1. In the Google Cloud console, go to the Media CDN page.
  2. Go to Media CDN
  3. Click the Services tab.
  4. For your service, see the Addresses column.

4031b6d1eac89041.png

To test that your service is correctly configured to cache content, use the curl command-line tool to issue requests and check the responses.

curl -svo /dev/null "http://MEDIA_CDN_IP_ADDRESS/file.txt"

The command should produces output similar to the following:

< HTTP/2 200 OK
...
media-cdn-service-extensions-test
...

Now you have successfully created a MediaCDN deployment with Cloud Storage as Origin.

7. Configure Artifact Registry for Service Extensions

Before creating a Service Extensions, we need to configure the Artifact Registry. Artifact Registry is Google Cloud's universal package manager for managing build artifacts. Service Extension (Proxy-Wasm) plugins are published to Artifact Registry. Once published to Artifact Registry, Proxy-Wasm plugins can be deployed to your Media CDN deployment.

We will use the gcloud artifacts repositories create command to create the repository

gcloud artifacts repositories create service-extension-$PROJECT_ID \
    --repository-format=docker \
    --location=$LOCATION \
    --description="Repo for Service Extension" \
    --async

Optionally, you can create a Repository using the GUI as the following:

  1. In the Google Cloud console, go to the Artifact Registry page.
  2. Click the + CREATE REPOSITORY button.
  3. Enter a Name for the repository. i.e. ‘service-extension-$PROJECT_ID'.
  4. Format - ‘Docker,' Mode - ‘Standard', Location Type - ‘Region', and pick ‘us-central1 (Iowa)'
  5. Click the CREATE button.

b525b3bc0867dc42.png

The newly created Artifact Registry Repository resource should appear on the Repositories page.

Once the Repository resource has been created, run the following command in Cloud Shell to configure your Cloud Shell docker client to push and pull packages using this repository.

gcloud auth configure-docker $LOCATION-docker.pkg.dev

Output:

...
Adding credentials for: us-central1-docker.pkg.dev
Docker configuration file updated.

8. Configure Service Extensions on Media CDN

Now, we will demonstrate how to write and build a Service Extension (Proxy-Wasm) plugin that can be deployed to Media CDN, using the Rust programming language.

In this example, we will create a Proxy-Wasm plugin that verifies each HTTP request contains an Authorization header with the value "secret". If the request does not contain this header, the plugin will generate an HTTP 403 Forbidden response.

A quick refresher on Service Extensions- there are three key resources: WasmAction, WasmPlugin, and WasmPluginVersion.

  • A WasmAction resource is what gets attached to your Media CDN EdgeCacheService. A WasmAction references a WasmPlugin resource.
  • A WasmPlugin resource has a main-version which corresponds to the current active WasmPluginVersion.
  • A WasmPluginVersions reference a container image from Artifact Registry. As you make changes to your proxy-wasm plugins, you create different WasmPluginVersions.

Please reference the below diagram to better understand the relationship between these resources.

22b3548b3a61c379.png

Write and build a Service Extension plugin

  1. Install the Rust toolchain by following the instructions at https://www.rust-lang.org/tools/install.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
  1. Next, add Wasm support to your Rust toolchain by running the following command:
rustup target add wasm32-wasi
  1. Create a Rust package called my-wasm-plugin:
cargo new --lib my-wasm-plugin

Output:

Created library `my-wasm-plugin` package
  1. Enter the directory my-wasm-plugin and you should see a Cargo.toml file, and a src directory.
cd my-wasm-plugin
ls

Output:

Cargo.toml  src
  1. Next, configure your Rust package by editing the Cargo.toml file. After the [dependencies] line in the Cargo.toml file, add the following:
proxy-wasm = "0.2"
log = "0.4"

[lib]
crate-type = ["cdylib"]

[profile.release]
lto = true
opt-level = 3
codegen-units = 1
panic = "abort"
strip = "debuginfo"
  1. After your edits, the Cargo.toml file should look roughly like so:
[package]
name = "my-wasm-plugin"
version = "0.1.0"
edition = "2021"

[dependencies]
proxy-wasm = "0.2"
log = "0.4"

[lib]
crate-type = ["cdylib"]

[profile.release]
lto = true
opt-level = 3
codegen-units = 1
panic = "abort"
strip = "debuginfo"
  1. . Copy over the full content of the sample_code file to the lib.rs file in the src directory in Cloud Shell.
  1. After your edits, the lib.rs file should look roughly like so:
use log::info;
use proxy_wasm::traits::*;
use proxy_wasm::types::*;

...

struct DemoPlugin;

impl HttpContext for DemoPlugin {
    fn on_http_request_headers(&mut self, _: usize, _: bool) -> Action {
        if self.get_http_request_header("Authorization") == Some(String::from("secret")) {
            info!("Access granted.");
            Action::Continue
        } else {
            self.send_http_response(403, vec![], Some(b"Access forbidden.\n"));
            Action::Pause
        }
    }
}

impl Context for DemoPlugin {}
  1. Now that we have configured the Cargo.toml manifest file and written our Proxy-Wasm code in lib.rs file , we can build our Proxy-Wasm plugin.
cargo build --release --target wasm32-wasi

Once the build completes successfully, you will see a message as shown below:

Finished release [optimized] target(s) in 1.01s

Let's also verify the target directory and check the files been created:

ls ./target

You will see the output as shown below:

CACHEDIR.TAG release wasm32-wasi

Publish a Proxy-Wasm plugin to Artifact Registry

Now, we will publish our Proxy-Wasm plugin to the Artifact Registry Repository you created before so it can be deployed to Media CDN.

We first package the Proxy-Wasm plugins in a container image.

  1. Create a file called Dockerfile in the same directory my-wasm-plugin, with the following contents:
FROM scratch 
COPY target
/wasm32-wasi/release/my_wasm_plugin.wasm plugin.wasm
  1. Next, build the container image:
docker build --no-cache --platform wasm -t my-wasm-plugin .

(non-x86 processors only) Next, build the container image:

docker build --no-cache --platform wasm --provenance=false -t my-wasm-plugin . 

Output

[+] Building 0.2s (5/5) FINISHED                                     docker:default
...
  1. Next, publish or "push" your Proxy-Wasm plugin to Artifact Registry. We will tag our container image with the ‘prod' tag.
docker tag my-wasm-plugin $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin:prod

Now we go ahead and push the tagged ‘prod' container image to the repository.

docker push $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin:prod

Output:

The push refers to repository 
...
8564ddd9910a: Pushed 
prod: digest: sha256:f3ae4e392eb45393bfd9c200cf8c0c261762f7f39dde5c7cd4b9a8951c6f2812 size: 525

Now Let's verify the container image of Proxy-Wasm plugin was successfully pushed to Artifact Registry, you should see a similar output:

gcloud artifacts docker images list $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin --include-tags

Output:

Listing items under project 
...
IMAGE                                         DIGEST           TAGS  CREATE_TIME          UPDATE_TIME
<LOCATION>-docker.pkg.dev/.../my-wasm-plugin  sha256:08c12...  prod  2021-11-10T23:31:27  2021-11-10T23:31:27

Associate a Proxy-Wasm plugin with your Media CDN deployment

Now we are ready to associate the Proxy-Wasm plugin to your Media CDN deployment.

Proxy-Wasm plugins are associated with Media CDN routes in the EdgeCacheService resource.

  1. First, we create a Wasm-plugin resource for our Proxy-Wasm plugin.
gcloud alpha service-extensions wasm-plugins create my-wasm-plugin-resource
  1. Next, we create a WasmPluginVersion.
gcloud alpha service-extensions wasm-plugin-versions create my-version-1 \
    --wasm-plugin=my-wasm-plugin-resource \
    --image="$LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin:prod"
  1. Next, we specify the main version for our Proxy-Wasm plugin.
gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource \
    --main-version=my-version-1

Now Let's verify the Proxy-Wasm plugin was successfully associated to Container Image resides in Artifact Registry Repository, you should see a similar output:

gcloud alpha service-extensions wasm-plugin-versions list --wasm-plugin=my-wasm-plugin-resource

Output:

NAME   WASM_IMAGE WASM_IMAGE_DIGEST CONFIG_SIZE  CONFIG_IMAGE CONFIG_IMAGE_DIGEST UPDATE_TIME                                            
c7cfa2 <LOCATION>-docker.pkg.dev/.../my-wasm-plugin@sha256:6d663... ... ... 
...
  1. Next, we create a WasmAction resource referencing our Wasm plugin resource.
gcloud alpha service-extensions wasm-actions create my-wasm-action-resource \
    --wasm-plugin=my-wasm-plugin-resource

Let's also verify the WasmAction resource was successfully associated to the Proxy-Wasm plugin, you should see a similar output:

gcloud alpha service-extensions wasm-actions list

Output:

NAME                                     WASMPLUGIN                                            
my-wasm-action-resource                  projects/805782461588/locations/global/wasmPlugins/myenvoyfilter-resource
...
  1. Now, we need to export the configuration of our Media CDN EdgeCacheService:
gcloud edge-cache services export media-cdn --destination=my-service.yaml
  1. Then, open the my-service.yaml file and add a wasmAction to the routeAction for the given route, which references the WasmPlugin resource created earlier.
wasmAction: "my-wasm-action-resource"
  1. After your edits, the my-service.yaml file should look roughly like so:
...

pathMatchers
:
 
- name: routes
    routeRules
:
   
- headerAction: {}
      matchRules
:
     
- prefixMatch: /
      origin: projects/
<PROJECT_NUM>/locations/global/edgeCacheOrigins/cloud-storage-origin
      priority
: '1'
      routeAction
:
        cdnPolicy
:
          cacheKeyPolicy
: {}
          cacheMode
: FORCE_CACHE_ALL
          defaultTtl
: 3600s
          signedRequestMode
: DISABLED
        wasmAction
: "my-wasm-action-resource"
...
  1. Then, we save the updated configuration with Proxy-Wasm configuration to the my-service-with-wasm.yaml file.
  1. Finally, we import the updated configuration for the production Media CDN environment:
$ gcloud alpha edge-cache services import media-cdn --source=my-service-with-wasm.yaml

9. Validate Service Extensions Proxy-Wasm plugin on Media CDN

To test that your service is correctly configured to cache content, use the curl command-line tool to issue requests and check the responses.

curl -svo /dev/null "http://IP_ADDRESS/file.txt"

The command should produces output similar to the following:

< HTTP/2 403 Forbidden
...
Access forbidden.
...

Now, issue the request again with an Authorization header and its value of secret

curl -svo /dev/null "http://IP_ADDRESS/file.txt" -H "Authorization: secret"

The command should produces output similar to the following:

< HTTP/2 200 OK
...
media-cdn-service-extensions-test
...

10. Optional: Managing Proxy-Wasm plugins

Updating a Proxy-Wasm plugin

As you make improvements or add new functionality to your Proxy-Wasm plugins, you will need to deploy your updated plugins to Media CDN. Below, we go through the steps to deploy an updated version of a plugin.

As an example, you might update the sample plugin code to evaluate the Authorization header against another value for authentication, by modifying the code like the following.

First, update the src/lib.rs source file with the code shown below:

use log::{info, warn};
use proxy_wasm::traits::*;
use proxy_wasm::types::*;

...

struct DemoPlugin;

impl HttpContext for DemoPlugin {
    fn on_http_request_headers(&mut self, _: usize, _: bool) -> Action {
        if self.get_http_request_header("Authorization") == Some(String::from("another_secret")) {
            info!("Access granted.");
            Action::Continue
        } else {
            warn!("Access forbidden.");
            self.send_http_response(403, vec![], Some(b"Access forbidden.\n"));
            Action::Pause
        }
    }
}

impl Context for DemoPlugin {}

Next, build, package, and publish the updated plugin:

cargo build --release --target wasm32-wasi
docker build --no-cache --platform wasm -t my-wasm-plugin .
docker tag my-wasm-plugin $LOCATION-docker.pkg.dev/$PROJECT_NUM/$REPOSITORY/my-wasm-plugin:prod
docker push $LOCATION-docker.pkg.dev/$PROJECT_NUM/$REPOSITORY>/my-wasm-plugin:prod

Once the container image is updated in Artifact Registry, we need to create a new WasmPluginVersion and then update the –main-version of the WasmPlugin to reference the new version.

gcloud alpha service-extensions wasm-plugin-versions create my-version-2 \
    --wasm-plugin=my-wasm-plugin-resource \
   --image="$LOCATION-docker.pkg.dev/$PROJECT_NUM/$REPOSITORY>/my-wasm-plugin:prod"
gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource \
    --main-version=my-version-2

Now, you have successfully updated the version of the container image to be imported from Artifact Registry and pushed live to your Media CDN deployment.

Rolling back to a previous version

To roll back to a previous version of a plugin, you can update the Wasm plugin resource to reference a previous version.

First, we list the available versions:

gcloud alpha service-extensions wasm-plugin-versions list --wasm-plugin=my-wasm-plugin-resource

You should see the output:

NAME   WASM_IMAGE WASM_IMAGE_DIGEST CONFIG_SIZE  CONFIG_IMAGE CONFIG_IMAGE_DIGEST UPDATE_TIME                                            
c7cfa2 <LOCATION>-docker.pkg.dev/.../my-wasm-plugin@sha256:6d663... ... ... 
a2a8ce <LOCATION>-docker.pkg.dev/.../my-wasm-plugin@sha256:08c12... ... ... 

Next, we update the Wasm plugin resource to reference the previous version, "a2a8ce":

$ gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource \
    --main-version="a2a8ce"

Once the operation is successful, you should see this output:

✓ WASM Plugin [my-wasm-plugin-resource] is now serving version "a2a8ce"

Since Media CDN saves the image digest of your Docker image each time a new Wasm-plugin resource is created, the rollback will use the version of your code that was running before the last rollout.

gcloud alpha service-extensions wasm-plugins describe my-wasm-plugin-resource \
  --expand-config

For version "a2a8ce", that is the version with digest sha256:08c12...:

name: "my-wasm-plugin-resource"
mainVersion: "a2a8ce"
mainVersionDetails:
  image: "<LOCATION>-docker.pkg.dev/<PROJECT>/<REPOSITORY>/my-wasm-plugin"
  imageDigest: "<LOCATION>-docker.pkg.dev/<PROJECT>/<REPOSITORY>/my-wasm-plugin@sha256:08c121dd7fd1e4d3a116a28300e9fc1fa41b2e9775620ebf3d96cb7119bd9976"

Deleting a WasmAction & WasmPlugin

To delete a WasmAction, WasmPlugin, and the associated WasmPluginVersions, please follow these steps.

First, remove the reference to the WasmAction in your Media CDN EdgeCacheService config.

Reference line to be removed:

wasmAction: "my-wasm-action-resource"

Then, we update the edited EdgeCacheService config.

gcloud alpha edge-cache services import prod-media-service --source=my-service.yaml

Next, update the main-version of your WasmPlugin to an empty string "".

gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource --main-version=
""

Finally, perform the below deletion steps in order.

gcloud alpha service-extensions wasm-actions delete my-wasm-action-resource

gcloud alpha service-extensions wasm-plugin-versions delete my-version \ --wasm-plugin=my-wasm-plugin-resource

gcloud alpha service-extensions wasm-plugins delete my-wasm-plugin-resource

11. Clean up the Lab environment

After you have completed the CodeLab, don't forget to clean up the lab resources - otherwise they'll keep running and accruing costs.

The following commands will delete the Media CDN EdgeCache Service, EdgeCache Config, and the Service Extensions Plugin. Perform the below deletion steps in order.

gcloud edge-cache services delete media-cdn

gcloud edge-cache origins delete cloud-storage-origin

gcloud alpha service-extensions wasm-actions delete my-wasm-action-resource

gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource --main-version=""

gcloud alpha service-extensions wasm-plugin-versions delete my-version-1 --wasm-plugin=my-wasm-plugin-resource

gcloud alpha service-extensions wasm-plugins delete my-wasm-plugin-resource

gcloud artifacts repositories delete service-extension-$PROJECT_ID --location=$LOCATION

Each of the commands above should ask you to confirm the deletion of the resource.

12. Congratulations!

Congratulations, you've completed the Service Extensions on Media CDN codelab!

What we've covered

  • How to set up Media CDN with a Cloud Storage Bucket set as Origin
  • How to create a Service Extension plugin with custom HTTP authentication and associate it with Media CDN
  • How to validate that the Service Extension plugin is working as expected
  • (optional) How to manage a Service Extension plugin like updating, referencing, rolling back, and deleting a specific plugin version

What's next?

Check out some of these codelabs...

Further reading

Reference docs