About this codelab
1. Introduction
Last Updated: 2024-05-01
Content Delivery Networks (CDNs) improve user performance by caching frequently accessed content closer to the end users, terminating connections closer to the clients, re-using connections to the origin, and through adoption of modern networking protocols and customizations.
Media CDN, GCP's global edge network for streaming media, provides many built-in or "core" capabilities.The core capabilities are intended to address the most common use cases, but you may also have requirements that aren't addressed by this core feature set.
Service Extensions for Media CDN, sometimes also referred as Edge Programmability, allows you to run your own code at the edge to customize the behavior of Media CDN. This unlocks additional use cases ranging from normalizing cache key, custom token authentication and token revocation, additional custom log fields, A/B testing, and custom error page.
What you'll build
In this code lab, we will walk through the steps to deploy an Edge Compute-enabled CDN delivery environment with Media CDN (CDN) + Service Extensions (Edge Programmability) + Cloud Storage (source of CDN).
What you'll learn
- How to set up Media CDN with a Cloud Storage Bucket set as Origin
- How to create a Service Extension plugin with custom HTTP authentication and associate it with Media CDN
- How to validate that the Service Extension plugin is working as expected
- (optional) How to manage a Service Extension plugin like updating, referencing, rolling back, and deleting a specific plugin version
What you'll need
- Basic Networking and knowledge of HTTP
- Basic Unix/Linux command line knowledge
2. Before you begin
Request for Media CDN Allowlist & Service Extensions Allowlist
Before getting started, you will need to ensure your project has been added to the private preview allowlist for both Media CDN and Service Extensions for Media CDN.
- To request access to both Media CDN and Service Extensions for Media CDN, please contact your Google Account Team to create an access request on your behalf for Media CDN and Service Extensions
3. Setup and Requirements
Start Cloud Shell
While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.
From the GCP Console click the Cloud Shell icon on the top right toolbar:
It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:
This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.
Before you begin
IAM Roles & Access
The Identity and Access Management (IAM) permissions required to create Media CDN and Artifact Registry resources are the following:
- roles/networkservices.edgeCacheAdmin
- roles/networkservices.edgeCacheUser
- roles/networkservices.edgeCacheViewer
- roles/artifactregistry.repoAdmin
Inside Cloud Shell, make sure that your project_id, project_num, location, and repository environment variables are set up.
gcloud config list project gcloud config set project [YOUR-PROJECT-NAME] PROJECT_ID=[YOUR-PROJECT-NAME] PROJECT_NUM=[YOUR-PROJECT-NUMBER] LOCATION=us-central1 REPOSITORY=service-extension-$PROJECT_ID
Enable APIs
Enable Media CDN & Service Extensions APIs through the commands below
gcloud services enable networkservices.googleapis.com gcloud services enable networkactions.googleapis.com gcloud services enable edgecache.googleapis.com gcloud services enable artifactregistry.googleapis.com
4. Create a Cloud Storage Bucket
Media CDN content can originate from locations such as a Cloud Storage bucket, a third-party storage location, or any public accessible HTTP(HTTPS) endpoint.
In this codelab, we'll store content in a Cloud Storage bucket.
We will use the gsutil mb command to create the bucket
gsutil mb gs://mediacdn-bucket-$PROJECT_ID
Optionally, you can create a Cloud Storage bucket using the GUI as the following:
- In the Google Cloud console, go to the Cloud Storage page.
- Click the CREATE button.
- Enter a name for the bucket. - i.e."mediacdn-bucket-$PROJECT_ID".
- Leave the rest settings as default.
- Click the CREATE Button.
5. Upload a test object to the Cloud Storage Bucket
Now we will upload an object into the Cloud Storage bucket.
- Create a file in cloud shell then upload it into the bucket using gsutil
echo media-cdn-service-extensions-test > file.txt gsutil cp file.txt gs://mediacdn-bucket-$PROJECT_ID
- Grant Media CDN access to the bucket
gsutil iam ch \ serviceAccount:service-$PROJECT_NUM@gcp-sa-mediaedgefill.iam.gserviceaccount.com:objectViewer gs://mediacdn-bucket-$PROJECT_ID
6. Configure Media CDN
Next we will create a Media CDN configuration.
Each Media CDN configuration consists of two main resources:
- EdgeCacheService, responsible for client-facing configuration (TLS, IP addressing), routing, CDN configuration (cache modes, TTLs, signing), and security policies.
- EdgeCacheOrigin, responsible for per-origin configuration for any HTTP-based origin, as well as retry conditions when content is not available or reachable.
Configure an Edge Cache Origin
Now Let's create an origin that points to the Cloud Storage bucket you just created.
- In the Google Cloud console, go to the Media CDN page.
- Click the ORIGINS tab.
- Click CREATE ORIGIN.
- Enter ‘cloud-storage-origin' as the name for the edge cache origin.
- Under Origin address:
- choose ‘Select a Google Cloud Storage bucket'.
- BROWSE to the Cloud Storage bucket named ‘mediacdn-bucket-$PROJECT_ID'.
- click SELECT.
- Leave the rest settings as default.
- Click CREATE ORIGIN.
The newly created EdgeCacheOrigin resource appears in the list of origins in your project on the Origins page.
Configure an Edge Cache Service
- In the Google Cloud console, go to the Media CDN page.
- Click the SERVICES tab.
- Click CREATE SERVICE.
- Enter a unique name for your service - i.e. ‘media-cdn' - and then click Next.
- In the Routing section, click ADD HOST RULE.
- Enter wildcard - "*" in the Hosts field.
- Click ADD ROUTE RULE.
- For Priority, specify "1".
- Click ADD A MATCH CONDITION, for Path match, select "Prefix match" as Match type, specify "/" in Path match field, and then click Done.
- Select Fetch from an Origin under Primary action, and then select the origin that you configured in the drop down list.
- Click ADVANCED CONFIGURATIONS to extend more configuration options.
- In Route action, click ADD AN ITEM. Then, do the following:
- For Type, select "CDN policy".
- For Cache mode, select "Force cache all".
- Leave the rest as default
- Click Done.
- Click Save.
- Click CREATE SERVICE.
The newly created EdgeCacheService resource appears on the Services page in the list of services in your project.
Retrieve the MediaCDN IP address and Testing
- In the Google Cloud console, go to the Media CDN page.
- Go to Media CDN
- Click the Services tab.
- For your service, see the Addresses column.
To test that your service is correctly configured to cache content, use the curl command-line tool to issue requests and check the responses.
curl -svo /dev/null "http://MEDIA_CDN_IP_ADDRESS/file.txt"
The command should produces output similar to the following:
< HTTP/2 200 OK ... media-cdn-service-extensions-test ...
Now you have successfully created a MediaCDN deployment with Cloud Storage as Origin.
7. Configure Artifact Registry for Service Extensions
Before creating a Service Extensions, we need to configure the Artifact Registry. Artifact Registry is Google Cloud's universal package manager for managing build artifacts. Service Extension (Proxy-Wasm) plugins are published to Artifact Registry. Once published to Artifact Registry, Proxy-Wasm plugins can be deployed to your Media CDN deployment.
We will use the gcloud artifacts repositories create command to create the repository
gcloud artifacts repositories create service-extension-$PROJECT_ID \ --repository-format=docker \ --location=$LOCATION \ --description="Repo for Service Extension" \ --async
Optionally, you can create a Repository using the GUI as the following:
- In the Google Cloud console, go to the Artifact Registry page.
- Click the + CREATE REPOSITORY button.
- Enter a Name for the repository. i.e. ‘service-extension-$PROJECT_ID'.
- Format - ‘Docker,' Mode - ‘Standard', Location Type - ‘Region', and pick ‘us-central1 (Iowa)'
- Click the CREATE button.
The newly created Artifact Registry Repository resource should appear on the Repositories page.
Once the Repository resource has been created, run the following command in Cloud Shell to configure your Cloud Shell docker client to push and pull packages using this repository.
gcloud auth configure-docker $LOCATION-docker.pkg.dev
Output:
... Adding credentials for: us-central1-docker.pkg.dev Docker configuration file updated.
8. Configure Service Extensions on Media CDN
Now, we will demonstrate how to write and build a Service Extension (Proxy-Wasm) plugin that can be deployed to Media CDN, using the Rust programming language.
In this example, we will create a Proxy-Wasm plugin that verifies each HTTP request contains an Authorization header with the value "secret". If the request does not contain this header, the plugin will generate an HTTP 403 Forbidden response.
A quick refresher on Service Extensions- there are three key resources: WasmAction, WasmPlugin, and WasmPluginVersion.
- A WasmAction resource is what gets attached to your Media CDN EdgeCacheService. A WasmAction references a WasmPlugin resource.
- A WasmPlugin resource has a main-version which corresponds to the current active WasmPluginVersion.
- A WasmPluginVersions reference a container image from Artifact Registry. As you make changes to your proxy-wasm plugins, you create different WasmPluginVersions.
Please reference the below diagram to better understand the relationship between these resources.
Write and build a Service Extension plugin
- Install the Rust toolchain by following the instructions at https://www.rust-lang.org/tools/install.
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
- Next, add Wasm support to your Rust toolchain by running the following command:
rustup target add wasm32-wasi
- Create a Rust package called my-wasm-plugin:
cargo new --lib my-wasm-plugin
Output:
Created library `my-wasm-plugin` package
- Enter the directory my-wasm-plugin and you should see a
Cargo.toml
file, and asrc
directory.
cd my-wasm-plugin ls
Output:
Cargo.toml src
- Next, configure your Rust package by editing the
Cargo.toml
file. After the[dependencies]
line in the Cargo.toml file, add the following:
proxy-wasm = "0.2"
log = "0.4"
[lib]
crate-type = ["cdylib"]
[profile.release]
lto = true
opt-level = 3
codegen-units = 1
panic = "abort"
strip = "debuginfo"
- After your edits, the
Cargo.toml
file should look roughly like so:
[package]
name = "my-wasm-plugin"
version = "0.1.0"
edition = "2021"
[dependencies]
proxy-wasm = "0.2"
log = "0.4"
[lib]
crate-type = ["cdylib"]
[profile.release]
lto = true
opt-level = 3
codegen-units = 1
panic = "abort"
strip = "debuginfo"
- . Copy over the full content of the sample_code file to the
lib.rs
file in thesrc
directory in Cloud Shell.
- After your edits, the
lib.rs
file should look roughly like so:
use log::info;
use proxy_wasm::traits::*;
use proxy_wasm::types::*;
...
struct DemoPlugin;
impl HttpContext for DemoPlugin {
fn on_http_request_headers(&mut self, _: usize, _: bool) -> Action {
if self.get_http_request_header("Authorization") == Some(String::from("secret")) {
info!("Access granted.");
Action::Continue
} else {
self.send_http_response(403, vec![], Some(b"Access forbidden.\n"));
Action::Pause
}
}
}
impl Context for DemoPlugin {}
- Now that we have configured the
Cargo.toml
manifest file and written our Proxy-Wasm code inlib.rs
file , we can build our Proxy-Wasm plugin.
cargo build --release --target wasm32-wasi
Once the build completes successfully, you will see a message as shown below:
Finished release [optimized] target(s) in 1.01s
Let's also verify the target
directory and check the files been created:
ls ./target
You will see the output as shown below:
CACHEDIR.TAG release wasm32-wasi
Publish a Proxy-Wasm plugin to Artifact Registry
Now, we will publish our Proxy-Wasm plugin to the Artifact Registry Repository you created before so it can be deployed to Media CDN.
We first package the Proxy-Wasm plugins in a container image.
- Create a file called
Dockerfile
in the same directory my-wasm-plugin, with the following contents:
FROM scratch
COPY target/wasm32-wasi/release/my_wasm_plugin.wasm plugin.wasm
- Next, build the container image:
docker build --no-cache --platform wasm -t my-wasm-plugin .
(non-x86 processors only) Next, build the container image:
docker build --no-cache --platform wasm --provenance=false -t my-wasm-plugin .
Output
[+] Building 0.2s (5/5) FINISHED docker:default ...
- Next, publish or "push" your Proxy-Wasm plugin to Artifact Registry. We will tag our container image with the ‘prod' tag.
docker tag my-wasm-plugin $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin:prod
Now we go ahead and push the tagged ‘prod' container image to the repository.
docker push $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin:prod
Output:
The push refers to repository ... 8564ddd9910a: Pushed prod: digest: sha256:f3ae4e392eb45393bfd9c200cf8c0c261762f7f39dde5c7cd4b9a8951c6f2812 size: 525
Now Let's verify the container image of Proxy-Wasm plugin was successfully pushed to Artifact Registry, you should see a similar output:
gcloud artifacts docker images list $LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin --include-tags
Output:
Listing items under project ... IMAGE DIGEST TAGS CREATE_TIME UPDATE_TIME <LOCATION>-docker.pkg.dev/.../my-wasm-plugin sha256:08c12... prod 2021-11-10T23:31:27 2021-11-10T23:31:27
Associate a Proxy-Wasm plugin with your Media CDN deployment
Now we are ready to associate the Proxy-Wasm plugin to your Media CDN deployment.
Proxy-Wasm plugins are associated with Media CDN routes in the EdgeCacheService resource.
- First, we create a Wasm-plugin resource for our Proxy-Wasm plugin.
gcloud alpha service-extensions wasm-plugins create my-wasm-plugin-resource
- Next, we create a WasmPluginVersion.
gcloud alpha service-extensions wasm-plugin-versions create my-version-1 \ --wasm-plugin=my-wasm-plugin-resource \ --image="$LOCATION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/my-wasm-plugin:prod"
- Next, we specify the main version for our Proxy-Wasm plugin.
gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource \ --main-version=my-version-1
Now Let's verify the Proxy-Wasm plugin was successfully associated to Container Image resides in Artifact Registry Repository, you should see a similar output:
gcloud alpha service-extensions wasm-plugin-versions list --wasm-plugin=my-wasm-plugin-resource
Output:
NAME WASM_IMAGE WASM_IMAGE_DIGEST CONFIG_SIZE CONFIG_IMAGE CONFIG_IMAGE_DIGEST UPDATE_TIME c7cfa2 <LOCATION>-docker.pkg.dev/.../my-wasm-plugin@sha256:6d663... ... ... ...
- Next, we create a WasmAction resource referencing our Wasm plugin resource.
gcloud alpha service-extensions wasm-actions create my-wasm-action-resource \ --wasm-plugin=my-wasm-plugin-resource
Let's also verify the WasmAction resource was successfully associated to the Proxy-Wasm plugin, you should see a similar output:
gcloud alpha service-extensions wasm-actions list
Output:
NAME WASMPLUGIN my-wasm-action-resource projects/805782461588/locations/global/wasmPlugins/myenvoyfilter-resource ...
- Now, we need to export the configuration of our Media CDN EdgeCacheService:
gcloud edge-cache services export media-cdn --destination=my-service.yaml
- Then, open the my-service.yaml file and add a wasmAction to the routeAction for the given route, which references the WasmPlugin resource created earlier.
wasmAction: "my-wasm-action-resource"
- After your edits, the my-service.yaml file should look roughly like so:
...
pathMatchers:
- name: routes
routeRules:
- headerAction: {}
matchRules:
- prefixMatch: /
origin: projects/<PROJECT_NUM>/locations/global/edgeCacheOrigins/cloud-storage-origin
priority: '1'
routeAction:
cdnPolicy:
cacheKeyPolicy: {}
cacheMode: FORCE_CACHE_ALL
defaultTtl: 3600s
signedRequestMode: DISABLED
wasmAction: "my-wasm-action-resource"
...
- Then, we save the updated configuration with Proxy-Wasm configuration to the
my-service-with-wasm.yaml
file.
- Finally, we import the updated configuration for the production Media CDN environment:
$ gcloud alpha edge-cache services import media-cdn --source=my-service-with-wasm.yaml
9. Validate Service Extensions Proxy-Wasm plugin on Media CDN
To test that your service is correctly configured to cache content, use the curl command-line tool to issue requests and check the responses.
curl -svo /dev/null "http://IP_ADDRESS/file.txt"
The command should produces output similar to the following:
< HTTP/2 403 Forbidden ... Access forbidden. ...
Now, issue the request again with an Authorization header and its value of secret
curl -svo /dev/null "http://IP_ADDRESS/file.txt" -H "Authorization: secret"
The command should produces output similar to the following:
< HTTP/2 200 OK ... media-cdn-service-extensions-test ...
10. Optional: Managing Proxy-Wasm plugins
Updating a Proxy-Wasm plugin
As you make improvements or add new functionality to your Proxy-Wasm plugins, you will need to deploy your updated plugins to Media CDN. Below, we go through the steps to deploy an updated version of a plugin.
As an example, you might update the sample plugin code to evaluate the Authorization header against another value for authentication, by modifying the code like the following.
First, update the src/lib.rs source file with the code shown below:
use log::{info, warn};
use proxy_wasm::traits::*;
use proxy_wasm::types::*;
...
struct DemoPlugin;
impl HttpContext for DemoPlugin {
fn on_http_request_headers(&mut self, _: usize, _: bool) -> Action {
if self.get_http_request_header("Authorization") == Some(String::from("another_secret")) {
info!("Access granted.");
Action::Continue
} else {
warn!("Access forbidden.");
self.send_http_response(403, vec![], Some(b"Access forbidden.\n"));
Action::Pause
}
}
}
impl Context for DemoPlugin {}
Next, build, package, and publish the updated plugin:
cargo build --release --target wasm32-wasi docker build --no-cache --platform wasm -t my-wasm-plugin . docker tag my-wasm-plugin $LOCATION-docker.pkg.dev/$PROJECT_NUM/$REPOSITORY/my-wasm-plugin:prod docker push $LOCATION-docker.pkg.dev/$PROJECT_NUM/$REPOSITORY>/my-wasm-plugin:prod
Once the container image is updated in Artifact Registry, we need to create a new WasmPluginVersion and then update the –main-version of the WasmPlugin to reference the new version.
gcloud alpha service-extensions wasm-plugin-versions create my-version-2 \ --wasm-plugin=my-wasm-plugin-resource \ --image="$LOCATION-docker.pkg.dev/$PROJECT_NUM/$REPOSITORY>/my-wasm-plugin:prod"
gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource \ --main-version=my-version-2
Now, you have successfully updated the version of the container image to be imported from Artifact Registry and pushed live to your Media CDN deployment.
Rolling back to a previous version
To roll back to a previous version of a plugin, you can update the Wasm plugin resource to reference a previous version.
First, we list the available versions:
gcloud alpha service-extensions wasm-plugin-versions list --wasm-plugin=my-wasm-plugin-resource
You should see the output:
NAME WASM_IMAGE WASM_IMAGE_DIGEST CONFIG_SIZE CONFIG_IMAGE CONFIG_IMAGE_DIGEST UPDATE_TIME c7cfa2 <LOCATION>-docker.pkg.dev/.../my-wasm-plugin@sha256:6d663... ... ... a2a8ce <LOCATION>-docker.pkg.dev/.../my-wasm-plugin@sha256:08c12... ... ...
Next, we update the Wasm plugin resource to reference the previous version, "a2a8ce":
$ gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource \ --main-version="a2a8ce"
Once the operation is successful, you should see this output:
✓ WASM Plugin [my-wasm-plugin-resource] is now serving version "a2a8ce"
Since Media CDN saves the image digest of your Docker image each time a new Wasm-plugin resource is created, the rollback will use the version of your code that was running before the last rollout.
gcloud alpha service-extensions wasm-plugins describe my-wasm-plugin-resource \ --expand-config
For version "a2a8ce", that is the version with digest sha256:08c12...:
name: "my-wasm-plugin-resource" mainVersion: "a2a8ce" mainVersionDetails: image: "<LOCATION>-docker.pkg.dev/<PROJECT>/<REPOSITORY>/my-wasm-plugin" imageDigest: "<LOCATION>-docker.pkg.dev/<PROJECT>/<REPOSITORY>/my-wasm-plugin@sha256:08c121dd7fd1e4d3a116a28300e9fc1fa41b2e9775620ebf3d96cb7119bd9976"
Deleting a WasmAction & WasmPlugin
To delete a WasmAction, WasmPlugin, and the associated WasmPluginVersions, please follow these steps.
First, remove the reference to the WasmAction in your Media CDN EdgeCacheService config.
Reference line to be removed:
wasmAction: "my-wasm-action-resource"
Then, we update the edited EdgeCacheService config.
gcloud alpha edge-cache services import prod-media-service --source=my-service.yaml
Next, update the main-version of your WasmPlugin to an empty string "".
gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource --main-version=
""
Finally, perform the below deletion steps in order.
gcloud alpha service-extensions wasm-actions delete my-wasm-action-resource gcloud alpha service-extensions wasm-plugin-versions delete my-version \ --wasm-plugin=my-wasm-plugin-resource gcloud alpha service-extensions wasm-plugins delete my-wasm-plugin-resource
11. Clean up the Lab environment
After you have completed the CodeLab, don't forget to clean up the lab resources - otherwise they'll keep running and accruing costs.
The following commands will delete the Media CDN EdgeCache Service, EdgeCache Config, and the Service Extensions Plugin. Perform the below deletion steps in order.
gcloud edge-cache services delete media-cdn gcloud edge-cache origins delete cloud-storage-origin gcloud alpha service-extensions wasm-actions delete my-wasm-action-resource gcloud alpha service-extensions wasm-plugins update my-wasm-plugin-resource --main-version="" gcloud alpha service-extensions wasm-plugin-versions delete my-version-1 --wasm-plugin=my-wasm-plugin-resource gcloud alpha service-extensions wasm-plugins delete my-wasm-plugin-resource gcloud artifacts repositories delete service-extension-$PROJECT_ID --location=$LOCATION
Each of the commands above should ask you to confirm the deletion of the resource.
12. Congratulations!
Congratulations, you've completed the Service Extensions on Media CDN codelab!
What we've covered
- How to set up Media CDN with a Cloud Storage Bucket set as Origin
- How to create a Service Extension plugin with custom HTTP authentication and associate it with Media CDN
- How to validate that the Service Extension plugin is working as expected
- (optional) How to manage a Service Extension plugin like updating, referencing, rolling back, and deleting a specific plugin version
What's next?
Check out some of these codelabs...