About this codelab
1. Introduction
Cloud KMS Autokey simplifies creating and using customer-managed encryption keys (CMEKs) by automating provisioning and assignment. With Autokey, your key rings, keys, and service accounts don't need to be planned and provisioned before they're needed. Instead, Autokey generates your keys on demand as your resources are created, relying on delegated permissions instead of Cloud KMS administrators.
Using keys generated by Autokey can help you consistently align with industry standards and recommended practices for data security, including the HSM protection level, separation of duties, key rotation, location, and key specificity. Autokey creates keys that follow both general guidelines and guidelines specific to the resource type for Google Cloud services that integrate with Cloud KMS Autokey. After they are created, keys requested using Autokey function identically to other Cloud HSM keys with the same settings.
What you'll build
In this codelab, you're going to launch protected resources using Cloud KMS Autokey by creating:
- a folder resource
- a project that will contain your keys
- a service agent to act as your key-managing assistant
- a project that will host your protected resources
- BigQuery datasets, persistent disks, and Cloud Storage buckets encrypted with Cloud KMS Autokey
What you'll need
- Google Cloud Organization
- Your Google Cloud principal must have the following roles at the Organization level to complete this lab:
- Cloud KMS Autokey Admin (roles/cloudkms.autokeyAdmin)
- Folder IAM Admin (roles/resourcemanager.folderIamAdmin)
- Billing Account user (roles/billing.user)
- Google Cloud Projects with Billing Enabled
- Basic Linux Experience
2. Create a Folder
Folders are nodes in the Cloud Platform Resource Hierarchy. A folder can contain projects, other folders, or a combination of both. Organization resources can use folders to group projects under the organization resource node in a hierarchy. To create a folder:
- Go to the Manage resources page in the Google Cloud console
- Make sure that your organization resource name is selected in the organization drop-down list at the top of the page.
- Click Create folder
- Select Standard Folder
- In the Folder name box, enter your new folder's name. For this lab, consider "Autokey-Folder"
- Under Destination, click Browse, then select the organization resource or folder under which you want to create your new folder.
- Click Create.
3. Create a Resource Project
It's important to create a resource project to contain the resources - such as BigQuery datasets, persistent disks, and Cloud Storage Buckets - you want to encrypt with Cloud KMS Autokey. If you attempt to create resources protected by Autokey in the key project, Autokey rejects the request for a new key. To create the resource project:
- Go to the Manage resources page in the Google Cloud console
- Click Create Project.
- On the Select organization drop-down list at the top of the page, select the "Autokey-Folder" folder.
- In the New Project window that appears, enter a project name and select a billing account as applicable. For this lab, consider "Autokey Encrypted Resources"
- In the Location box, select the "Autokey-Folder" folder. That resource will be the hierarchical parent of the new project. Your settings should look similar to this:
- Copy the Project ID - in the example above the Project ID is "causal-hour-43319-m4" but your ID will be different - into the text editor of your choice.
- Click Create
- Select the Cloud Shell icon on the top right corner of your screen
- Once the Cloud Shell is active, save your Autokey Project ID as a variable by running the following command:
export RESOURCE_PROJECT=<paste your Resource Project ID>
Because my Project ID is "key-management-433319" my command looks like so:
export AUTOKEY_PROJECT=causal-hour-43319-m4
- Run the following command to execute commands from your Key Project:
gcloud config set project $RESOURCE_PROJECT
When prompted, authorize Cloud Shell by clicking "Authorize"
- Because this project will contain resources, we need to enable the APIs for the services Autokey will protect. Run the following command:
gcloud services enable storage.googleapis.com bigquery.googleapis.com compute.googleapis.com
4. Create a Key Project
We recommend creating a project to contain Cloud KMS resources created by Autokey. This will be referred to as the "key project" going forward. The key project can be created inside the same folder where you plan to enable Autokey. You shouldn't create other resources inside the key project. If you attempt to create resources protected by Autokey in the key project, Autokey rejects the request for a new key. To create the key project:
- Go to the Manage resources page in the Google Cloud console
- Click Create Project.
- On the Select organization drop-down list at the top of the page, select the "Autokey-Folder" folder.
- In the New Project window that appears, enter a project name and select a billing account as applicable. For this lab, consider "Key Management"
- In the Location box, select the "Autokey-Folder" folder. That resource will be the hierarchical parent of the new project. Your settings should look similar to this:
- Copy the Project ID - in the example above the Project ID is "key-management-433319" but your ID will be different - into the text editor of your choice.
- Click Create.
5. Prepare the Autokey key project
Now that each project is created, it's time to configure the key project to use Cloud KMS Autokey.
- Select the Cloud Shell icon on the top right corner of your screen
- Once the Cloud Shell is active, save your Autokey Project ID as a variable by running the following command:
export AUTOKEY_PROJECT=<paste your Autokey Project ID>
Because my Project ID is "key-management-433319" my command looks like so:
export AUTOKEY_PROJECT=key-management-433319
- Run the following command to execute commands from your Key Project:
gcloud config set project $AUTOKEY_PROJECT
When prompted, authorize Cloud Shell by clicking "Authorize"
- Enable the Cloud KMS API by running the following command
gcloud services enable cloudkms.googleapis.com kmsinventory.googleapis.com
- Run the following command to save your Project Number as a variable named AUTOKEY_PROJECT_NUMBER
export AUTOKEY_PROJECT_NUMBER=$(gcloud projects list \
--filter="$(gcloud config get-value project)" \
--format="value(PROJECT_NUMBER)")
- Save your Principal's email as a variable by running the following command:
export KEY_ADMIN_EMAIL=<paste your Principal's email>
- Grant Cloud KMS administrator permissions on the key project to your Cloud KMS administrator users
gcloud projects add-iam-policy-binding $AUTOKEY_PROJECT_NUMBER \
--role=roles/cloudkms.admin \
--member=user:$KEY_ADMIN_EMAIL
6. Set up the Cloud KMS service agents
The Cloud KMS service agent for a key project creates keys and applies IAM policy bindings during resource creation, on behalf of a human Cloud KMS administrator. To be able to create and assign keys, the Cloud KMS service agent requires Cloud KMS administrator permissions.
- Find the Organization ID with the following command:
gcloud organizations list | grep -P -i 'ID:' | grep -i '[0-9]'
- Copy the Organization ID - this is the numerical result that is highlighted in red
- Save the Organization ID as a variable named ORG_ID:
export ORG_ID=<paste your Organization ID>
- Create the Cloud KMS service agent by running the following command:
gcloud beta services identity create --service=cloudkms.googleapis.com \
--project=$AUTOKEY_PROJECT_NUMBER
- Grant Cloud KMS administrator role to the service agent:
gcloud projects add-iam-policy-binding $AUTOKEY_PROJECT_NUMBER \
--role=roles/cloudkms.admin \
--member=serviceAccount:service-$AUTOKEY_PROJECT_NUMBER@gcp-sa-cloudkms.iam.gserviceaccount.com
- Add an IAM policy binding so that you can view your Cloud KMS Key usage. Key Usage provides information for each key including the number of protected resources, projects, and unique Google Cloud products that use the key. This level of detail is available to anyone with the Cloud KMS Viewer role on the key. Run the following command:
gcloud organizations add-iam-policy-binding $ORGANIZATION_ID \
--member="serviceAccount:service-org-$ORGANIZATION_ID@gcp-sa-cloudkms.iam.gserviceaccount.com" \
--role='roles/cloudkms.orgServiceAgent'
- Grant your Cloud KMS service account the Cloud KMS Organization Service Agent (cloudkms.orgServiceAgent) role on your organization resource.
gcloud organizations add-iam-policy-binding $ORG_ID \
--member="serviceAccount:service-org-$ORG_ID@gcp-sa-cloudkms.iam.gserviceaccount.com" \
--role='roles/cloudkms.orgServiceAgent'
- Grant the Cloud KMS Protected Resources Viewer role on your Organization resource to anyone who needs to view key usage details.
gcloud organizations add-iam-policy-binding $ORG_ID \
--member="user:$KEY_ADMIN_EMAIL" \
--role='roles/cloudkms.protectedResourcesViewer'
7. Grant Autokey user roles
Before your developers can use Autokey, you must grant them the required role. You can grant the role at the folder level or at the project level. This role lets developers request keys from the Cloud KMS service agent while creating resources in that folder or project.
The first step in granting the role is to save the Folder ID.
- Find the Folder ID with the following command:
gcloud projects describe $AUTOKEY_PROJECT | grep 'id' | grep -P -i '[0-9]+'
- Copy the Folder ID - this is the part that is highlighted in red
- Save the Folder ID as a variable named FOLDER_ID
export FOLDER_ID=<paste the folder ID>
For the purposes of this lab, we will define the key administrator as the Autokey user. However, in production use cases and in organizations with more than one individual, the key administrator should be different from the developer using Autokey.
- Grant the roles/cloudkms.autokeyUser role at the folder level:
gcloud resource-manager folders add-iam-policy-binding \
$FOLDER_ID --role=roles/cloudkms.autokeyUser \
--member=user:$KEY_ADMIN_EMAIL
8. Enable Cloud KMS Autokey on a resource folder
In this step, you will enable Cloud KMS Autokey on a resource folder and identify the Cloud KMS project that will contain Autokey resources for that folder. Enabling Autokey on this folder enables Autokey for all resource projects within the folder.
- In the Google Cloud console, go to the KMS controls page.
- Click on Select Folder
- From the context picker, select the folder where you want to enable Autokey. This will be the same folder you created earlier and contains your resource project and the key management project. It should look like this:
- Click Enable.
- Click Browse to select the key project
- Select your key management project and then click Submit.
A message confirms that Cloud KMS Autokey is enabled on the folder. The KMS Controls page should look like this:
9. Create protected resources using Cloud KMS Autokey
Compute Engine Persistent Disks
Autokey creates a new key for each disk, image, and machine image in the same location as the resource that is being created.
To create a disk, complete the following steps:
- In the Google Cloud console, go to the Disks page.
- Click Create disk and enter the properties for the new disk.
- Under Encryption, select Cloud KMS key.
- For Key type, select Cloud KMS with Autokey, and then click Request a new key. A message indicates when your key has been successfully created and is ready for use.
- To finish creating the disk, click Create.
You can follow a similar process to create protected VM instance, image, and machine image resources.
Google Cloud Storage Buckets
Autokey creates a new key in the same location as the bucket. The key created by Autokey is assigned as the bucket default key.
Autokey doesn't create keys for objects. By default, objects created in a bucket use the bucket default key. If you want to encrypt an object using a key other than the bucket default key, you can manually create a CMEK and use that key when creating the object.
- In the Google Cloud console, go to the Create a bucket page.
- Pick a globally unique, permanent name.
- Choose a data location.
- Proceed to the "Choose how to protect object data" section
- Click on "Choose how to protect object data" to expand the section
- Expand the Data encryption section and select Cloud KMS key.
- For Key type, select Cloud KMS with Autokey, and then click Request a new key. A message indicates when your key has been successfully created and is ready for use.
- To finish creating the bucket, click Create. If you are prompted with a dialog box explaining that "Public access will be prevented" click Confirm.
BigQuery Dataset
For each new dataset, Autokey creates a new key, in the same location as the resource itself, which becomes the dataset default key. Autokey doesn't create keys for tables, queries, temporary tables, or models. By default, these resources are protected by the dataset default key. If you want to protect a resource in a dataset using a key other than the dataset default key, you can manually create a CMEK and use that key when creating the resource.
To create a BigQuery dataset, you must have the BigQuery User role first.
- Return to the Cloud Shell
- Set your Cloud Shell to execute commands from the resource project
gcloud config set project $RESOURCE_PROJECT
- Run the following command to save your Project Number as a variable named RESOURCE_PROJECT_NUMBER
export RESOURCE_PROJECT_NUMBER=$(gcloud projects list --filter="$(gcloud config get-value project)" --format="value(PROJECT_NUMBER)")
- Grant yourself the BigQuery User role
gcloud projects add-iam-policy-binding $RESOURCE_PROJECT_NUMBER \
--role=roles/bigquery.user \
--member=user:$KEY_ADMIN_EMAIL
Now that you have the BigQuery User role, you can create a dataset and protect it with Autokey!
- In the Google Cloud console, go to the BigQuery page.
- Follow the instructions to Create a dataset until you get to Advanced options > Encryption.
- Under Encryption select Cloud KMS key.
- For Key type, select Cloud KMS with Autokey, and then click Request a new key. A message indicates when your key has been successfully created and is ready for use.
- To finish creating the dataset, click Create dataset.
10. Explore your Keys
In this step, you will explore the keys Cloud KMS Autokey created on your behalf by visiting the Key Inventory page. The Key Inventory page provides comprehensive information about the cryptographic keys in your project. Note that the data might be delayed. For example, if you create a new protected resource, the protected resource and associated key version aren't immediately added to the Usage tracking tab. See more limitations here.
- In the Google Cloud console, go to the Key Inventory page.
- Optional: To filter the list of keys, enter your search terms in the filter_list Filter box and then press enter. For example, you can filter by location, key ring, status, or other properties of the keys.
- Click the name of the key for which you want to view usage information.
- Click on "Overview". Notice that you have a key per resource created. Each key name includes the name of the resource the key is protecting (e.g. "compute-disk" or "storage-bucket"). Cloud KMS Autokey ensures that each key is scheduled for rotation 365 days after its creation and each key is assigned the "HSM" protection level.
- Click the Usage Tracking tab. Notice the level of information presented: each resource the key is encrypting is shown here, alongside the project, location, and creation date.
- Optional: To filter the list of protected resources, enter your search terms in the filter_list Filter box and then press Enter.
11. Congratulations
Congratulations, you've successfully created Google Cloud resources and automatically encrypted them on-demand with Cloud KMS Autokey!
You now know the key steps required to set up Autokey and use it to automatically encrypt your resources with Cloud KMS keys.
12. What's next?
Upload Data to your Autokey-encrypted resources
- Create a Google Compute Engine (GCE) instance
- Attach your Autokey-protected persistent disk to your GCE instance
- Upload data into your BigQuery dataset
- Upload objects into a Google Cloud Storage Bucket
- Load Google Cloud Storage data into BigQuery