HashiCorp Vault is a popular open source tool for secrets management that codifies many of the best practices around secrets management including time-based access controls, principles of least privilege, encryption, dynamic credentials, and much more. This codelab teaches you how to use Vault's Google Cloud KMS plugin to encrypt and decrypt data through Vault using Cloud KMS.

What you'll learn

Self-paced environment setup

If you don't already have a Google Account (Gmail or Google Apps), you must create one. Sign-in to Google Cloud Platform console (console.cloud.google.com) and create a new project:

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document).

New users of Google Cloud Platform are eligible for a $300 free trial.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

Before deploying Vault in production, first install Vault locally. This will enable you to use the vault CLI locally and will be used to interact with the cluster later.

You could browse to the Vault website, but this section will teach you how to download, verify, and install Vault securely. Even though Vault is downloaded over a TLS connection, it may still be possible for a skilled attacker to compromise the underlying storage system or network transport. For that reason, in addition to serving the binaries over TLS, HashiCorp also signs the checksums of each release with their private key. Thus, to verify the integrity of a download, we must:

  1. Import and trust HashiCorp's GPG public key
  2. Download the Vault binary
  3. Download the Vault checksums
  4. Download the Vault checksum signature
  5. Verify the signature of the checksum against HashiCorp's GPG key
  6. Verify the checksums of the binary against the file

This way, even if an attacker were able to compromise the network transport and underlying storage component, they wouldn't be able to sign the checksums with HashiCorp's GPG key. If this operation is successful, we have an extremely high degree of confidence that the software is untainted.

Since that process can be tedious, we will leverage a Docker container to do it for us. Execute the following command to install Vault locally. We install Vault into $HOME/bin because that will persist between restarts on Cloud Shell.

$ docker run -v $HOME/bin:/software sethvargo/hashicorp-installer vault 1.0.0-beta1
$ sudo chown -R $(whoami):$(whoami) $HOME/bin/

Add the bin to our path:

$ export PATH=$HOME/bin:$PATH

Finally, optionally, explore the Vault CLI help. Most Vault commands will not work because there is no Vault server running. Do not start a Vault server yet.

$ vault -h

Start a local, development Vault server. This Vault server runs entirely in memory and does not represent a best practices installation. However, it is useful for getting started quickly and exploring Vault's functionality. We will also create an initial token in Vault with the value of "root", which will be used to authenticate to the Vault server.

$ export VAULT_ADDR=
$ vault server -dev &> vault.log &

Vault is now running in the background. You can query its status to verify:

$ vault status

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
# ...

Enable the Google Cloud KMS API. This only needs to be done once per project to make the API accessible.

$ gcloud services enable cloudkms.googleapis.com

Enable the Cloud KMS secrets engine in Vault.

$ vault secrets enable gcpkms

Create a Cloud KMS key by issuing commands Vault. During this process, Vault makes the necessary API calls to Cloud KMS to create a key ring, crypto key, and crypto key version. The metadata is stored in Vault, but the actual encryption keys are stored and managed by Google Cloud.

$ vault write gcpkms/keys/my-key \
    key_ring=projects/${GOOGLE_CLOUD_PROJECT}/locations/global/keyRings/vault \

That's it! Vault now has an internal reference to this external Cloud KMS key. When a request is made to encrypt or decrypt data, it will be delegated to Cloud KMS.

You can verify that the key was created using the gcloud CLI:

$ gcloud kms keyrings list --location global


Encrypt plaintext data using this KMS key with Vault. You will receive back ciphertext. Neither Vault nor Google Cloud KMS store the ciphertext. You are responsible for persisting the ciphertext externally, like on the filesystem or in a database.

$ vault write gcpkms/encrypt/my-key plaintext="hello world"

Key            Value
---            -----
ciphertext     CiQA3/Sf9aWbWSI4XK...
key_version    1

When you want to retrieve the plaintext, give Vault the stored ciphertext. Vault will make the necessary API calls to KMS to decrypt the ciphertext. Be sure to replace the ciphertext with the value you received from the encrypt call.

$ vault write gcpkms/decrypt/my-key ciphertext="CiQA3/Sf9aWbWSI4XK..."

Key          Value
---          -----
plaintext    hello world

It is also possible to rotate crypto keys, upgrade ciphertext data to the latest version, and delete crypto key versions.

You learned how to run and configure HashiCorp Vault on Google Cloud to encrypt, decrypt and manage Cloud KMS keys.

Clean up

If you are done exploring, please consider deleting your project.

Learn More


This work is licensed under a Creative Commons Attribution 2.0 Generic License.