Using Secret Manager with Python

Using Secret Manager with Python

About this codelab

Last updated Sep 5, 2022

Written by Katie McLaughlin

1. Overview

In this codelab, you will focus on using Secret Manager in Python.

Secret Manager allows you to store, manage, and access secrets as binary blobs or text strings. With the appropriate permissions, you can view the contents of the secret.

Secret Manager works well for storing configuration information such as database passwords, API keys, or TLS certificates needed by an application at runtime.

What you'll learn

• How to use Cloud Shell
• How to install the Secret Manager client library for Python
• How to create and access secrets using the Python client library
• How to access secrets in Cloud Functions using the Python client library

What you'll need

• A Google Cloud Project
• A Browser, such as Chrome or Firefox
• Familiarity using Python 3

Survey

How will you use this tutorial?

Read it through onlyRead it and complete the exercises

How would you rate your experience with Python?

NoviceIntermediateProficient

How would you rate your experience with using Google Cloud services?

NoviceIntermediateProficient

2. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (it is typically identified as PROJECT_ID). If you don't like the generated ID, you may generate another random one. Alternatively, you can try your own and see if it's available. It cannot be changed after this step and will remain for the duration of the project.
• For your information, there is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. Even if a project is deleted, the ID can never be used again

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, you can delete the resources you created or delete the whole project. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

Activate Cloud Shell

1. From the Cloud Console, click Activate Cloud Shell .

If you've never started Cloud Shell before, you're presented with an intermediate screen (below the fold) describing what it is. If that's the case, click Continue (and you won't ever see it again). Here's what that one-time screen looks like:

It should only take a few moments to provision and connect to Cloud Shell.

This virtual machine is loaded with all the development tools you need. It offers a persistent 5GB home directory and runs in Google Cloud, greatly enhancing network performance and authentication. Much, if not all, of your work in this codelab can be done with simply a browser or your Chromebook.

Once connected to Cloud Shell, you should see that you are already authenticated and that the project is already set to your project ID.

2. Run the following command in Cloud Shell to confirm that you are authenticated:

gcloud auth list

Command output

 Credentialed Accounts
ACTIVE  ACCOUNT
*       <my_account>@<my_domain.com>

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

Note: The gcloud command-line tool is the powerful and unified command-line tool in Google Cloud. It comes preinstalled in Cloud Shell. You will notice its support for tab completion. For more information, see gcloud command-line tool overview.

3. Run the following command in Cloud Shell to confirm that the gcloud command knows about your project:

gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If it is not, you can set it with this command:

gcloud config set project <PROJECT_ID>

Command output

Updated property [core/project].

3. Enable the Secret Manager API

Before you can begin using the Secret Manager API, you must enable the API. Using Cloud Shell, you can enable the API with the following command:

gcloud services enable secretmanager.googleapis.com

You should see output like this:

Operation "operations/acf.cc11852d-40af-47ad-9d59-477a12847c9e" finished successfully.

Note: If this operation is unsuccessful, go back to the previous step and check your setup.

4. Installing the Secret Manager client library for Python

Install the Secret Manager Client Library:

pip3 install --user google-cloud-secret-manager==2.10.0

5. Start Interactive Python

For part of this tutorial, you'll use an interactive Python interpreter called IPython, which is preinstalled in Cloud Shell. Start a session by running ipython in Cloud Shell:

ipython

You should see something like this:

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

Note: If needed, you can quit your IPython session with the exit command.

6. Creating Secrets

A secret contains one or more secret versions. They can be created using the gcloud command-line, but they can also be created using Python.

In order to use a secret, you first need to create the secret with the name of the secret, then you add a version of the secret, being the value of the secret.

Set your Project ID within IPython:

PROJECT_ID = "<PROJECT_ID>"

Create a secret

Copy the following code into your IPython session:

from google.cloud import secretmanager

def create_secret(secret_id):
    # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the parent project.
    parent = f"projects/{PROJECT_ID}"

    # Build a dict of settings for the secret
    secret = {'replication': {'automatic': {}}}

    # Create the secret
    response = client.create_secret(secret_id=secret_id, parent=parent, secret=secret)

    # Print the new secret name.
    print(f'Created secret: {response.name}')   

Call the function to create a new secret called my_secret_value:

create_secret("my_secret_value")

You should see the following output:

Created secret: projects/<PROJECT_NUM>/secrets/my_secret_value

Note: If you get a PermissionDenied error (403), revisit the Enable the Secret Manager API step.

Add a secret version

Now that the secret exists, you can assign it a value by creating a version.

Copy the following code into your IPython session:

def add_secret_version(secret_id, payload):
    # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the parent secret.
    parent = f"projects/{PROJECT_ID}/secrets/{secret_id}"

    # Convert the string payload into a bytes. This step can be omitted if you
    # pass in bytes instead of a str for the payload argument.
    payload = payload.encode('UTF-8')

    # Add the secret version.
    response = client.add_secret_version(parent=parent, payload={'data': payload})

    # Print the new secret version name.
    print(f'Added secret version: {response.name}')   

Call the function to create a add a new secret version:

add_secret_version("my_secret_value", "Hello Secret Manager")

You should see the following output:

Added secret version: projects/<PROJECT_NUM>/secrets/my_secret_value/versions/1

Secrets can have multiple versions. Call the function again with a different value:

add_secret_version("my_secret_value", "Hello Again, Secret Manager")

You should see the following output:

Added secret version: projects/<PROJECT_NUM>/secrets/my_secret_value/versions/2

Notice how the new version of our secret is significantly longer than our original. This attribute will be referenced later.

7. Accessing secrets

Accessing a secret version returns the secret contents, as well as additional metadata about the secret version. When you access a secret version, you can either specify a specific version, or just ask for the latest version by specifying "latest".

Warning: Do not print the value of secrets in production.

Secrets should be kept secret. Store database credentials as secrets then use them to authenticate, or store certifications and use them; but do not directly print out your secrets, as this defeats the purpose of keeping them secret.

You're going to perform operations on our secrets, assessing its value without printing it out directly. Instead you'll print out a hash of the value of the secret.

Copy the following code into your IPython session:

def access_secret_version(secret_id, version_id="latest"):
    # Create the Secret Manager client.
    client = secretmanager.SecretManagerServiceClient()

    # Build the resource name of the secret version.
    name = f"projects/{PROJECT_ID}/secrets/{secret_id}/versions/{version_id}"

    # Access the secret version.
    response = client.access_secret_version(name=name)

    # Return the decoded payload.
    return response.payload.data.decode('UTF-8')
    
import hashlib

def secret_hash(secret_value): 
  # return the sha224 hash of the secret value
  return hashlib.sha224(bytes(secret_value, "utf-8")).hexdigest()

Call the function to retrieve the secret as a hash of it's value:

secret_hash(access_secret_version("my_secret_value"))

You should see output that resembles a hash (the exact value may not match this output):

83f8a4edb555cde4271029354395c9f4b7d79706ffa90c746e021d11

Since you did not specify a version, the latest value was retrieved.

Call the function adding the expected version number to confirm:

secret_hash(access_secret_version("my_secret_value", version_id=2))

You should see the same output as the last command:

83f8a4edb555cde4271029354395c9f4b7d79706ffa90c746e021d11

Call the function again, but this time specifying the first version:

secret_hash(access_secret_version("my_secret_value", version_id=1))

You should see a different hash this time, indicating a different output:

9a3fc8b809ddc611c82aee950c636c7557e220893560ec2c1eeeb177

By referencing the latest version of a secret in your code, you can implement secret rotation.

8. Using Secret Manager with Cloud Functions

You can make use of secrets within many parts of Google Cloud. In this section, you'll focus on Cloud Functions, Google's event-driven serverless compute offering.

If you are interested in using Python in Cloud Functions, you can follow the HTTP Google Cloud Functions in Python Codelab.

Close IPython by calling the exit function:

exit

You should be returned to your Cloud Shell:

yourname@cloudshell:~ (<PROJECT_ID>)$

Before you can begin using the Cloud Functions API, you must enable the API. Using Cloud Shell, you can enable the API with the following command:

gcloud services enable cloudfunctions.googleapis.com cloudbuild.googleapis.com

Create a new folder to build our function, creating empty files to write to:

mkdir secret-manager-api-demo
cd secret-manager-api-demo
touch main.py
touch requirements.txt

Open the code editor from the top right side of the Cloud Shell:

Navigate to the main.py file inside the secret-manager-api-demo folder. This is where you'll be putting all your code.

9. Writing a Cloud Function to access secrets

While storing and retrieving secret values from the command line or IPython terminal is useful, it's much more useful to be able to access these secrets within a function.

Using the access_secret_version function you created earlier, you can use that as a base for your Cloud Function.

Copy the following code into the main.py file:

main.py

import os

from google.cloud import secretmanager

project_id = os.environ["PROJECT_ID"]

client = secretmanager.SecretManagerServiceClient()
name = f"projects/{project_id}/secrets/my_secret_value/versions/latest"
response = client.access_secret_version(name=name)
my_secret_value = response.payload.data.decode("UTF-8")


def secret_hello(request):
    if "Again" in my_secret_value:
        return "We meet again!\n"

    return "Hello there.\n"

Note how the function is doing this work outside of the Python function itself. This will speed up your function by only accessing the secret once when the Cloud Function is instantiated, and not once for every single request.

Before you can deploy your function, you need to finalize the setup of the environment. This requires that you set up your function dependency.

Create a new file called requirements.txt, and add the google-cloud-secret-manager package to it:

requirements.txt

google-cloud-secret-manager==2.10.0

You should now have a folder containing just a main.py and a requirements.txt.

Allowing access to your secret

Before you can deploy your function, you need to allow Cloud Functions the ability to access your secret.

Switch back to the terminal:

Grant access to the Cloud Functions Service Account to access your secret:

export PROJECT_ID=$(gcloud config get-value core/project)

gcloud secrets add-iam-policy-binding my_secret_value \
    --role roles/secretmanager.secretAccessor \
    --member serviceAccount:${PROJECT_ID}@appspot.gserviceaccount.com

You should see the following output:

Updated IAM policy for secret [my_secret_value].
bindings:
- members:
  - serviceAccount:<PROJECT_ID>@appspot.gserviceaccount.com
  role: roles/secretmanager.secretAccessor
etag: BwWiRUt2oB4=
version: 1

10. Deploying your Cloud Function

Given your setup in the previous sections, you can now deploy and test your Cloud Function.

Within the folder containing just the two files you created, deploy the function:

gcloud functions deploy secret_hello \
    --runtime python39 \
    --set-env-vars PROJECT_ID=${PROJECT_ID} \
    --trigger-http \
    --allow-unauthenticated

You should see the following output (truncated):

Deploying function (may take a while - up to 2 minutes)...done.

...

entryPoint: secret_hello
httpsTrigger:
  url: https://<REGION>-<PROJECT_ID>.cloudfunctions.net/secret_hello
...
status: ACTIVE
...

Retrieve the URL of your function (httpsTrigger.url metadata) with the following command:

FUNCTION_URL=$(gcloud functions describe secret_hello --format 'value(httpsTrigger.url)')

Now, test the function can be accessed with the expected return value, by calling your function:

curl $FUNCTION_URL

You should see the following output:

We meet again!

This function references the most recent version of the secret, which was set to contain the string "Again", so this function is operating as expected.

11. Congratulations!

You learned how to use the Secret Manager API using Python!

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial:

• In the Cloud Console, go to the Manage resources page.
• In the project list, select your project then click Delete.
• In the dialog, type the project ID and then click Shut down to delete the project.

Learn more

• Secret Manager: https://cloud.google.com/secret-manager/
• Python on Google Cloud: https://cloud.google.com/python/
• Cloud Client Libraries for Python: https://googlecloudplatform.github.io/google-cloud-python/

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.