In this codelab, you will use gsutil to create a bucket and perform operations on objects. gsutil is a command-line tool that you can use to work with Google Cloud Storage. The gsutil tool has commands such as mb and cp to perform operations. Each command has a set of options that you can use to customize settings further.

In this codelab, you will use gsutil to create a bucket and perform operations on objects.

For complete information about the gsutil command-line options, see:

What you'll learn

What you'll need

To complete this lab, you need:

Create a project

Open the Cloud Platform Console. Click Select a project, and then click the + icon to create a project.

In the New Project dialog, for Project name, type GsutilProject. Click Create.

Download the code

Click the Activate Google Cloud Shell button at the top of the Cloud Platform Console.

In the Cloud Shell session, execute the following command to clone a git repository that contains sample data for this lab.

git clone https://github.com/GoogleCloudPlatform/training-data-analyst

To switch to the blogs directory, execute the following command:

cd training-data-analyst/blogs

Create a bucket

To create a bucket with a unique bucket name and Multi-Regional storage class, execute the following command:

gsutil mb -c multi_regional gs://mybucket-<unique name>/

Use this bucket name for the remaining commands in the lab.

Upload objects to your bucket

To upload objects to your bucket with the bucket's default storage class, execute the following command:

gsutil -m cp -r endpointslambda gs://mybucket-<unique name>

With the -m option, gsutil performs a parallel (multi-threaded/multi-processing) copy. The -r option allows gsutil to recurse through directories.

List objects

To list objects in your bucket, execute the following command:

gsutil ls gs://mybucket-<unique name>/*

The output will show a list of files in the bucket.

Sync changes with bucket

Rename and delete files using the following commands:

mv endpointslambda/Apache2_0License.txt endpointslambda/Apache2_0LicenseFile.txt

rm endpointslambda/aeflex-endpoints/app.yaml

To synchronize local changes with the bucket, execute the following command:

gsutil -m rsync -d -r endpointslambda gs://mybucket-<unique name>/endpointslambda

The -d option deletes the Readme.md file from the bucket. The -r option runs the command recursively on directories.

To verify that the bucket is now in sync with your local changes, list the files in the bucket again by executing the following command:

gsutil ls gs://mybucket-<unique name>/*

Make objects public

To allow public access to all files under the endpointslambda folder in your bucket, execute the following command:

gsutil acl ch -r -u AllUsers:R gs://mybucket-<unique name>/endpointslambda

Because we are not using the -m option in this command, you will see a note in the command output stating that the command might run faster using the -m option.

To confirm that the files are viewable by the public, open the following link in a new incognito or private browser window:

http://storage.googleapis.com/mybucket-<unique name>/endpointslambda/Apache2_0LicenseFile.txt

This URL uses the Cloud Storage API link to view the object without authentication. For more information, see Accessing Public Data.

Copy with different storage class

To copy a file with the Nearline storage class instead of the bucket's default Multi-regional storage class, execute the following command:

gsutil cp -s nearline ghcn/ghcn_on_bq.ipynb gs://mybucket-<unique name>

Check storage classes

To check the storage classes and view other detailed information about the objects in your bucket, execute the following command:

gsutil ls -Lr gs://mybucket-<unique name> | more

Press the space key to continue viewing the rest of the command's output.

The output shows that the ghcn_on_bq.ipynb object has NEARLINE storage class while the other objects have MULTI_REGIONAL storage class.

Command output

gs://mybucket-xx-04222017/ghcn_on_bq.ipynb:
    Creation time:          Sat, 22 Apr 2017 13:23:40 GMT
    Update time:            Sat, 22 Apr 2017 13:23:40 GMT
    Storage class:          NEARLINE
    Content-Length:         980176
    Content-Type:           application/octet-stream
...
gs://mybucket-xx-04222017/endpointslambda/:
gs://mybucket-xx-04222017/endpointslambda/Apache2_0LicenseFile.txt:
    Creation time:          Sat, 22 Apr 2017 12:15:12 GMT
    Update time:            Sat, 22 Apr 2017 12:30:14 GMT
    Storage class:          MULTI_REGIONAL
    Content-Length:         10178
    Content-Type:           text/plain
...

Delete your bucket

Before deleting a bucket, you must first delete all objects in the bucket. To delete all objects, execute the following command:

gsutil -m rm gs://mybucket-<unique name>/**

To delete the bucket, execute the following command:

gsutil rb gs://mybucket-<unique name>

Delete the project

In the Product & Services menu, click IAM & Admin.

Click Settings, then click the Delete.

In the Shut down project dialog, read the notes and then, type the project ID.

Click Shut down.

Close the Cloud Platform Console.

You have now learned how to perform operations on Cloud Storage buckets and objects!