Spotted a bug? Have a great idea? Help us make great!

In this codelab, you'll learn how to use some advanced features of Google Cloud Security and Privacy APIs, including:

  • Setting up a secure Cloud Storage bucket
  • Managing keys and encrypted data using Key Management Storage
  • Viewing Cloud Storage audit logs

You'll take abridged data from the Enron Corpus, encrypt it and load it into Cloud Storage.

What you'll learn

  • How to encrypt data and manage encryption keys using Key Management Service (KMS).

What you'll use

  • A Google Cloud Project with billing enabled.
  • Google Cloud Storage.
  • Google Cloud SDK.

Self-paced environment setup

If you don't already have a Google Account (Gmail or Google Apps), you must create one. Sign-in to Google Cloud Platform console ( and create a new project:

Screenshot from 2016-02-10 12:45:26.png

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document).

New users of Google Cloud Platform are eligible for a $300 free trial.

Launch Cloud Shell

While Google Cloud and KMS can be operated remotely from your laptop, in this codelab we will be using Google Cloud Shell, a command line environment running in the Cloud.

This Debian-based virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. This means that all you will need for this codelab is a browser (yes, it works on a Chromebook).

To activate Google Cloud Shell, from the developer console simply click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):


Then accept the terms of service and click the "Start Cloud Shell" link:


Screen Shot 2017-06-14 at 10.13.43 PM.png

Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your PROJECT_ID :

gcloud auth list

Command output

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)
gcloud config list project

Command output

project = <PROJECT_ID>

If for some reason the project is not set, simply issue the following command :

gcloud config set project <PROJECT_ID>

Looking for your PROJECT_ID? Check out what ID you used in the setup steps or look it up in the console dashboard:


IMPORTANT: Finally, set the default zone and project configuration:

gcloud config set compute/zone us-central1-f

You can choose a variety of different zones. Learn more in the Regions & Zones documentation.

In order to store the data in this codelab, you'll need your own Cloud Storage bucket. Pick a name for it, such as YOUR_NAME_enron_emails. Cloud Storage bucket names have to be unique across all buckets, so change YOUR_NAME to your own name or some other unique identifier. For more information, see the Cloud Storage bucket naming guidelines.


Create the bucket by running the following commands in your shell:

gsutil mb gs://${BUCKET_NAME}

After your bucket has been created, you are ready to go. Running this command should also help to verify that you've got the gsutil command line client setup correctly, authentication is working, and you have write access to the cloud project you're operating under.

The Enron Corpus is a large database of over 600,000 emails generated by 158 employees of the Enron Corporation. This data has been copied to the GCS bucket gs://enron_emails/.

Let's download one of the source files locally so that we can see what it looks like. Run the command to copy down an email text file:

gsutil cp gs://enron_emails/allen-p/inbox/1. .

Now tail the downloaded file to verify the email text is there:

tail 1.

This should display the contents of the plaintext mail file. There are two types of files we'll be looking at for this codelab: plaintext mail files and image files. If you're interested, use the same mechanism to check out what is in those other files.

Before you can use KMS, you need to enable KMS in your project. This can be done in the using the gcloud command-line utility. Run the following command in your shell:

gcloud services enable

Cloud KMS is now enabled for your project!

In order to encrypt the data, you need to create a key ring and an encryption key. Key rings are useful for grouping keys. Keys can be grouped by environment (like test, staging, and prod) or by some other conceptual grouping. For this codelab, our key ring will be called test and our encryption key will be called codelab.


Execute the gcloud command to create the key ring. You will specify the location as global, but this can also be a specific region.

gcloud kms keyrings create $KEYRING_NAME --location global

Next, using the new key ring, create an encryption key named codelab.

gcloud kms keys create $CRYPTOKEY_NAME --location global \
      --keyring $KEYRING_NAME \
      --purpose encryption

You've created a key ring and encryption key! Open the Encryption Keys Web UI and view the resources you've created. This is available under IAM & Admin in the main menu:

The Key Management Web UI allows you to view and manage your encryption keys and key rings. You will use this UI later when you manage permissions.

Next, let's actually try to encrypt some data! Take the contents of the email we previously looked at and base64 encode it.

PLAINTEXT=$(cat 1. | base64 -w0)

Using the encrypt endpoint, you can send the base64-encoded text you want to encrypt to the specified key. The response will be a JSON payload containing the encrypted text in the attribute ciphertext.

curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
  -d "{\"plaintext\":\"$PLAINTEXT\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type: application/json"

Now that your data is encrypted, you can save it to a file and upload it to your GCS bucket. To grab the encrypted text from the JSON response and save it to a file, you will use the command-line utility jq. The response from the previous call can be piped into jq, which can parse out the ciphertext property to the file 1.encrypted.

curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
  -d "{\"plaintext\":\"$PLAINTEXT\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type:application/json" \
| jq .ciphertext -r > 1.encrypted

To verify the encrypted data can be decrypted, call the decrypt endpoint to verify the decrypted text matches the original email. The encrypted data has information on which key version was used to encrypt it, so the specific version is never supplied to the decrypt endpoint.

curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:decrypt" \
  -d "{\"ciphertext\":\"$(cat 1.encrypted)\"}" \
  -H "Authorization:Bearer $(gcloud auth application-default print-access-token)"\
  -H "Content-Type:application/json" \
| jq .plaintext -r | base64 -d

Now that you have verified the text has been encrypted successfully, upload the encrypted file to your GCS bucket.

gsutil cp 1.encrypted gs://${BUCKET_NAME}

In KMS, there are two major permissions to focus on. One permissions allows a user or service account to manage KMS resources, the other allows a user or service account to use keys to encrypt and decrypt data.

The permission to manage keys is cloudkms.admin, and allows anyone with the permission to create key rings and create, modify, disable, and destroy encryption keys. The permission to encrypt and decrypt is cloudkms.cryptoKeyEncrypterDecrypter, and is used to call the encrypt and decrypt API endpoints.

For this exercise, you will use the current authorized user to assign IAM permissions. To get the current authorized user, run the command below.

USER_EMAIL=$(gcloud auth list --limit=1 2>/dev/null | grep '@' | awk '{print $2}') 

Next, you will assign that user the ability to manage KMS resources. Run the following gcloud command to assign the IAM permission to manage the key ring you just created:

gcloud kms keyrings add-iam-policy-binding $KEYRING_NAME \
    --location global \
    --member user:$USER_EMAIL \
    --role roles/cloudkms.admin

Since encryption keys belong to key rings, and key rings belong to projects, a user with a specific role or permission at a higher level in that hierarchy inherits the same permissions on the child resources. That is, a user who has the role of owner on a Project is also an owner on all the key rings and encryption keys in that project. Similarly, if a user is granted the cloudkms.admin role on a key ring, they have the associated permissions on the encryption keys in that key ring.

Without the permissions provided by the cloudkms.cryptoKeyEncrypterDecrypter role, the authorized user will not be able to use the keys to encrypt or decrypt data. Run the following gcloud command to assign the IAM permission to encrypt and decrypt data for any encryption key under the key ring you created:

gcloud kms keyrings add-iam-policy-binding $KEYRING_NAME \
    --location global \
    --member user:$USER_EMAIL \
    --role roles/cloudkms.cryptoKeyEncrypterDecrypter

Now you can view the assigned permissions in the Key Management Web UI. Click the key ring.

This will open up a menu where you can see the accounts and permissions for the KeyRing you just added.

Now that you have an understanding of how to encrypt a single file, and have permission to do so, you can run a script to backup all files in a directory. For this example, we will copy all emails for allen-p, encrypt them, and upload them to a Cloud Storage bucket. First, copy all emails for allen-p into your current working directory:

gsutil -m cp -r gs://enron_emails/allen-p .

Now copy and paste the following into the Cloud Shell terminal to backup and encrypt all the files in the allen-p directory to your Cloud Storage bucket:

FILES=$(find $MYDIR -type f -not -name "*.encrypted")
for file in $FILES; do
  PLAINTEXT=$(cat $file | base64 -w0)
  curl -v "$DEVSHELL_PROJECT_ID/locations/global/keyRings/$KEYRING_NAME/cryptoKeys/$CRYPTOKEY_NAME:encrypt" \
    -d "{\"plaintext\":\"$PLAINTEXT\"}" \
    -H "Authorization:Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type:application/json" \
  | jq .ciphertext -r > $file.encrypted
gsutil -m cp $MYDIR/*/*.encrypted gs://${BUCKET_NAME}

This script loops over all the files in a given directory, encrypts them using the KMS API, and uploads them to Google Cloud Storage.

After the script completes, you can view the encrypted files in the Google Cloud Storage UI. You should see something like this:

Google Cloud Audit Logging consists of two log streams, Admin Activity and Data Access, which are generated by Google Cloud Platform services to help you answer the question of "who did what, where, and when?" within your Google Cloud Platform projects.

To view the activity for any resource in KMS, click the key ring and select View Activity. This will take you to the Cloud Activity UI, where you should see the creation and all modifications made to the KeyRing.

You've now encrypted and uploaded data using KMS and Cloud Storage!


Let's release compute resources created during the lab. If you no longer need it, go ahead and delete the Cloud Storage bucket you created in the beginning of the lab.

gsutil rm -r gs://${BUCKET_NAME}

As KMS resources (encryption keys and key rings) are permanent, there is no way to remove them. However, you can destroy the key version used in this project so it cannot be used again:

gcloud kms keys versions destroy 1 \
  --location=global \

What we've covered

  • Using IAM to manage KMS permissions.
  • Using KMS to encrypt data.
  • Using GCS to store encrypted data.
  • Using Cloud Audit Logging to view all activity for keys and key rings.

Next steps

  • Use KMS to encrypt a column in a database like BigQuery or CloudSQL.
  • Set a rotation schedule on your KMS keys for even better security.

Learn more

Give us your feedback

  • Feel free to use the link at the bottom left of this page to file issues or share feedback!