Last week, I wrote about setting up a Google Cloud Storage bucket. This week, I’m writing about how to access the files in your bucket.
Once you have a Google Cloud Storage bucket set up, you can access it through the browser. This is an easy way to upload and download files and folders from and to your computer, but it’s not very developer friendly.
Two other ways to easily access the bucket are through the command line or through a script with Python (or C++, C#, Go, Java, Node.js, PHP, or Ruby). Installing the Google Cloud CLI is a snap and I found using the Python library was also straightforward. The trickiest part is setting up the authentication, so I’m going to walk through some of the basics to get you started.
To use the CLI:
First, install the Google Cloud CLI, by following the instructions here.
Now, you’ll need to initialize the CLI by following the instructions here.
From the command line, you have two options for authorization:
gcloud auth login
I would recommend this option if you don’t think that you’ll access the bucket often. This will just open a browser window where you’ll sign into your Google account using single sign-on.
gcloud auth activate-service account
This requires setting up a service account, which takes more time, but once you’re set up, it won’t ask you again. Details on setting up a service account will be outlined in the steps for accessing the bucket through Python.
Either of these options should automatically connect you with your new storage bucket unless you have other gcloud projects out there. If you have other projects, you’ll have the extra step of selecting the current project.
Once you’ve set up authorization and initialized the tools, you’ll be able to use gsutil. The main documentation is here.
Since you’ll mostly want to use this for uploading and downloading files, we’ll take a quick look at the cp command. More detailed instructions are here, but let’s go over a simple upload and download scenario.
I highly recommend using the -m flag if you are uploading or downloading an entire directory. Per Google’s documentation, it performs a parallel multi-threaded/multiprocessing copy, and from my experience, it is a lot faster.
To upload a directory (-r flag for directory files):
gsutil -m cp -r local-path gs://my-storage-bucket/storage-directory-path
To download a file (we just switch the local path with the bucket path):
gsutil -m cp -r gs://my-storage-bucket/my-file local-directory-path
To access bucket files in your code (I’m using Python):
You’ll need to set up a service account with Google Cloud Platform. We start by going to the Google Cloud console:
https://console.cloud.google.com
At the dashboard or home page, we want to select IAM & Admin. We can do this by searching with the search bar or using the hamburger menu in the left-hand corner.
Click the Create Service Account button.
Add a service account name (the service account id will autofill). Then add a brief description and click Create and Continue.
You’ll select a role — probably Editor or Owner. Then click Continue. Optionally, you can grant others access to this service account. Click Done.
You should see your new service account listed. If you are using this for gsutil (the command line tool), you can stop here. If you want to access your bucket through Python, keep reading.
Click on the 3 dots to the right and select Manage Keys from the drop-down menu.
Click the Add Key button and select Create new key.
Select the JSON key type and Create. This will automatically download the key locally. Save this key somewhere you’ll remember. You’ll need the path later!
Use your favorite way of managing environment variables. I usually use a dot-env file. You will need to set up an environment variable like this:
GOOGLE_APPLICATION_CREDENTIALS='/path/to/my/json/key/apple-pi-1234b567t2b.json'
You’ll also need to install google-storage for your python project:
pip install google-cloud-storage
And then import the storage library:
from google.cloud import storage
More information on how to use Google Storage in Python:
https://cloud.google.com/storage/docs/downloading-objects
I find it’s easier to search for what I want through Stack Overflow because the GCP docs can be hard to navigate.
Now that you’re set up to use Cloud Storage, it’s time to get coding!
