{"id":76,"date":"2022-05-12T21:52:56","date_gmt":"2022-05-12T21:52:56","guid":{"rendered":"https:\/\/blogs.oregonstate.edu\/pimode\/?p=76"},"modified":"2022-05-12T21:52:56","modified_gmt":"2022-05-12T21:52:56","slug":"accessing-google-cloud-storage","status":"publish","type":"post","link":"https:\/\/blogs.oregonstate.edu\/pimode\/2022\/05\/12\/accessing-google-cloud-storage\/","title":{"rendered":"Accessing Google Cloud Storage"},"content":{"rendered":"
\"\"<\/figure>\n\n\n

Last week, I wrote about setting up a Google Cloud Storage bucket. This week, I\u2019m writing about how to access the files in your bucket.<\/p>\n\n\n\n

Once you have a Google Cloud Storage bucket set up, you can access it through the browser. This is an easy way to upload and download files and folders from<\/em> and to<\/em> your computer, but it\u2019s not very developer friendly.<\/p>\n\n\n\n

\"\"
The browser menu allows you to upload, download, and delete files and directories.<\/figcaption><\/figure><\/div>\n\n\n\n

Two other ways to easily access the bucket are through the command line or through a script with Python (or C++, C#, Go, Java, Node.js, PHP, or Ruby). Installing the Google Cloud CLI is a snap and I found using the Python library was also straightforward. The trickiest part is setting up the authentication, so I\u2019m going to walk through some of the basics to get you started.<\/p>\n\n\n\n

To use the CLI:<\/strong><\/h3>\n\n\n\n

First, install the Google Cloud CLI, by following the instructions here<\/a>.<\/p>\n\n\n\n

Now, you\u2019ll need to initialize the CLI by following the instructions here<\/a>.<\/p>\n\n\n\n

From the command line, you have two options for authorization:<\/p>\n\n\n\n

gcloud auth login<\/strong><\/kbd><\/p>\n\n\n\n

I would recommend this option if you don\u2019t think that you\u2019ll access the bucket often. This will just open a browser window where you\u2019ll sign into your Google account using single sign-on.<\/p>\n\n\n\n

gcloud auth activate-service account<\/strong><\/kbd><\/p>\n\n\n\n

This requires setting up a service account, which takes more time, but once you\u2019re set up, it won\u2019t ask you again. Details on setting up a service account will be outlined in the steps for accessing the bucket through Python.<\/em><\/p>\n\n\n\n

Either of these options should automatically connect you with your new storage bucket unless you have other gcloud projects out there. If you have other projects, you\u2019ll have the extra step of selecting the current project.<\/p>\n\n\n\n

Once you\u2019ve set up authorization and initialized the tools, you\u2019ll be able to use gsutil<\/strong><\/kbd>. The main documentation is here<\/a>.<\/p>\n\n\n\n

Since you\u2019ll mostly want to use this for uploading and downloading files, we\u2019ll take a quick look at the cp<\/strong><\/kbd> command. More detailed instructions are here<\/a>, but let\u2019s go over a simple upload and download scenario.<\/p>\n\n\n\n

I highly recommend using the -m<\/kbd> flag if you are uploading or downloading an entire directory. Per Google\u2019s documentation, it performs a parallel multi-threaded\/multiprocessing copy<\/em>, and from my experience, it is a lot faster.<\/p>\n\n\n\n

To upload a directory (-r<\/kbd> flag for directory files)<\/strong>:<\/p>\n\n\n

\ngsutil -m cp -r local-path gs:\/\/my-storage-bucket\/storage-directory-path\n<\/pre><\/div>\n\n\n
To download a file (we just switch the local path with the bucket path):<\/strong><\/p>\n\n\n
\ngsutil -m cp -r gs:\/\/my-storage-bucket\/my-file local-directory-path\n<\/pre><\/div>\n\n\n
To access bucket files in your code (I\u2019m using Python):<\/strong><\/h3>\n\n\n\n
You\u2019ll need to set up a service account with Google Cloud Platform. We start by going to the Google Cloud console:<\/p>\n\n\n\n
https:\/\/console.cloud.google.com<\/a><\/p>\n\n\n\n
At the dashboard or home page, we want to select IAM & Admin<\/strong>. We can do this by searching with the search bar or using the hamburger menu in the left-hand corner.<\/p>\n\n\n\n
\"\"
Use the hamburger menu to select IAM & Admin and then select Service Accounts<\/figcaption><\/figure><\/div>\n\n\n\n
Click the Create Service Account<\/strong> button.<\/p>\n\n\n\n
\"\"
Click the Create Service Account button<\/figcaption><\/figure><\/div>\n\n\n\n
Add a service account name (the service account id will autofill). Then add a brief description and click Create and Continue<\/strong>.<\/p>\n\n\n\n
\"\"
Give the new Service Account a name and description<\/figcaption><\/figure><\/div>\n\n\n\n
You\u2019ll select a role — probably Editor or Owner. Then click Continue<\/strong>. Optionally, you can grant others access to this service account. Click Done<\/strong>.<\/p>\n\n\n\n
You should see your new service account listed. If you are using this for gsutil (the command line tool), you can stop here. If you want to access your bucket through Python, keep reading.<\/p>\n\n\n\n
Click on the 3 dots to the right and select Manage Keys<\/strong> from the drop-down menu.<\/p>\n\n\n\n
\"\"
Click the 3 stacked dots under Actions<\/figcaption><\/figure><\/div>\n\n\n\n
Click the Add Key<\/strong> button and select Create new key<\/strong>.<\/p>\n\n\n\n
\"\"
Click the Add Key button and select Create new key<\/figcaption><\/figure><\/div>\n\n\n\n
Select the JSON<\/strong> key type and Create<\/strong>. This will automatically download the key locally. Save this key somewhere you\u2019ll remember. You\u2019ll need the path later!<\/p>\n\n\n\n
Use your favorite way of managing environment variables. I usually use a dot-env<\/a> file. You will need to set up an environment variable like this:<\/p>\n\n\n\n
GOOGLE_APPLICATION_CREDENTIALS='\/path\/to\/my\/json\/key\/apple-pi-1234b567t2b.json'<\/code><\/p>\n\n\n\n
You\u2019ll also need to install google-storage for your python project:<\/p>\n\n\n\n
pip install google-cloud-storage<\/code><\/p>\n\n\n\n
And then import the storage library:<\/p>\n\n\n\n
from <\/strong>google.cloud import <\/strong>storage<\/code><\/p>\n\n\n\n
More information on how to use Google Storage in Python: <\/p>\n\n\n\n
https:\/\/cloud.google.com\/storage\/docs\/downloading-objects<\/a><\/p>\n\n\n\n
I find it\u2019s easier to search for what I want through Stack Overflow because the GCP docs can be hard to navigate. <\/p>\n\n\n\n
Now that you’re set up to use Cloud Storage, it’s time to get coding!<\/p>\n","protected":false},"excerpt":{"rendered":"
Last week, I wrote about setting up a Google Cloud Storage bucket. This week, I\u2019m writing about how to access the files in your bucket. Once you have a Google Cloud Storage bucket set up, you can access it through the browser. This is an easy way to upload and download files and folders from […]<\/p>\n","protected":false},"author":12249,"featured_media":77,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-76","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/posts\/76","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/users\/12249"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/comments?post=76"}],"version-history":[{"count":1,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/posts\/76\/revisions"}],"predecessor-version":[{"id":84,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/posts\/76\/revisions\/84"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/media\/77"}],"wp:attachment":[{"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/media?parent=76"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/categories?post=76"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.oregonstate.edu\/pimode\/wp-json\/wp\/v2\/tags?post=76"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}