Google Cloud Storage

Connecting a Google Cloud Storage Data Source

What is Google Cloud Storage?

Google Cloud Storage (GCS) is a popular way to store data in the cloud. Like Amazon S3, data is stored in "buckets". Sense supports syncing data sets from both public and private buckets. You will need access to the Google Cloud Platform to take advantage of GCS.

Creating a bucket

For instructions on setting up a Google Cloud Storage bucket, see here.

Bucket Permissions

GCS buckets have varying levels of permissions. In general, public buckets are available to everyone without credentials to access them, whereas private buckets require Service Account Keys to use. Buckets are private by default, but you can make them public by following these steps. A private bucket will need to grant authenticated users read access in order for Sense to use it. A service account will need to be added as a member of the bucket with a "Storage Object Viewer" role. Private buckets will require credentials tied to this service account. You can see more on creating a Service Account Key JSON file.

Adding a Google Cloud Storage Data Source

Once you have a GCS bucket to connect to, you can use the "Sources" tab to sync the data.

  1. In the "Sources" tab of your project, click the "Google Cloud Storage" option

  2. Enter a name for your data source in the Source Name field. This is how it will display in your list of data sources. Example, Chess Images

  3. Enter your GCS Bucket Name. This is the globally-unique name for your bucket, so be sure this is spelled correctly with no trailing spaces. Example, my-images

  4. Enter an optional folder path in the Folder name field. Only files from this directory will be synced. Ex: images/cows

  5. If this is a private bucket, enable the "Private bucket" toggle. This will display Upload Service Account Key JSON File button which allows you to upload the JSON file with credentials to access a private bucket. More info

  6. Enter an optional File Name Filter - You can use this field to filter files within a data source by name. Check out the File Name Filter Guide for more information on how to use this filter. Example, *-june-*

  7. If there are videos in your data source that you want to use, use the Video Sampling settings to choose how many frames per second you want the video to be split into. Supported files will be processed by default. Select "Do not use Video Data" to ignore video files.

  8. When you are finished, click "Add Source". If there are any errors, they will be displayed on the fields in red.

The bucket will begin syncing momentarily. Images will be synced to the GCS source and the count will increase.

Video frames are synced to your Upload source which represents Sense Storage.