Amazon S3

Connecting an Amazon S3 Data Source

What is Amazon S3?

Amazon Simple Storage Service (S3) is a popular solution for storing data in the cloud. Amazon S3 stores resources in "buckets". Sense supports syncing data sets from both public and private buckets. You will need access to an Amazon Web Services (AWS) account to take advantage of Amazon S3.

Read more about Creating a bucket in AWS.

Adding an Amazon S3 Data Source

Once you've uploaded your data set to your bucket, you can use Sense to add your Amazon S3 data source.

  1. In the "Sources" tab of your project, click the "Amazon S3" option

  2. Enter a name for your data source in the Data Source Name field. This is how it will display in your list of data sources. Example, Chess Images.

  3. Enter your Bucket Name. This is the globally-unique name for your bucket, so be sure this is spelled correctly with no trailing spaces. Example, my-images

  4. Enter an optional folder path in the Folder name field. Only files from this directory will be synced. Ex: images/cows

  5. If your bucket is private you will need to authenticate it by entering an Access Key ID and a Secret Access Key in their respective fields.

  6. Under "Filters", enter an optional File Name Filter. You can use this field to filter files within a data source by name. Check out the File Name Filter Guide for more information on how to use this filter. Example, *-june-*

  7. If there are videos in your data source that you want to use, use the Video Sampling settings to choose how many frames per second you want the video to be split into. Supported files will be processed by default. Select "Do not use Video Data" to ignore video files.

  8. When you are finished, click "Add Source". If there are any errors, they will be displayed on the fields in red.

The bucket will begin syncing momentarily. Images will be synced to the Amazon S3 source and the count will increase.

Video frames are synced to your Upload source which represents Sense Storage.

Creating a bucket

For instructions on setting up an Amazon S3 bucket, see here.

Public vs. Private buckets

AWS S3 buckets have varying levels of permissions. In general, public buckets are available to everyone, whereas private buckets require security tokens to access. If your bucket is private, be sure the "Public Bucket" is unchecked when adding the data source. You will need to add your security credentials using the Access Key ID and Secret Access Key fields. Buckets are private by default, but you can easily make them public during creation. For more detailed information on how to give buckets public read access, see this page. Note: if you are using a public bucket, you have to grant Public Access to Everyone along with List objects access. For more information, see step 4c here.