Universal Data Tool
  • Universal Data Tool Docs
  • Installation
  • Running On-Premise
  • Collaborative Labeling
  • Building and Labeling Datasets
    • Image Segmentation
    • Image Classification
    • Text Classification
    • Named Entity Recognition
    • Entity Relations / Part of Speech Tagging
    • Audio Transcription
    • Data Entry
    • Video Segmentation
    • Composite Interfaces
    • Landmark / Pose Annotation
  • Importing Data
    • Upload or Open Directories
    • Import File URLs
    • Import COCO Images
    • Import from Google Drive
    • Import from AWS S3 Bucket
    • Import from CSV or JSON
    • Import using AWS Cognito
    • Import Text Snippets
  • The Format .udt.json
    • What is the .udt.json format?
    • What is the .udt.csv format?
  • Machine Learning
    • Jupyter Notebook Integration
    • Import Datasets into Pandas
    • Fast.ai
      • Fast.ai Image Classification
      • Fast.ai Image Segmentation
  • Integrate with Any Web Page
    • Integrate with the Javascript Library
    • Getting Started with React
  • Train your Workforce
    • Getting Started with UDT Courses
  • Frequently Asked Questions
Powered by GitBook
On this page
  • Overview
  • 1. Configure Bucket
  • 2. Add IAM Credentials
  • 3. Add Keys to the Universal Data Tool
  • 4. Browse Buckets and Import
  • Bonus: Import via Uploading to S3

Was this helpful?

  1. Importing Data

Import from AWS S3 Bucket

Import samples from an S3 bucket directly into the Universal Data Tool!

PreviousImport from Google DriveNextImport from CSV or JSON

Last updated 4 years ago

Was this helpful?

Overview

If you have an S3 Bucket full of data to label, we'll need to do a couple of things to make that data accessible to UDT.

  1. Configure the bucket so the files can be loaded

  2. Create a user (or use an existing user) that can access the bucket

  3. Import the data into S3

It should only take a couple minutes, let's do it!

1. Configure Bucket

First we need to make sure our files can be loaded from the web. To do this, we need to add a CORs policy in our Bucket Permissions. We can do this from the web on the .

You can paste these CORs permissions in to make files web-accessible.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

After you save that, you can pick the directories you'd like to be accessible by clicking on the directory, then clicking Actions > Make Public If your entire bucket is public, you can skip this step.

2. Add IAM Credentials

Next, we need to get the keys to allow the UDT to browse for the S3 bucket files. We can do this by copying our AWS Access Key and Secret Access Key.

It's a good idea to limit the permissions of the user you're getting the access keys from, that way this key can only be used for it's intended purpose!

Navigate to the IAM service and select (or create) a user. The user must have permissions to access S3 buckets. Then click Create Access Key to create your keys!

One simple, but dangerous way to give the AWS permissions is to give the S3FullAccess permission, as show below. Fine-grained permissions are more secure!

3. Add Keys to the Universal Data Tool

4. Browse Buckets and Import

You can now create a new UDT Dataset and navigate to Samples > Import from S3 (which will be enabled). You'll be able to select from all the buckets accessible to this user.

Bonus: Import via Uploading to S3

You can also upload to S3 directly from the UDT, after doing so, your files will automatically be added to your UDT dataset.

Navigate to or open the UDT. Click "Add Authentication" and paste your keys.

AWS Buckets page
udt.dev
Paste the CORs configuration in here
This will create URLs for each item in your S3 bucket.
Clicking Create Access Key will create the keys you need
Add Authentication Dialog
You can select any buckets accessible to that user.
You can then import files from S3 directories.
Use the Upload to S3 to directly add files to UDT!