Prerequisite: Upload data to S3

Credentials

To upload to S3, these AWS credentials are required:

• BUCKET_NAME

• Programatic access:

  • ACCESS_KEY

  • SECRET_KEY

• AWS UI Dashboard access:

  • Username

  • Password

💡 It is recommended to upload (large) files via CLI and view uploaded files via AWS UI

Option 1: Upload via CLI

Prerequisite

• Set up AWS CLI

  • When run aws configure, you may set:

    • region: ap-southeast-1

    • output format: as default

AWS official link: Getting started with the AWS CLI

Usage

• Use sync command to copy from local files to bucket (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)

# Upload current directory (recursively) to 
# location `s3://client-linh/upload/3d`
aws s3 sync . s3://client-linh/upload/2d

# Upload directory `Downloads` (recursively) to
# location `s3://client-linh/upload/2d/batch1`
aws s3 sync Downloads s3://client-linh/upload/2d/batch1

ℹ️ You may resume the uploading anytime using sync command. It only uploads the local file if it does not exist on S3 bucket or if the local file is different from the remote file on the S3 bucket.

⚠️ Use --delete parameter to delete files that exists in the bucket but not on the local source. Please be careful when use this --delete parameter because it may accidentally delete your local files which don't exist in the bucket.

Option 2: Upload and view files via AWS UI

Usage

1. Access the link to the bucket: https://s3.console.aws.amazon.com/s3/buckets/<BUCKET_NAME>

• For example, if BUCKET_NAME=test-bucket, then the link is https://s3.console.aws.amazon.com/s3/buckets/test-bucket

2. You are prompted to sign-in

• Choose “IAM user” and input linh-ai as account alias

• Provide “Username” and “Password”

• When you are inside AWS S3 dashboard, you are able to view the uploaded objects

• Click “Upload” to upload files to S3

Option 3: Upload via Python script

Prerequisite

• Install AWS S3 boto3 library (https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)

Examples

• Upload files: Uploading files - Boto3 Docs 1.21.44 documentation

• Upload directory: Upload folder contents to AWS S3

Troubleshoot

• If uploaded files are too large, it is recommended to use AWS CLI. But if you still want to upload via Python script, you may tweak some settings for multipart uploads (https://stackoverflow.com/questions/50105094/python-upload-large-files-s3-fast)