Amazon S3

• S3 is a universal namespace, that is, names must be unique globally.
• S3 creates a DNS: https://s3-<region>.amazonaws.com/<bucket>.
• When you upload a file to S3 you will receive an HTTP 200 code if the upload was successful.

S3 provides simple key, value store. Files can be from 0 bytes to 5 TB. S3 gives unlimited storage.

Objects consist of the following:

• Key (name)
• Value (data: a sequence of bytes).
• Version ID
• Metadata (data about the data. e.g. date)
• Subresources (bucket-specific configuration)
  • Bucket Policies, Access Control Lists (ACL).
  • Cross Origin Resource Sharing (CORS)
  • Transfer Acceleration

You can load files to S3 much faster by enabling multipart upload.

Objects are stored in Buckets. A Bucket is similar to a folder.

Buckets reside in a certain AWS region.

• Read after write consistency for PUTS of new objects.
• Eventual consistency for overwrite PUTS and DELETE (can take some time to propagate).

Updates to S3 are atomic. You will either get the old file or the new file. You won't get partial or corrupted data.

Data is spread across multiple devices and facilities; S3 is designed to withstand failure.

• Built for 99.99% availability for the S3 platform.
• Amazon guarantee 99.9% availability (note 3x9).
• Amazon guarantees 99.999999999% durability for S3 information (note 11x9). This is the eleven 9s durability agreement.

• By default, all newly created buckets are PRIVATE.
• You can set up access control to your buckets using;
  • Bucket Policies
    • Applied at the bucket level.
  • Access Control Lists (ACL)
    • Applied at an object level.
• S3 buckets can be configured to create access logs which log all requests made to the S3 bucket. This logs can be setup to be saved to another bucket. Of course, logging is important for security.

• In Transit;
  • SSL/TLS (HTTPS)
• At Rest;
  • Server Side Encryption
    1. S3 Managed Keys (SSE-S3)
       • Most common method. Uses AES-256.
    2. AWS Key Management Service, Managed Keys (SSE-KMS)
       • More expensive method. Provides an extra layer of protection using an envelope key (which has separate permissions). This key protects your encryption key. In addition, SSE-KMS provides an audit trail of when keys were used and by whom. Lastly, you can also manage your keys yourself.
    3. Customer Provided Keys (SSE-C)
       • You manage the keys but Amazon manages the encryption.
• Client Side Encryption
  • Encrypt on your computer and then upload.

• S3
  • Durable, immediately available, frequently accessed.
  • 99.99% availability, 99.99999999999% durability, stored redundantly across multiple devices in multiple facilities and is designed to sustain the loss of 2 facilities concurrently.
• S3 - IA (Infrequently Accessed)
  • Durable, immediately available, infrequently accessed.
  • Lower fee than S3, but you are charged a retrieval fee.
• S3 - One Zone IA
  • Similar to IA, but data is stored in a single Availability Zone. Durability is the same, but we have 99.5% availability.
  • Cost is 20% less than regular S3 - IA.
• RRS (Reduced Redundancy Storage)
  • Data that is easily reproducible, such as thumnails etc.
  • Designed to provide 99.99% durability and 99.99% availability of objects over a given year. We can only lose 1 facility.
• Glacier
  • Very cheap, but used for archival only. It takes 3-5 hours to restore from Glacier. First byte latency is minutes or hours instead of milliseconds like S3 services.
  • Costs as little as $0.01 per gigabyte per month.

In S3, costs can come from

• Storage per GB.
• Requests (Get, Put, Copy, etc)
• Storage Management Pricing (cost tracking)
• Data Management Pricing (mostly just transferring out of S3)
• Transfer Acceleration

Amazon S3 Transfer Acceleration enables fast, easy, and secure transfers of files over long distances between your end users and an S3 bucket. Transfer Acceleration takes advantage of Amazon CloudFront's globally distributes edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path. Note that Transfer Acceleration does not provide caching, it's only for data transfer.

A distinct URL is used for transfer acceleration: <bucket>.s3-accelerate.amazonaws.com.

You can enable Transfer Acceleration by clicking on a bucket and selecting Properties -> Transfer acceleration.

You can tag buckets and objects. Individual objects don't inherit the tag from the bucket.

"Versioning is a means of keeping multiple variants of an object in the same bucket" (https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html). It's a great backup tool.

Every time a file changes there will be a copy of both the original and the change. Of course, versioning can increase S3 costs. Even deleting is a version (a delete marker is made)!

Once enabled, versioning cannot be disabled. Versioning can only be suspended after turned on.

Versioning integrates with lifecycle rules.

MFA Delete helps stop people from accidentally deleting a version. This multi-factor authentication provides an extra layer of security.

Cross-region replication enables automatic and asynchronous copying of objects across buckets in different AWS regions. Enable in the S3 console by clicking on a bucket then Management then Replication. Note that cross-region replication requires versioning enabled on both the source and destination buckets. Regions must be unique.

Note that only new objects will be replicated. Files in an existing bucket are not replicated automatically.

Currently, cross region replication cannot replicate to multiple buckets or use daisy chaining.

Version deletes are not replicated.

Click a Bucket then select Management->/Lifecycle/. Here you can define lifecycle rules to automatically manage S3 objects. The following actions can be done:

• Transition to Standard - Infrequent Access Storage Class (after a minimum of 30 days. Data must also be at least 128KB).
• Archive to the Glacier Storage Class (after any number of days or a minimum of 30 days after transition to Standard-IA).
• Permanently delete.

Lifecycle management can be used in conjunction with vertioning. It can be applied to current versions and previous versions. Can also be used without versioning.

S3 allows static website hosting. These websites may contain client-side scripts. Dynamic, service-side processing is not supported.

The URL for static website hosting looks like: <bucket>.s3-website-<region>.amazonaws.com. For example http://elliotp.s3-website-us-east-1.amazonaws.com.

S3 static website hosting scales automatically and is very cheap.

Cross-origin resource sharing (CORS) defines a way for client web applications that are loaded in one domain to interact with resources in a different domain. By default resources in one bucket cannot access resources located in another. You can configure your bucket to allow cross-origin requests by creating a CORS configuration.

If your S3 buckets are routinely receiving >100 PUT/LIST/DELETE or >300 GET requests per second, then there are some best practice guidelines that will help optimize S3 performance.