Amazon S3 Multipart Upload and Transfer Acceleration – SAP-C02 Study Guide

Purpose of This Lesson

Amazon S3 provides two features for improving object transfer performance and reliability:

• Multipart Upload divides a large object into independently uploaded parts.
• S3 Transfer Acceleration uses CloudFront edge locations and the AWS global network to improve long-distance transfers into an S3 bucket.

These features solve different problems. Multipart Upload primarily improves resilience, parallelism, and recovery from interrupted uploads. Transfer Acceleration primarily addresses network distance between the client and the S3 bucket.

Key Concepts

S3 Multipart Upload

Multipart Upload allows an object to be split into multiple parts that can be uploaded:

• Independently
• In parallel
• In any order
• Using separate requests

After all parts are uploaded, S3 assembles them into a single object when the application sends a completion request.

Multipart Upload is useful when:

• Objects are large.
• Network connections are unreliable.
• Uploads need to run in parallel.
• A failed upload should resume from the failed part instead of restarting the entire object.
• Applications need more control over the upload process.

Important size boundaries:

• Multipart Upload supports objects from 5 MB through 5 TB.
• It is generally recommended for objects larger than approximately 100 MB.
• It is required for objects larger than 5 GB.

The upload workflow is:

1. Initiate a multipart upload and receive an upload ID.
2. Upload each part, specifying the upload ID and part number.
3. Track uploaded parts as needed.
4. Complete the multipart upload, causing S3 to assemble the final object.
5. Abort incomplete uploads when they are no longer needed.

An incomplete multipart upload consumes storage and can create unexpected costs. Configure an S3 Lifecycle rule to abort incomplete multipart uploads after an appropriate number of days.

The AWS CLI, AWS SDKs, and high-level S3 tools can perform multipart uploads. Many AWS-managed tools automatically use multipart behavior for sufficiently large files.

S3 Transfer Acceleration

S3 Transfer Acceleration provides an alternate endpoint for uploads and downloads through CloudFront edge locations. Instead of sending traffic directly across the public internet to the S3 bucket’s Region, the client connects to a nearby edge location. Traffic then travels across the AWS global network to the destination bucket.

The normal data path is conceptually:

Client -> Internet -> S3 bucket Region

With Transfer Acceleration:

Client -> Nearby CloudFront edge location -> AWS global network -> S3 bucket

The accelerated endpoint uses the S3 acceleration hostname format, such as:

bucket-name.s3-accelerate.amazonaws.com

Transfer Acceleration must be enabled on the bucket before the accelerated endpoint can be used. Clients can still access the bucket through the standard S3 endpoint when acceleration is not appropriate.

Transfer Acceleration is most relevant when:

• Clients are geographically distant from the bucket’s Region.
• Uploads are large or frequent.
• Internet routing to the bucket Region is inefficient.
• The application cannot practically move the bucket closer to every client.

Transfer Acceleration is not automatically faster. Performance depends on the client’s location, the destination Region, network conditions, and the object’s characteristics. AWS provides an S3 Transfer Acceleration speed comparison tool to test representative locations before adopting the feature.

It also has an additional transfer charge. Validate the performance benefit and cost rather than enabling it by default.

Combining the Features

Multipart Upload and Transfer Acceleration can be used together. For example, a globally distributed application can upload a large object in parallel parts through the S3 accelerated endpoint.

However, they address different bottlenecks:

• Multipart Upload: object size, failure recovery, parallelism, and resumability.
• Transfer Acceleration: geographic distance and network path quality.

Using both does not guarantee a performance improvement. Testing is still necessary.

Architecture Decision Guide

Requirement	Preferred approach	Reason
Upload an object larger than 5 GB	Multipart Upload	Required for objects above the single-upload size limit.
Upload a large object over an unreliable connection	Multipart Upload	Failed parts can be retried without restarting the entire object.
Improve throughput for a large upload from a distant client	Multipart Upload, potentially combined with Transfer Acceleration	Multipart Upload provides parallelism; acceleration may improve the long-distance network path.
Upload from a client near the S3 bucket Region	Standard S3 endpoint, with Multipart Upload for large files	Transfer Acceleration may add cost without meaningful benefit.
Many globally distributed clients upload to one centralized bucket	Evaluate Transfer Acceleration	Edge ingress may reduce the impact of geographic distance.
Optimize cost when transfer speed is acceptable	Standard S3 endpoint	Avoids the additional Transfer Acceleration charge.
Clean up abandoned multipart sessions	S3 Lifecycle rule for incomplete multipart uploads	Prevents storage and cost leakage.
Resume an interrupted large-file upload	Multipart Upload	Individual parts can be retried or resumed.

Exam-Relevant Takeaways

• Multipart Upload supports objects from 5 MB to 5 TB.
• Objects larger than 5 GB must use Multipart Upload.
• Multipart parts can be uploaded independently, in parallel, and in any order.
• A multipart upload is not complete until S3 receives the completion request.
• Incomplete multipart uploads should be aborted manually or through an S3 Lifecycle configuration.
• Transfer Acceleration uses CloudFront edge locations, but it is an S3 feature and uses an S3 acceleration endpoint.
• Transfer Acceleration is enabled at the bucket level.
• Acceleration is primarily useful for clients far from the bucket’s Region.
• Transfer Acceleration is not guaranteed to be faster and incurs an additional charge when used.
• Test with representative client locations and object sizes before making Transfer Acceleration a standard architecture choice.

Common Exam Traps

• Confusing Multipart Upload with Transfer Acceleration: Multipart Upload improves the upload process itself; Transfer Acceleration changes the network entry path.
• Assuming Transfer Acceleration is always faster: A nearby client, poor edge routing, or a favorable direct route can make standard S3 faster or cheaper.
• Choosing Transfer Acceleration to solve an object-size limit: Transfer Acceleration does not replace Multipart Upload for objects larger than 5 GB.
• Forgetting to complete the upload: Uploaded parts do not become the final object until the multipart upload is completed.
• Ignoring abandoned parts: Parts from incomplete uploads can continue to consume storage.
• Assuming the bucket must be moved closer to users: Transfer Acceleration can be evaluated when centralizing the bucket is required, but it is not always the most cost-effective option.
• Treating the 100 MB recommendation as a hard requirement: Multipart Upload is supported from 5 MB upward; approximately 100 MB is a practical recommendation, not the minimum.
• Assuming a standard S3 URL automatically uses acceleration: Applications must use the S3 acceleration endpoint after acceleration is enabled.

Real-World Engineer Notes

• Select part sizes carefully. Very small parts increase request overhead, while very large parts reduce parallelism and make retries more expensive.
• Design clients to retry individual parts with exponential backoff.
• Persist the upload ID and completed part information if uploads must survive application restarts.
• Use checksums and validate the completed object when data integrity is important.
• Configure lifecycle cleanup for incomplete multipart uploads, especially for user-generated content or clients that may disconnect frequently.
• Measure transfer performance from actual user regions. A test from one location does not predict performance globally.
• Consider whether a nearer S3 bucket, regional upload strategy, AWS Snow Family option, or another data-ingestion design is more appropriate than Transfer Acceleration.
• Treat Transfer Acceleration as an optimization, not a reliability mechanism. Multipart retry logic and durable application state are still needed.

Quick Reference Summary

• Multipart Upload: Split and upload a large S3 object as independent parts.
• Recommended threshold: Objects larger than approximately 100 MB.
• Supported range: 5 MB to 5 TB.
• Required above: 5 GB.
• Main benefits: Parallelism, resumability, and independent retries.
• Operational requirement: Complete or abort the multipart upload.
• Transfer Acceleration: Client connects to a nearby CloudFront edge location before traffic travels over the AWS global network to S3.
• Best fit: Large transfers from geographically distant clients.
• Cost consideration: Additional transfer charges; test performance first.
• Can they be combined? Yes. Multipart Upload can use an accelerated S3 endpoint.

Flashcards

1. What is the primary purpose of S3 Multipart Upload?

To upload one object as independently managed parts, enabling parallelism, retries, and resumability.

2. What is the supported object-size range for Multipart Upload?

From 5 MB through 5 TB.

3. When is Multipart Upload mandatory?

For objects larger than 5 GB.

4. Are multipart parts required to be uploaded sequentially?

No. They can be uploaded independently, in parallel, and in any order.

5. What identifies a multipart upload session?

An upload ID returned when the multipart upload is initiated.

6. What happens after all parts are uploaded?

The client must send a completion request so S3 can assemble the parts into the final object.

7. What happens to an incomplete multipart upload?

Its uploaded parts remain until the upload is completed or aborted, potentially generating storage charges.

8. What does S3 Transfer Acceleration use to improve transfer paths?

CloudFront edge locations and the AWS global network.

9. Where is Transfer Acceleration enabled?

At the S3 bucket level.

10. When is Transfer Acceleration most likely to help?

When clients are geographically far from the bucket’s Region and the accelerated network path is better than the direct path.

11. Is Transfer Acceleration guaranteed to improve speed?

No. It must be tested because performance varies by source location and network conditions.

12. Can Multipart Upload and Transfer Acceleration be combined?

Yes. Large objects can be uploaded in parts through the accelerated endpoint.

Practice Questions

Question 1

A company accepts 20 GB video uploads from customers around the world. The videos are stored in a centralized S3 bucket. Uploads frequently fail near completion, forcing customers to restart the entire transfer. Which design best addresses the immediate reliability problem?

A. Enable S3 Transfer Acceleration only
B. Use S3 Multipart Upload with per-part retries
C. Increase the S3 bucket’s maximum object size
D. Store the videos in S3 Glacier Flexible Retrieval during upload

Correct answer: B

Explanation: Multipart Upload divides the object into independently uploaded parts. A failed part can be retried without restarting the complete 20 GB upload. Transfer Acceleration may improve the network path but does not by itself provide multipart retry behavior.

Question 2

A global application uploads large objects to an S3 bucket in one Region. Testing shows that users in distant Regions have substantially better performance through a nearby edge location, while users close to the bucket see no improvement. What should the architect do?

A. Enable Transfer Acceleration and use it for all clients without further analysis
B. Enable Transfer Acceleration and route only appropriate clients through the accelerated endpoint
C. Replace Multipart Upload with Transfer Acceleration
D. Create a CloudFront distribution with S3 as the origin and require all uploads through the distribution

Correct answer: B

Explanation: Transfer Acceleration can improve long-distance transfers, but it is not universally faster or cost-effective. Clients should use it where testing demonstrates a benefit. Multipart Upload may still be used for large objects.

Question 3

An application initiates thousands of S3 multipart uploads. Some clients disconnect and never return. The organization notices that storage costs are increasing even though the final object count has not changed. What is the best solution?

A. Enable S3 Versioning
B. Configure an S3 Lifecycle rule to abort incomplete multipart uploads
C. Change the S3 storage class to S3 Glacier Deep Archive
D. Enable Transfer Acceleration

Correct answer: B

Explanation: Uploaded parts from incomplete multipart sessions can remain stored. An S3 Lifecycle rule can automatically abort incomplete multipart uploads after a defined period.

Question 4

A 7 GB file must be uploaded to Amazon S3. The application developer proposes a single standard PUT request over a highly reliable private network. What is the correct architectural response?

A. Approve the design because a reliable network removes the object-size restriction
B. Use Multipart Upload because objects larger than 5 GB require it
C. Use Transfer Acceleration because all objects larger than 5 GB require acceleration
D. Compress the file until it is smaller than 5 GB, regardless of whether compression is effective

Correct answer: B

Explanation: S3 Multipart Upload is required for objects larger than 5 GB. Network reliability does not remove this requirement, and Transfer Acceleration is optional and intended to address network-path performance.

Question 5

A company wants to reduce upload costs. Its users are mostly located near the S3 bucket’s Region, and performance testing shows that Transfer Acceleration provides no meaningful improvement. Which option is most appropriate?

A. Use the standard S3 endpoint and Multipart Upload for sufficiently large objects
B. Use Transfer Acceleration because it always reduces S3 transfer costs
C. Route uploads through a CloudFront distribution
D. Upload every object as a single request to avoid multipart overhead

Correct answer: A

Explanation: The standard S3 endpoint avoids the additional Transfer Acceleration charge when acceleration provides no benefit. Multipart Upload should still be used when object size, reliability, or parallelism justifies it.