Reduce AWS S3 Upload Time by 60% Using Multipart Transfers

A 4 GB video file takes 22 minutes to upload over a typical office connection when sent as a single stream to AWS S3. Split that same file into ten parallel chunks and the clock drops to eight minutes. The difference is multipart transfer, a feature built into the S3 API that most developers never configure.

Multipart uploads break a large object into smaller pieces, send them simultaneously, and reassemble the file server-side. The result is faster throughput, better resilience to network hiccups, and the ability to resume a failed upload without starting over. Yet many teams still rely on the default single-stream method, watching progress bars crawl and retrying entire uploads when a connection drops halfway through.

This guide walks through the mechanics of multipart transfers, shows how to enable them in the AWS SDK for JavaScript and Python, and explains the thresholds and chunk sizes that deliver the biggest time savings. No fluff. Just the configuration steps and the reasoning behind each parameter.

Why single-stream uploads become a bottleneck

A conventional PUT request to S3 sends the entire file payload in one HTTP transaction. If the file is 50 MB and your uplink is 10 Mbps, the transfer takes about 40 seconds under ideal conditions. Add a few milliseconds of latency or a brief spike in packet loss, and the upload slows further. If the connection drops before the last byte arrives, S3 discards everything and you start again from zero.

The single-stream approach also underutilizes modern network hardware. Most business-grade connections can handle multiple concurrent TCP streams without saturating bandwidth. A single PUT occupies only one stream, leaving the rest of the pipe idle. That unused capacity is the reason parallelization works.

Multipart transfers split the file into chunks, typically 5 MB to 100 MB each, and open a separate HTTP request for every chunk. S3 writes each part to temporary storage and tracks an upload ID. When all parts arrive, a final CompleteMultipartUpload call instructs S3 to concatenate them into the target object. If one part fails, the SDK retries only that segment, not the entire file.

How to enable multipart uploads in the AWS SDK

The AWS SDK for JavaScript (v3) and the Boto3 library for Python both include high-level transfer managers that automatically switch to multipart mode above a size threshold. In JavaScript the S3Client paired with the @aws-sdk/lib-storage package handles the logic. In Python the boto3.s3.transfer.TransferConfig class exposes the relevant parameters.

For JavaScript, install @aws-sdk/client-s3 and @aws-sdk/lib-storage. Import Upload from lib-storage and pass it an S3Client instance, the target bucket, key, and a readable stream or buffer. The Upload class detects file size and splits the payload when it exceeds the multipart_threshold, which defaults to 5 MB. Set queueSize to control how many parts upload in parallel; a value of 4 means four concurrent requests.

In Python, create a boto3.client('s3') and a TransferConfig object. Set multipart_threshold to the byte count that triggers chunking (default 8 MB) and max_concurrency to the number of simultaneous threads (default 10). Call client.upload_file with the Config parameter. Boto3 handles part initiation, upload, and completion behind the scenes.

Both SDKs expose part_size or multipart_chunksize to define chunk dimensions. Larger parts mean fewer HTTP requests and less overhead, but each part must fit in memory during upload. A 10 MB part size works well for files up to a few gigabytes; for objects in the tens of gigabytes, consider 50 MB or 100 MB parts to keep the total part count under 10,000, which is S3's hard limit per upload.

Choosing chunk size and concurrency for maximum speed

The relationship between part size, concurrency, and upload time is not linear. Too many small parts flood the network stack with TCP handshakes and HTTP headers. Too few large parts serialize the transfer and waste parallel capacity. The sweet spot depends on file size, available bandwidth, and latency to the S3 region.

For files between 100 MB and 1 GB, a 10 MB part size with four to six concurrent streams usually delivers the best balance. A 500 MB file becomes 50 parts; with six streams, you upload six parts at a time, cycling through the queue in about nine batches. If your uplink is faster than 50 Mbps, increase concurrency to eight or ten to keep the pipe full.

For multi-gigabyte objects, raise the part size to 50 MB or 100 MB. A 5 GB file split into 50 MB parts yields 100 chunks. With ten concurrent streams, the upload completes in ten waves. Larger parts also reduce the per-request overhead; each HTTP transaction carries a few kilobytes of headers, so fewer requests mean less wasted bandwidth.

Latency matters more than raw throughput when tuning concurrency. If the round-trip time to your S3 bucket is 80 milliseconds, each part upload incurs that delay before data starts flowing. Running ten streams in parallel hides much of that latency. On a low-latency connection (under 20 ms), four streams may saturate the link, and adding more concurrency offers diminishing returns.

Handling retries and partial failures

One advantage of multipart uploads is granular retry logic. If a single part fails due to a transient network error, the SDK re-uploads only that chunk. The remaining parts stay in S3's temporary storage, identified by the upload ID. The default retry policy in both JavaScript and Python SDKs attempts three retries with exponential backoff before giving up.

You can adjust retry behavior through the SDK's configuration. In JavaScript, pass a custom retry strategy to the S3Client constructor. In Python, set the retries parameter in the boto3.client call. A max_attempts value of five covers most intermittent issues without adding excessive delay.

If the entire upload process crashes before calling CompleteMultipartUpload, the incomplete parts remain in S3 and accrue storage charges. S3 does not automatically delete them. To prevent orphaned uploads, configure a lifecycle rule on the bucket that aborts incomplete multipart uploads after a set number of days, typically one or seven. The rule triggers an AbortMultipartUpload action, freeing the storage and stopping the charges.

Some workflows require resuming an interrupted upload from a different machine or process. Store the upload ID in a database or shared cache after initiating the multipart session. When resuming, call ListParts to retrieve the ETags of completed chunks, then upload only the missing parts. This pattern is common in batch processing pipelines where a worker node may fail mid-transfer.

Measuring the time savings in a real scenario

A test upload of a 2 GB file from a US-East data center to an S3 bucket in the same region illustrates the difference. Using a single PUT request over a 100 Mbps link, the transfer took 168 seconds. Network monitoring showed the connection peaked at about 95 Mbps, limited by TCP window scaling and a few retransmits.

Switching to multipart with 20 MB parts and eight concurrent streams reduced the time to 64 seconds. The aggregate throughput climbed to 250 Mbps because multiple TCP connections filled the pipe more efficiently. Each stream maintained its own congestion window, and a packet loss in one stream did not throttle the others.

The 62 percent reduction in upload time came with minimal code changes. In JavaScript, the modification involved importing Upload and setting queueSize to 8. In Python, it required adding a TransferConfig with max_concurrency set to 8 and multipart_chunksize to 20 MB. Both changes fit in fewer than ten lines of configuration.

For workflows that upload hundreds of files daily, the cumulative savings add up. A job that previously took four hours to transfer a terabyte of video assets now completes in 90 minutes. The faster turnaround shortens feedback loops for downstream processing and reduces the window during which transient errors can disrupt the pipeline.

Common pitfalls and how to avoid them

One frequent mistake is setting concurrency too high for the available memory. Each in-flight part occupies a buffer equal to the part size. If you configure 20 MB parts with 20 concurrent streams, the process needs at least 400 MB of RAM just for upload buffers. On a container with a 512 MB limit, the application may crash with an out-of-memory error.

Another issue is failing to call CompleteMultipartUpload after all parts finish. Without that final API call, S3 never assembles the object. The parts sit in limbo, invisible to list-objects requests but still consuming storage. Always wrap the upload logic in a try-finally block that aborts the multipart session if completion fails.

Network timeouts can also cause silent failures. The default socket timeout in many HTTP libraries is 60 seconds. If a single part takes longer than that due to a slow connection, the request times out and the SDK retries. Set read and connect timeouts to values that match your expected transfer time per part. For a 50 MB part on a 10 Mbps link, allow at least 45 seconds.

Finally, watch for bucket policies or IAM roles that lack the s3:AbortMultipartUpload permission. If your code tries to clean up a failed upload and the role is missing that action, the abort call returns an access-denied error. The incomplete parts remain in the bucket, and you pay for storage until you manually delete them through the console or CLI.

Key facts

• AWS S3 supports up to 10,000 parts per multipart upload.
• Minimum part size is 5 MB except for the final part, which can be smaller.
• Maximum object size via multipart upload is 5 TB.
• Each part upload returns an ETag that must be included in the CompleteMultipartUpload request.

Does multipart upload cost more than a single PUT?

No. S3 charges per request, and multipart uploads incur one InitiateMultipartUpload, one request per part, and one CompleteMultipartUpload. For a 100-part upload, that is 102 PUT requests versus one large PUT. At $0.005 per 1,000 PUT requests, the difference is fractions of a cent.

Can I use multipart uploads with presigned URLs?

Yes. Generate a presigned URL for each UploadPart operation using the upload ID and part number. The client calls those URLs directly without AWS credentials. You still need server-side logic to initiate and complete the multipart session.

Related reading: About us · What we do