AWS Java SDK V2 CRT based S3 solution that can work with low memory requirement #894
Description
Describe the feature
The CRT-based S3 client in AWS Java SDK V2 offers significant benefits over the standard implementation, particularly in terms of reliability and retry behavior during multipart uploads. This is especially valuable when working with streaming uploads, where the non-CRT path may encounter NonRetryableException due to lack of support for re-reading streams.
We would like to request support for a low-memory mode or tunable configuration for the CRT-based S3 client that:
- Reduces the internal buffering size used for upload requests
- Avoids pre-allocating large direct memory buffers
- Supports streaming uploads (e.g., using AsyncRequestBody) with bounded memory usage
- Retains the core advantages of CRT (e.g., automatic retries, efficient HTTP/2 I/O)
Use Case
Our application needs to perform multipart uploads to S3 in constrained k8s environments. A lighter CRT client would allow us to take advantage of its superior retry and network handling without sacrificing system stability or incurring out-of-memory risks.
The current CRT-based client implementation appears to require a relatively high memory footprint, making it less suitable for memory-constrained deployment environments. In our case, we are targeting uploads to work with ≤ 250 MB memory, and the default configuration of the CRT client exceeds this limit, thus leading to K8s POD errors in our deployed environment.
Proposed Solution
No response
Other Information
Currently using a CRT based client with the maxConcurrency set to 1, performs almost 5 times slower than Java SDK v1 HTTP client without concurrency.
Acknowledgements
- I may be able to implement this feature request
- This feature might incur a breaking change