Request for opinion with POC implementation done: Update S3 storage backend to properly support rotating / expiring credentials

[nivintw]

This is another set of changes / feature that i've implemented to fit my purposes and want to gather feedback before cleaning up and sharing back.

Basically, when in contexts that use rotating AWS credentials the current approach keeps boto3 objects alive for too long, causing the credentials to reach expiration, and causing installs to fail due to internal server errors when the backend tries to communicate with S3 and can not.

Scenarios that benefit from the feature / require it:

1. when deployed to EC2 instance that has an instance profile
2. when deployed via k8s / helm / etc that have other infrastructure that dynamically updates the credentials in the container
3. Any other deployment scenario that requires non-static AWS credentials

The set of changes are scoped to the S3 backend, and are actually relatively minor, although admittedly my current approach is definitely a little "brute force".

In general, the advice i've been given / seen personally is that when interacting with AWS SDKs you don't want to have any long-lived objects for the reasons described above.

Looking for feedback on if a PR would be appreciated here, and sharing publicly to help hold myself accountable to getting the changes shared upstream.

Thanks!

[stevearc]

From the sound of it, there are issues with certain types of deployments that this would fix. I'd be happy to look at a PR!

[nivintw]

PR opened; welcome all feedback. TYVM