V4 Development Tracker

[normj]

V4 Development Tracker

This issue is for tracking the changes being made for V4 of the AWS SDK for .NET. Development of V4 is happening in the v4-development branch in this repository. V4 is an evolutionary major version change with minimal breaking changes so application code using the AWS SDK can upgrade to V4 with low effort.

Update April 28th, 2025

V4 has been released general availability. We recommend all users of the AWS SDK for .NET to upgrade to V4 to take advantage of V4's performance improvements and continue to get support for the SDK before V3 reaches maintenance mode.

https://aws.amazon.com/blogs/developer/general-availability-of-aws-sdk-for-net-v4-0/

Updated libraries that depend on the AWS SDK for .NET

The following are the list of AWS .NET libraries that have updated to V4.

• AWS.AspNetCore.DistributedCacheProvider (0.21.0-preview)
• Amazon.Extensions.S3.Encryption (3.0.0)
• AWS.Messaging (0.22.0)
• Amazon.Extensions.CognitoAuthentication (3.0.0)
• Amazon.Extensions.Configuration.SystemsManager (7.0.0)
• AWS.Logger (4.0.0)
• Amazon.AspNetCore.DataProtection.SSM (4.0.0)
• Amazon.AspNetCore.Identity.Cognito (4.0.0)
• Amazon.Lambda.DynamoDBEvents.SDK.Convertor (2.0.0)
• Amazon.Lambda.KinesisEvents (3.0.0)
• AWSSDK.SecretsManager.Caching (2.0.0)
• OpenTelemetry.Instrumentation.AWS (1.12.0)
• Aspire.Hosting.AWS (9.2.0)

Update April 25th, 2025

In preparation of V4 GA we have made some branch changes in the repository. Checkout this GitHub issue for details. #3771

Update April 9th, 2025

Preview 13 has been released that includes support for AWS service operations that require bi-directional streaming over HTTP 2. This support is available for .NET 8 and later. Examples operations are Transcribe Streaming's StartStreamTranscription API and the new Bedrock InvokeModelWithBidirectionalStream operation. The InvokeModelWithBidirectionalStream brings support to .NET Amazon Nova Sonics's speech to speech feature. The .NET Framework target of the SDK will not have the bi-directional operations due to .NET Framework not supporting HTTP 2.

At this point we consider V4 feature complete for GA. We will stay in preview mode for a little longer to collect feedback on the latest preview. We also have a bit of internal work hooking up V4 to our daily release cycle and finishing up documentation tasks. There will possibly be a few more small previews before GA for minor tweaks and fixes based on feedback.

Our ask to the AWS .NET community is to try out the latest preview of V4 to help confirm for us that V4 is ready to be GA.

Update November 11, 2024

New blog post released for preview 4 catching up some of the major changes since preview 1. https://aws.amazon.com/blogs/developer/preview-4-of-aws-sdk-for-net-v4/

Update August 15, 2024

First preview packages have been released. Blog post for the release https://aws.amazon.com/blogs/developer/preview-1-of-aws-sdk-for-net-v4/

Breaking changes

The list of breaking changes made in V4.

• .NET Framework 3.5 target has been removed.
• .NET Framework 4.5 target has been updated to 4.7.2
• Properties using value types on classes used for making requests and responses have been changed to use nullable value types. Essentially properties of type bool are changed to bool?.
• Properties using collections on classes used for making requests and responses will now default to null. The V3 behavior of initializing collections can be restored by setting the Amazon.AWSConfigs.InitializeCollections to true. This property also exists in V3 for users that want to try this behavior change before upgrading to V4.
• S3 service clients configured for us-east-1 will no longer be able to access buckets in other regions. Buckets musts be accessed with S3 service clients configured for the region the bucket is in. This change avoids surprises in applications using the S3 service client when under the covers it makes multiple AWS requests to access buckets that are not in service clients configured us-east-1 region.
• The IAM action identifiers have been removed from the Amazon.Auth.AccessControlPolicy.ActionIdentifiers namespace. These identifiers were previously marked as obsolete due to be being out of date with no mechanism to keep them up to date. Code using these ActionIdentifiers should be upgrade to use the string value of the IAM action name.
• The S3 encryption client has been removed from the AWSSDK.S3 package. This client had been marked as obsolete with the encryption client moved to its separate package Amazon.Extensions.S3.Encryption package. The following migration guide to transition to the separate package. https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/s3-encryption-migration.html
• The DetermineServiceURL method on the ClientConfig, the base class of service client configs like AmazonS3Config, was removed. This had been marked obsolete and was tied to an internal endpoint resolution system that was removed in V4. Application code calling this method should switch to using the DetermineServiceOperationEndpoint method on the service client.
• When using credential providers that rely on Amazon Security Token Service (STS) the calls will always use the regional endpoint. V3 by default used us-east-1 regardless of the configured region when running in the public partition. As part of this change the StsRegionalEndpointsValue enum was removed. Any code using that enum should be removed.
• On service client configs the RetryMode defaults to Standard instead of Legacy. The Legacy enum value was removed.
• On service client configs the DefaultConfigurationMode defaults to Standard instead of Legacy. The Legacy enum value was removed.
• DefaultClientConfig in AWSSDK.Extensions.NETCore.Setup no longer extends from service client config base class ClientConfig. The service client config properties have been replicated on DefaultClientConfig using nullable value types to allow detecting when a value has been set on DefaultClientConfig when copying the values to the service client config being created for the service client.
• CloudFront's CookieSigner and UrlSigner have been moved to a separate extension package called AWSSDK.Extensions.CloudFront.Signers to support SSL3 and take a dependency on BouncyCastle.Cryptography.
• EC2's GetDecryptedPassword has been moved to a separate package called AWSSDK.Extensions.EC2.DecryptPassword to support OpenSSL 3 and take a dependency on BouncyCastle.Cryptography.
• The SDK's source copy of BouncyCastle has been removed.
• S3's Tagging-Directive has been exposed as a public property on CopyObjectRequest and is no longer automatically set to COPY. If a user doesn't specify a tagging directive the S3 backend automatically assumes it is COPY but if a user explicitly sets it to null, then we won't set the value at all.
• AmazonS3Config's UseArnRegion will take the value set in the environment variable over that set in the ~/.aws/config file. This follows other config environment variable + config file precedence.
• Leading slashes will no longer be trimmed in CopyObject and CopyPart operations. The config option DisableTrimmingLeadingSlash has been removed.
• Removed obsolete properties such EndTime then changed EndTimeUtc to EndTime. This could lead to offset times if anyone was still using the marked obsolete original EndTime for example. A compile time error will occur for anyone using EndTimeUtc. Removing BackwardCompatibility properties and code in the generator is part of this change. Also id this to the manual S3 models.
• Response unmarshallers for TimeStamps and list TimeStamps for formats TimestampFormat.ISO8601 || TimestampFormat.RFC822 datetimes were being parsed into local times. Adjusted DateTime parsing to return UTC times. (BaseResponseUnmarshaller.tt, MultiValueHeaderParser.cs)
• Fixed the DateTimeUnmarshaller which was parsing datetime strings into and returning them as local time which in some cases were still getting converted back to UTC on a prior bug fix but not always. DateTime strings unmarshalled are assumed to be UTC time and will be specified and unmarshalled as UTC. (SimpleTypeUnmarshaller.cs)
• ConvertFromUnixEpochSeconds/ConvertFromUnixEpochMilliseconds incorrectly returning the Unix Epoch time as localtime instead of by definition is a UTC time. This changes the behavior where these methods were used. These were assigned to headers typically: https://github.com/search?q=repo%3Aaws%2Faws-sdk-net+ConvertFromUnixEpoch&type=code (AWSSDKUtils.cs)
• DynamoDB RetrieveDateTimeInUtc has been switched to true as the default.
• Removal of AWSConfigsS3.UseSignatureVersion4 and AWSConfigsS3.UseSigV4SetExplicitly to always use SigV4.
• Removal of ClientConfig.SignatureVersion only used by S3 for backward compatibility.
• Removal of Endpoint.SignatureVersionOverride from the obsolete Endpoint class used to handle overriding signature versions for S3.
• Updated AmazonS3Util.PostUpload and S3PostUploadSignedPolicy.GetSignedPolicy to use SigV4. S3PostUploadSignedPolicy.GetSignedPolicyV4 removed and overwrote to S3PostUploadSignedPolicy.GetSignedPolicy.
• Removed EC2: DiskImageImporter, ImportCleanup, EC2Metadata, and S3UploadPolicy classes.
• Support for EC2 IMDS v1 protocol has been removed. The SDK will always use IMDS v2 protocol when fetching credentials and other metadata from IMDS.

Other noteworthy changes

The following DateTime related changes were made in these two PRs:
#3572
#3601

• Fixed Epoch date to UTC per definition where the epoch date was created in local time. (TokenBucket.cs)
• Stopped using expiry times as localtime. Changed to UTC. (InstanceProfileAWSCredentials.cs, ProcessAWSCredentials.cs, RefreshingAWSCredentials.cs)
• Ensured DateTime.Max and DateTime.Min are marked with a DateTime.Kind DateTimeKind.Utc for proper calculations.
• Instead of assuming SAML credentials are localtime then converting to UTC assume that the time given is UTC to workproperly with credential expiration being in UTC time for other credential providers. (SAMLImutableCredentials.cs)
• Console logger outputs timestamps as a UTC date incase output is sent off the local machine and for easier comparison with other UTC dates. We could add a preference flag here though. (Logger.Console.cs)
• RetryPolicies return UTC server time instead of a UTC time converted to local time. (RetryPolicy.cs)
• AWSPublicIpAddressRanges mixing UTC and local time. (AWSPublicIpAddressRanges.cs)
• GetFormattedTimestampISO8601 incorrectly creating a DateTime object as local time even though it is passed in as UTC. Then formatting it as a UTC string. Although incorrect the DaeTimeKind.Local flag didn't actually harm anything and the behavior is the same by not using it.

Removed Code

• The obsolete ECSTaskCredentials class is removed. Please use the GenericContainerCredentials provider which also supports EKS Pod Identities.
• The obsolete StoredProfileAWSCredentials and StoredProfileCredentials is removed. Please use the NetSDKCredentialsFile or the SharedCredentialsFile.
• The obsolete HasCachedAccessTokenAvailable method on the SSOAWSCredentials class is removed.
• The obsolete EnvironmentAWSCredentials is removed. Please use AppConfigAWSCredentials instead.
• The obsolete StoredProfileFederatedCredentials is removed. Please use FederatedAWSCredentials instead.
• The obsolete StoredProfileSAMLCredentials class in SecurityTokenService is removed. Please use FederatedAWSCredentials instead.
• The obsolete EndpointDiscoveryEnabled class is removed. Please use the EnvironmentVariableAWSConfiguration instead.
• The obsolete UseSigV4 property on the AmazonWebServiceRequest class is removed. SignatureVersion will be used directly instead.
• The obsolete TryGetSection methods in ProfileIniFile that don't take in nested properties is removed. Please use the TryGetSection method in ProfileIniFile that accounts for nested properties.
• The obsolete Parameters dictionary in WebServiceRequestEventArgs is obsolete. ParameteCollection will be used instead.
• The obsolete DoesS3BucketExistAsync(string bucketName) in ICoreAmazonS3 is removed because it always uses HTTP. Please use Amazon.S3.Util.AmazonS3Util.DoesS3BucketExistV2Async instead.
• The following methods in AWSSDKUtils are removed ResolveResourcePath, ProtectEncodedSlashUrlEncode, ConvertToUnixEpochMilliSeconds
• EC2_METADATA_SVC, EC2_METADATA_ROOT, EC2_USERDATA_ROOT, EC2_DYNAMICDATA_ROOT, EC2_APITOKEN_URL in EC2InstanceMetadata are removed. The fields were marked as obsolete in v3.
• The obsolete ProfileManager class in Amazon.Util is removed. Please use the SharedCredentialsFile or the NetSDKCredentialsFile instead.
• The following obsolete properties on AWSConfigs are removed: Logging, ResponseLogging, and LogMetrics. Use LoggingConfig instead.
• public static Condition NewCondition(DateComparisonType type, DateTime date) in Amazon.Auth/AccessControlPolicy/ConditionFactory is removed. Please use NewConditionUtc instead.
• In the ClientConfig class, the obsolete DetermineDnsSuffix has been removed. Use the service specific client.DetermineServiceOperationEndpoint method instead. The ReadEntireResponse property has been removed, use the AWSConfigs.LoggingConfig.LogResponses or ClientConfig.LogResponse instead.
• Removal of the obsolete EventBridgeSigner class.

DynamoDB Changes

In V4 we're addressing issues around testability, improving configuring the high-level Table and DynamoDBContext clients, and potentially additional fixes that require breaking changes.

• In DynamoDB's high-level programming model, initializing the Table with a mocked IAmazonDynamoDB now returns an InvalidOperationException instead of a NullReferenceException . Async Table methods should now work with a mocked client but you may still see exceptions when calling sync methods from .NET/Core/Standard. See PR #3388, this mostly addresses DynamoDB Table misrepresents dependency on IAmazonDynamoDB #1589.
• In DynamoDB's object persistence programming model, we've separated the shared DynamoDBOperationConfig into new operation-specific objects (SaveConfig, LoadConfig, QueryConfig, etc.). The shared config has grown overtime, and contains properties that don't apply to every operation which can lead to confusion. we've marked the methods that take DynamoDBOperationConfig as obsolete. We won't remove them from V4 yet, though may do so in a future version. See PR #3421
• In DynamoDB's object persistence programming model, we've removed MetadataCachingMode and DisableFetchingTableMetadata from DynamoDBOperationConfig, and did not include these on the new operation-specific configs that were introduced above. These are table-level settings that should be specified on the global AWSConfigsDynamoDB.Context or on DynamoDBContextConfig. See PR #3422
• In DynamoDB's object persistence programming model, DynamoDBOperationConfig no longer inherits from DynamoDBContextConfig. This prevents you from passing a DynamoDBOperationConfig in to the constructor for DynamoDBContext, where some properties on the operation-specific config (such as OverrideTableName) do not apply. See PR #3422
• DynamoDBStreams has been removed from the DynamoDB package and is now in its own package and namespace.
• Community PR #3476 Allow to remove TableNamePrefix on individual operation level. Thanks Oleksandr Liakhevych
• We have added new operation specific interfaces to allow customers to mock DynamoDB operations. See PR #3450. The factory methods on IDynamoDBContext have been updated to return the new interfaces.
  • Object Persistence Programming Model
    • Mock BatchGet operations via IBatchGet and IMultiTableBatchGet interfaces.
    • Mock BatchWrite operations via IBatchWrite and IMultiTableBatchWrite interfaces.
    • Mock TransactGet operations via ITransactGet and IMultiTableTransactGet interfaces.
    • Mock TransactWrite operations via ITransactWrite and IMultiTableTransactWrite interfaces.
    • Mock Scan and Query operations via the IAsyncSearch interface.
  • Document Programming Model
    • Mock Table operations via the ITable interface.
    • Mock Scan and Query operations via the ISearch interface.
    • Mock TransactWrite operations via IDocumentTransactWrite and IMultiTableDocumentTransactWrite interfaces.
    • Mock TransactGet operations via IDocumentTransactGet and IMultiTableDocumentTransactGet interfaces.
    • Mock BatchWrite operations via IDocumentBatchWrite and IMultiTableDocumentBatchWrite interfaces.
    • Mock BatchGet operations via IDocumentBatchGet and IMultiTableDocumentBatchGet interfaces.
• The FromJson and ToJson methods on the Document object now use System.Text.Json instead of LitJson for serialization. A benefit of using System.Text.Json is this parser supports using the .NET Decimal type supporting higher precision for numeric floating point properties.
• DocumentModel and DataModel (Object Persistence Model) high level libraries have been updated to support Native AOT.
  • A limitation of the Native AOT support is nested types in .NET types used for serialization in the DataModel might go unnoticed by the trimming component of the .NET compiler. In that case you could receive an exception like System.InvalidOperationException: Type <type> is unsupported, it cannot be instantiated. The work around is to add the DynamicDependency somewhere in the code path that informs the trimmer about the dependency on the sub type. The constructor of the top level .NET type being saved is a likely place to use the constructor. The code below shows how to use the DynamicDependency attribute:

[DynamoDBTable("TestTable")]
class TypeWithNestedTypeProperty
{
    [DynamicDependency(DynamicallyAccessedMemberTypes.All, typeof(SubType))]
    public TypeWithNestedTypeProperty()
    {

    }

    [DynamoDBHashKey]
    public string Id { get; set; }
    public string Name { get; set; }

    public SubType SubType { get; set; }
}

class SubType
{
    public string SubName { get; set; }
}

• Implemented polymorphism support with object persistence high level library. implement polymorphism support for DynamoDB entries #3643

Logging

How to enable logging for the SDK has been reworked for V4. Logging to the console and system diagnostics works the same as V3 by setting the AWSConfigs.LoggingConfig.LogTo property to either LoggingOptions.Console or LoggingOptions.SystemDiagnostics. The LoggingOptions option for log4net has been removed along with the SDK's internal logic for using reflection to attach to in memory instance of log4net.

To include the SDK's logging into a logging framework a separate adaptor package is used to connect the SDK with the the logging framework. We have released AWSSDK.Extensions.Logging.Log4NetAdaptor for 'log4net' and AWSSDK.Extensions.Logging.ILoggerAdaptor for Microsoft.Extensions.Logging.

Example configuring Microsoft.Extensions.Logging

Add AWSSDK.Extensions.Logging.ILoggerAdaptor NuGet package and call the ConfigureAWSSDKLogging extension method from the ILoggerFactory.

var builder = WebApplication.CreateBuilder(args);

var app = builder.Build();

app.Services.GetRequiredService<ILoggerFactory>()
    .ConfigureAWSSDKLogging();

Example configuring for log4net

Add AWSSDK.Extensions.Logging.Log4NetAdaptor NuGet package and call the static ConfigureAWSSDKLogging method from Log4NetAWSExtensions.

using Amazon.DynamoDBv2;
using Amazon.Extensions.Logging.Log4NetAdaptor;
using log4net;

[assembly: log4net.Config.XmlConfigurator(ConfigFile = "log4net.config")]

Log4NetAWSExtensions.ConfigureAWSSDKLogging();
var logger = LogManager.GetLogger(typeof(Program));

---

The following changes were made in the SDK. They do not require changes to applications code using the SDK.

• The following dependencies were added to the .NET Framework 4.7.2 and .NET Standard 2.0 targets
  • System.Buffers
  • System.Memory
  • System.Text.Json
• AWSSDK.Extensions.NETCore.Setup has been made Native AOT compatible.
• For JSON based services, the SDK will now use System.Text.Json for serialization. For Non-NetFramework targets, the SDK will utilize buffer pooling for writing. For reading, the SDK will utilize buffer pooling for all targets.
• In .NET Standard 2.0, .NET Core 3.1 and .NET 8 the AssemblyVersion will match the AssemblyFileVersion. In .NET Framework 4.7.2 target AssemblyVersion will continue to use the V3 pattern of locking the version to the first 2 parts of the the AssemblyFileVersion. This is done because .NET Framework treats the AssemblyVersion as part of the identity assembly. This causes assembly collisions and forced recompilation when mixing versions of the SDK.
• The embedded endpoints.json file in AWSSDK.Core was removed. This was a 1 MB json file that was parsed at startup for the SDK to determine the service endpoint for a region. Endpoint resolution has been replaced with a new system that generates rules into each individual service assembly for determining the service endpoints.
• Community PR #3359 Optimize GetExtension execution time on .NET 8. Thanks Steven Weerdenburg
• Community PR #3293 improving AWSSDK.ToHex performance. Thanks Steven Weerdenburg
• Community PR #3292 improving AWSSDK.CopyTo performance. Thanks Daniel Marbach
• Community PR #3324 optimizing the AWS SigV4 signer. Thanks Daniel Marbach
• Community PR #3363 improving AWSSDKUtils.ParameterAsString performance. Thanks Daniel Marbach
• Community PR #3365 improving AWSSDKUtils.DetermineService performance. Thanks Daniel Marbach
• Community PR #3307 Optimizing AWSSDKUtils.UrlEncode performance. Thanks Daniel Marbach
• Community PR #3425 Avoid allocating byte[] when converting MemoryStream to String. Thanks Paulo Morgado
• Community PR #3423 Refactor user agent construction to avoid creating unnecessary string objects. Thanks Paulo Morgado
• Community PR #3439 Refactor JSON request marshaller to avoid creating unnecessary intermediate strings. Thanks Paulo Morgado
• Community PR #3458 Add ToString methods for better enum string conversion.
  Thanks Paulo Morgado
• Community PR #3570 Add possibility to add AWS services to DI with a key. Thanks Oleksii Zuiev
• Communit PR #3545 Implement Microsoft.Extensions.AI's IChatClient / IEmbeddingGenerator for IAmazonBedrockRuntime. Thanks Stephen Toub

[Dreamescaper]

Is there any way to test those packages? Some preview nuget source?

[dscpinheiro]

We'll publish the preview packages to NuGet (e.g. <PackageReference Include="AWSSDK.AccessAnalyzer" Version="4.0.0.0-preview" />), we're just working on some changes to our internal build system at the moment (we'll update this issue as soon as the packages are available).

[normj]

First preview packages have been released. Blog post for the release https://aws.amazon.com/blogs/developer/preview-1-of-aws-sdk-for-net-v4/

[martincostello]

Are you able to provide a preview version of Amazon.AspNetCore.DataProtection.SSM that depends on AWSSDK.Extensions.NETCore.Setup?

Trying to enable native AoT for a project that depends on the SSM package gives a bunch of warnings:

ILC : Trim analysis error IL2026: Microsoft.Extensions.DependencyInjection.ExtensionMethods.PersistKeysToAWSSystemsManager(IDataProtectionBuilder,String,Action`1<PersistOptions>): Using member 'Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.TryAddAWSService<IAmazonSimpleSystemsManagement>(IServiceCollection,ServiceLifetime)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. 
C:\codebuild\tmp\output\src1982900738\src\aws-sdk-net\sdk\src\Core\Amazon.Runtime\ConstantClass.cs(84): Trim analysis error IL2072: Amazon.Runtime.ConstantClass.Intern(): 'type' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields' in call to 'Amazon.Runtime.ConstantClass.LoadFields(Type)'. The return value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
  ILC: Method '[AWSSDK.Extensions.NETCore.Setup]Amazon.Extensions.NETCore.Setup.DefaultClientConfig..ctor()' will always throw because: Missing method 'Void Amazon.Runtime.ClientConfig..ctor()'
  ILC: Method '[Amazon.AspNetCore.DataProtection.SSM]Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository+<GetAllElementsAsync>d__9.MoveNext()' will always throw because: Missing method 'Void Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest.set_WithDecryption(Boolean)'

It's possible some of the above are also coming from internal libraries for transient dependencies that haven't been compiled against v4 yet either.

The app is still blocked from 0 AoT warnings due to a dependency on another third-party library that hasn't published some fixes I made yet, but otherwise this SSM package is the final hurdle.

[normj]

@martincostello I don't have a timeline yet but we would like push out preview versions of all of our high level packages like Amazon.AspNetCore.DataProtection.SSM but we have some internal work first we need to do setting up a dual preview pipeline across our repositories.

[martincostello]

FYI this is the error I get from an .NET 8 app running as a Lambda with the latest version of Amazon.AspNetCore.DataProtection.SSM. It builds fine, but then fails at runtime:

System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information.
 ---> System.MissingMethodException: Method not found: 'Void Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest.set_WithDecryption(Boolean)'.
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElementsAsync()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElementsAsync()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElements()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Serialize(AntiforgeryToken token)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.Serialize(IAntiforgeryFeature antiforgeryFeature)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.GetAndStoreTokens(HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.AntiforgeryExtensions.GetHtml(IAntiforgery antiforgery, HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.TagHelpers.FormTagHelper.Process(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.TagHelpers.TagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.RunAsync(TagHelperExecutionContext executionContext)
   at AspNetCoreGeneratedDocument.Pages_Shared_SignIn.ExecuteAsync() in /_/src/Codeflowbot/Pages/Shared/SignIn.cshtml:line 13
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|22_0(ResourceInvoker invoker, IActionResult result)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at MyApp.MyLoggerMiddleware.Invoke(HttpContext context) in /_/src/Codeflowbot/Middleware/CurrentUserLoggerMiddleware.cs:line 20
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at MyApp.MyMiddleware.Invoke(HttpContext context) in /_/src/Codeflowbot/Middleware/LambdaLoggerMiddleware.cs:line 38
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2.ProcessRequest(ILambdaContext lambdaContext, Object context, InvokeFeatures features, Boolean rethrowUnhandledError)

[normj]

@martincostello A preview version of Amazon.AspNetCore.DataProtection.SSM is out targeting V4 https://www.nuget.org/packages/Amazon.AspNetCore.DataProtection.SSM/4.0.0-preview.1

[martincostello]

Thanks @normj.

I've tested the latest version of the packages with three of our applications collectively using the v4 packages listed below, and I've not encountered any issues:

• Amazon.AspNetCore.DataProtection.SSM
• AWSSDK.DynamoDBv2
• AWSSDK.SimpleSystemsManagement
• AWSSDK.SecurityToken

I've also tried native AoT publishing with two of the three, and there were also no trim warnings.

[normj]

Awesome @martincostello! Thanks for the feedback.

[Dreamescaper]

Would it make sense to drop 'netcoreapp3.1' target framework as well in V4? It is not supported by Microsoft. Anyone who uses it for whatever reason - would still be able to use netstandard2.0 version.

[normj]

@Dreamescaper When we choose to drop a target we look at is it supported by Microsoft and our customer usage of the target. For example .NET 6 goes out of support by Microsoft tomorrow but we know there are still a lot of users still using .NET 6 and that would pretty rough to force everybody to update to get access to the latest AWS changes.

I can't give numbers but when we started the V4 effort last spring there was still a lot of .NET Core 3.1 usage. The usage is dropping pretty fast now but given the amount of required heads up time we have to give for dropping a platform it would delay V4. I suspect we will do a follow up 4.1 sometime in the near future that drops .NET Core 3.1. There are also some slight differences between the .NET Core 3.1 target and .NET Standard 2.0 target with dependencies and a few APIs that are only available in .NET Core 3.1 given it has access to more .NET APIs.

[SamuelCox]

Is there any rough timeline on a preview 2 or GA? Excited to use this and see the perf benefits.

[normj]

@SamuelCox We are currently on preview 4. I'm hesitant to put a timeframe because as a team that is involved with almost every AWS release you can imagine we get randomized quite a bit. The intention was for V4 to have a relatively short development cycle to get to GA avoid it getting delayed due to our team's nature of being pulled in so many directions. Given we are in re:Invent time and then moving into the holiday season here in the US I don't see us GAing in 2024.

[juanchovelezpro]

is there an ETA when this will be GA?

[normj]

@mbp Sorry it took me some time to get back to look into the trim warnings but I have put a PR to address the issue. #3749

[normj]

Updated the main description for preview 13 and status of V4.

[antoniofreire]

Hello.

Sorry if this isn't the right place to report, but i'm receiving a 414 response from AWS when trying to receive messages from a SQS queue using the 4.0.0-preview.13 version.

I've added a log to the request message and saw that, on each iteration, the User-Agent header is growing. It appears that the SDK is appending the following string to the header:

aws-sdk-dotnet-coreclr/4.0.0.0 ua/2.1 os/linux#6.1.130.139 md/ARCH#Arm64 lang/.NET_Core#9.0.4 md/aws-sdk-dotnet-core#4.0.0.0 api/SQS#4.0.0.0 md/ClientAsync cfg/init-coll#0 m/b,P,0,E

After the first requet, the SDK duplicates the portion aws-sdk-dotnet-coreclr/4.0.0.0 ua/2.1 os/linux#6.1.130.139 md/ARCH#Arm64 lang/.NET_Core#9.0.4 md/aws-sdk-dotnet-core#4.0.0.0 api/SQS#4.0.0.0 md/ClientAsync cfg/init-coll#0 of the User-Agent

[dscpinheiro]

Hello.

Sorry if this isn't the right place to report, but i'm receiving a 414 response from AWS when trying to receive messages from a SQS queue using the 4.0.0-preview.13 version.

I've added a log to the request message and saw that, on each iteration, the User-Agent header is growing. It appears that the SDK is appending the following string to the header:

aws-sdk-dotnet-coreclr/4.0.0.0 ua/2.1 os/linux#6.1.130.139 md/ARCH#Arm64 lang/.NET_Core#9.0.4 md/aws-sdk-dotnet-core#4.0.0.0 api/SQS#4.0.0.0 md/ClientAsync cfg/init-coll#0 m/b,P,0,E

After the first requet, the SDK duplicates the portion aws-sdk-dotnet-coreclr/4.0.0.0 ua/2.1 os/linux#6.1.130.139 md/ARCH#Arm64 lang/.NET_Core#9.0.4 md/aws-sdk-dotnet-core#4.0.0.0 api/SQS#4.0.0.0 md/ClientAsync cfg/init-coll#0 of the User-Agent

We did release an user-agent related change as part of preview 13 too. Please create a separate issue and we'll take a look (I'm interested in the trying to receive messages part - are you just calling the API in a loop or are you doing something else? Reproduction code will be needed for the investigation).

[normj]

Added an to the main description about some branch changes in preparation for V4 GA.

[normj]

Updated description for the GA announcement of V4. Thank you all for your support and feedback as developed V4.

[normj]

For anybody that noticed the AWSSDK.Extensions.Bedrock.MEAI package mistakenly got released without the preview suffix. Since it has a dependency on Microsoft.Extensions.AI.Abstractions which is still in preview then AWSSDK.Extensions.Bedrock.MEAI should still be preview even if the SDK went GA. We have unlisted the 4.0.0 version and a 4.0.0-preview.14 version will be released soon that will target the GA version of the SDK.

[martincostello]

Just bumping this old comment as it doesn't have a v4 version yet: #3362 (comment)

[martincostello]

Does Aspire.Hosting.AWS also need updating to support v4?

[normj]

@martincostello planning on updating Aspire.Hosting.AWS to V4 soon as well. I was thinking we would bundle that with updating the Aspire dependencies to Aspire 9.2.

[SamWilliamsGS]

This may be a stupid question, but I can't see any information in the documentation about why this change was made and how to migrate existing code to accomodate:

Properties using value types on classes used for making requests and responses have been changed to use nullable value types. Essentially properties of type bool are changed to bool?.

I understand the change, but what does a null mean for these properties? Was it always possible that these response properties could be null and now we're more accurately reflecting it in the SDK types? Or was there a change in AWS responses? Is there a recommended way to migrate existing code that relies on these bools always being a bool? 🙂 Thanks

[normj]

@SamWilliamsGS There are cases where a value for a value type property is optional from a response. For example in CloudWatch Logs the LogGroup has a RetentionInDays property modeled as a integer. Having a retention policy is optional for a log group and if not set then null is the right value for RetentionInDays.

Another problematic situation in V3 is there are some sets of APIs where you call the GET operation to get the object, do some mutation and then send the same instance in on an update cases. There were a class of bugs we had in V3 when we did that round trip and this value types that default to their default value like false for boolean could get misunderstood on the update call because it was originally no value and now looks like an explicit set to false.

To add more context when the SDK generates a type for the service operations the type could be used for either or both request and responses. Even if it is only used in one direction there is no rule it won't change in the future and be used in both directions. So we debated about different behavior depending on request or response usage but that turned not to be possible.

[peter-nguyen-contemi]

Hi, thank you for the great update. I'm not sure if this is the right thread for the question. But it would be great if the AWS.Cryptography.EncryptionSDK package will be updated to AWSSDK v4 too.

[normj]

But it would be great if the AWS.Cryptography.EncryptionSDK package will be updated to AWSSDK v4 too.

Agreed. That library isn't own by our team but I'm reaching out to them to figure out how to get the library updated.

[catester]

I understand the reason for change for value type properties like bool properties but now List<T> properties also come as null, so we need to check for null before foreach loops everywhere now:

if (response.CommonPrefixes != null)
    foreach (var prefix in response.CommonPrefixes)
    {
    }

if (response.S3Objects != null)
    foreach (var s3Object in response.S3Objects)
    {
    }

So get ready to have NullReferenceException crashes with your app when you update to v4.x.
Just documenting these finding for future surfers.

Other value type properties are easier to handle:

Size = s3Object.Size ?? 0;

DateModified = s3Object.LastModified ?? DateTime.MinValue;

[dscpinheiro]

@catester It's mentioned in the announcement, but you can use the Amazon.AWSConfigs.InitializeCollections property to keep the same behavior as V3 if you don't want (or can't) update your code base to check for null collections:

When updating to V4, if you are unsure whether your application is compatible with this runtime behavior change, the static property called Amazon.AWSConfigs.InitializeCollections can be set to true. This changes the runtime behavior to be the same as in V3, which initializes collection properties to an empty collection. Setting this property to true can be a good first stage when updating to V4, but doing so will negate the performance improvements and your ability to determine if a collection property is set or empty by default.

[catester]

Thanks, I read all but forgot that setting when running the app I think. Anyway it's not a big deal, I already added the null checks

[martincostello]

Does Aspire.Hosting.AWS also need updating to support v4?

Just to confirm, lack of support for this is blocking one of my projects from upgrading as it blocks my Aspire host tests.

[normj]

@martincostello We just pushed out version 9.2.0 of Aspire.Hosting.AWS that targets Aspire 9.2 and V4 of AWS SDK for .NET.

[martincostello]

@normj Thanks - successfully upgraded to v4 now 🚀

[maxbog]

@catester It's mentioned in the announcement, but you can use the Amazon.AWSConfigs.InitializeCollections property to keep the same behavior as V3 if you don't want (or can't) update your code base to check for null collections:

When updating to V4, if you are unsure whether your application is compatible with this runtime behavior change, the static property called Amazon.AWSConfigs.InitializeCollections can be set to true. This changes the runtime behavior to be the same as in V3, which initializes collection properties to an empty collection. Setting this property to true can be a good first stage when updating to V4, but doing so will negate the performance improvements and your ability to determine if a collection property is set or empty by default.

@dscpinheiro Hi, we just stumbled upon this problem and I understand the reasons behind possibly null fields and collections. However, I don't understand why the fields and return values are not marked as nullable. This is a major problem because the AWS SDK is asking us to check for null on values that are marked as not-null, which actually is a problem detected by IDEs and the suggested solution by the IDE is to remove the null check.

What plans do you have to address this issue? I'm not complaining about breaking backwards compatibility (that is fine), but about working against the language and IDEs. Right now, using V4 SDK is a sure way to introduce runtime NullReferenceExceptions with the library actively working to make sure those errors are hard to detect.

[kemsky]

AWSSDKUtils.UrlEncode was modified to match RFC 3986 but now it does encode colon :. Either it is not correct or method xml doc must be updated.