(s3-deployment): times out with large files

[relm923]

Describe the bug

Potentially caused by #33698

Starting with 2.185.0 s3-deployments have begun timing out resulting in UPDATE_FAILED stack updates. There is nothing obvious in the CustomResource CloudWatch logs

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.184.1

Expected Behavior

S3 deployment succeeds with small and large files

Current Behavior

S3 deployment hangs on large files

Reproduction Steps

Create s3 deployment with 10mb file and deploy. CustomResource will hang causing CloudFormation stack to timeout and rollback

Possible Solution

Theory: attempting to read each file into memory is causing the CustomResource/Lambda to hang when the file is too large.

Pre 2.185.0 files were read line by line. In 2.185.0 each file is read into memory to determine if it contains valid JSON

Assuming reading large files is the root cause:

• Skip replace_markers entirely if no markers passed
• Only attempt to parse files with .json extension

Additional Information/Context

No response

CDK CLI Version

2.1006.0

Framework Version

No response

Node.js Version

22.14.0

OS

MacOS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Investigation

Original Issue (#22661):

• Problem with unescaped quotes in JSON files when using Source.jsonData
• Strings containing quotes weren't being properly escaped, resulting in invalid JSON

The Fix (PR #33698):

Added JSON detection and special handling in the Lambda
Changed from simple string replacements to full JSON parsing/serialization
The key change was reading the entire file into memory to check if it's JSON:

aws-cdk/packages/@aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py

Lines 335 to 341 in 2cff67e

	try:
	with open(filename, 'r', encoding='utf-8') as f:
	content = f.read()
	is_json = is_json_content(content)
	except UnicodeDecodeError:
	# If we can't read as text, it's definitely not JSON
	is_json = False

The Regression (#34002):

• While this fix properly handles quoted strings in JSON
• It introduced a performance issue by reading entire files into memory
• This causes Lambda timeouts/memory issues with large files (10MB+)
• Previously files were processed line by line

Issue Chain

graph LR
    subgraph Before
        B1[Line by Line Processing]
        B2[Small Files OK]
        B3[No Quote Escaping]
        B1 --> B2
        B1 --> B3
    end

    subgraph Fix_PR33698
        F1[Added JSON Detection]
        F2[Full File Read]
        F3[Quote Escaping Added]
        F1 --> F2
        F1 --> F3
    end

    subgraph After
        A1[Memory Issues]
        A2[Timeouts]
        A3[Large Files Fail]
        F2 --> A1
        A1 --> A2
        A2 --> A3
    end

    B3 --> F1
    F3 --> A3

Loading

[pahud]

Yes this is a regression. I am making it a p0 and escalating for review.

[scorbiere]

I will open a PR to address this issue soon.
A workaround while the fix is not release yet is to increase the memory limit of the Lambda used in the custom resource, like in this example:

import * as cdk from 'aws-cdk-lib';
import { Bucket } from 'aws-cdk-lib/aws-s3';
import { Construct } from 'constructs';

export class Repro34002Stack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, {
      ...props,
      stackName: 'TestStack'
    });

    const b = new Bucket(this, 'DeployDestination')
    const s3Deployment = new cdk.aws_s3_deployment.BucketDeployment(this, 'testDeploy', {
      sources: [cdk.aws_s3_deployment.Source.asset('./testAssetComplex')],
      destinationBucket: b,
      memoryLimit: 1024,
    });
  }
}

The default value is 128, so depending on the file size, a value like 256 or 512 may be high enough.