W3C context propagation

[alessandrobologna]

Hi,
somehow I am not able to pass a trace context with the w3c headers (traceparent and tracestate) to a lambda function with application signals enabled and instrumented with this library.

It seems that setting OTEL_LAMBDA_DISABLE_AWS_CONTEXT_PROPAGATION has no effect (normally in the upstream libraries that prevents using the _X_AMZN_TRACE_ID environment variable to determine the parent).

Disabling x-ray on the lambda function doesn't seems to help: if application signal is enabled it seems that the env var _X_AMZN_TRACE_ID is always present.

It seems that the reason is this implementation for the customExtractor function

export const customExtractor = (event: any, _handlerContext: Context): OtelContext => {
  let parent: OtelContext | undefined = undefined;
  const lambdaTraceHeader = process.env[traceContextEnvironmentKey];
  if (lambdaTraceHeader) {
    parent = awsPropagator.extract(
      otelContext.active(),
      { [AWSXRAY_TRACE_ID_HEADER]: lambdaTraceHeader },
      headerGetter
    );
  }
  if (parent) {
    const spanContext = trace.getSpan(parent)?.spanContext();
    if (spanContext && isSpanContextValid(spanContext)) {
      return parent;
    }
  }
  const httpHeaders = event.headers || {};
  const extractedContext = propagation.extract(otelContext.active(), httpHeaders, headerGetter);
  if (trace.getSpan(extractedContext)?.spanContext()) {
    return extractedContext;
  }
  return ROOT_CONTEXT;
};

which effectively precludes extracting the trace context from the event headers (as the first condition is always true).
Is this behavior intentional? It makes it more difficult to trace requests originating from a non x-ray context.

[ezhang6811]

Hi @alessandrobologna,

I think Lambda team plans to release support for passthrough tracing mode soon, which will solve this issue. What region is your lambda function deployed in? Can follow up with an estimated target date.

[alessandrobologna]

Thank you @ezhang6811 . It's in us-east-1, but I think that pass through is already available (see TracingConfig) and the issue is present even if xray is completely disabled.

If you try with a simple lambda, even deployed from the console:

export const handler = async (event) => {
  const traceId = JSON.stringify(
    {
      "traceid": process.env['_X_AMZN_TRACE_ID']
    } 
  );
  console.log(traceId);
  const response = {
    statusCode: 200,
    body: traceId,
  };
  return response;
};

again, without enabling xray tracing (or appsignals), the result will look like:

2025-03-19T21:10:05.650Z	5a33ccbb-f2f0-4e9d-954e-be911a3bc229	INFO	
{
    "traceid": "Root=1-67db32ad-4421bf8e2d0cc8ff5bf4ea8c;Parent=3efc6247c2fd9b1b;Sampled=0;Lineage=1:aa71aef7:0"
}

You can also try adding a function url to invoke it. It seems that the environment variable is set no matter what, so that code prevents any other context propagation but xray.

[ezhang6811]

My understanding of Lambda's current behavior in "passthrough" mode is that, if the upstream request doesn't contain an X-Ray trace header, Lambda will create its own trace context with Sampled=0, suppressing downstream X-Ray tracing.

The new Lambda change corrects this behavior by generating either an invalid X-Ray trace header or none at all when in passthrough mode, which allows the handler to fall back on http headers from upstream service. Then the trace context from your upstream service to propagate to Lambda.

[alessandrobologna]

Thank you @ezhang6811, understood. Do you happen then to have an approximate timeline for this new feature to be introduced?

[ezhang6811]

Rollout starts today, the current best estimate for your region is in a few days time.