How to collect application signals metrics with a custom trace endpoint

[artkhod]

aws-otel-js-instrumentation/aws-distro-opentelemetry-node-autoinstrumentation/src/aws-opentelemetry-configurator.ts

Line 355 in 8661ac1

if (isLambdaEnvironment() && !hasCustomOtlpTraceEndpoint()) {

Hi, I have set up a custom trace endpoint and I also want to setup a custom metrics endpoint. However while doing so I have experimented with various setups and I am not able to get any metrics to be sent to my collector. Is it by design?

[liustve]

Hi @artkhod, could you be more clear on what environment you're running in? If you're running it in a non-Lambda environment, you can follow our ADOT JS guide for sending your traces and metrics to a custom collector endpoint.

The code you referenced is specific to the Lambda environment. If you're explicitly running in Lambda, you can still set the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT to your collector's endpoint (for metrics you also will need to set OTEL_METRICS_EXPORTER=otlp as we disable it by default in Laambda).

Just a heads-up: using the default OTLP exporters in Lambda may lead to increased runtime costs. They rely on HTTP over TCP, which involves connection handshakes and can add latency.

[artkhod]

Hi @liustve, I'm running it in Lambda environment; I'm not so much concerned about latency as I would want OTLP export to localhost (extension) rather than http intake endpoint.

[liustve]

I guess I'm not entirely clear on your Lambda setup but as I mentioned before the OTLP data can be exported to any HTTP endpoint that complies with OTel specifications: https://opentelemetry.io/docs/specs/otel/protocol/exporter/#configuration-options

The code you linked does not prevent you from doing this from within Lambda.

[artkhod]

@liustve as I understand the code I referenced + the next condition

aws-otel-js-instrumentation/aws-distro-opentelemetry-node-autoinstrumentation/src/aws-opentelemetry-configurator.ts

Line 370 in 8661ac1

if (!isLambdaEnvironment()) {

prevent me from exporting application signals metrics over otlp. A span processor responsible for the metrics simply is not going to be registered.

If the trace endpoint remains to be the default (x-ray), then the cloudwatch application signals metrics export works

[liustve]

Hi @artkhod, I think I misunderstood your initial intention. For application signals-specific metrics generated from the ADOT SDK, we disable that in the Lambda environment, regardless of whether a custom OTLP endpoint is set (this is because Application Signals on Lambda already does this on your behalf). I interpreted the initial code snippet you shared as configuring your metrics and spans to be forwarded to a custom OTLP endpoint from Lambda, apologies for the confusion.

[artkhod]

@liustve, so just to clear the confusion, is the layer published to arn:aws:lambda:eu-west-1:615299751070:layer:AWSOpenTelemetryDistroJs:8 any different from what is in this repo's lambda-layer folder. Because I am not sure what do you mean by "Application Signals on Lambda already does this on your behalf". Since as I understand it's the span processors which does that. Is there a way to control that metrics exporter on lambda to be enabled with an env var and disabled by default?

With this issue, I want to say that I would want to send those metrics (on lambda) to some other location over otlp by setting APPLICATION_SIGNALS_EXPORTER_ENDPOINT_CONFIG to be the localhost of the collector extension

[liustve]

@artkhod, to clarify, we have server-side Application Signals that capture telemetry from the ADOT JS layer and generate Application Signals metrics for you.

It is not possible to configure the Application Signals metrics exporter on Lambda to send data directly to your local collector.
That said, the Application Signals metric data is saved in a log group named /aws/applications-signals/data, which you can access and retrieve.

[artkhod]

@liustve, given that the trace endpoint is not custom, right?