DD_EXTENSION | ERROR | Max retries exceeded, returning request error - Sharp increase since upgrading Lambda & Extension version.

[AJMcKane]

Hi there,

Since upgrading the extension layer from v92 to v93 as well as the node layer version from 133 to 135, we've noticed two new behaviours:

Firstly, the error message in the issue title is now logging as an Error, I believe this is an intentional change as of PR #1036, though it doesn't quite match the details in that PR.

However secondly and more concerningly, we've seen a sharp increase in the amount of these errors coming through, to note, nothing else in our config has changed. We are still seeing successful APM traces and spans being ingested, at roughly the same volumes, but these logs would tell me we're dropping a significant amount of requests.

We're using the datadog-cdk-constructs-v2 to create our lambda wrapper and our config is here:

const dd = new DatadogLambda(stack, `Datadog-${serviceName}`, {
        nodeLayerVersion: 135,
        extensionLayerVersion: 93,
        apiKey,
        site,
        env: app.stage,
        service: serviceName,
        captureLambdaPayload: false,
        enableDatadogTracing: true,
        enableMergeXrayTraces: true,
        datadogAppSecMode: DatadogAppSecMode.EXTENSION,
        sourceCodeIntegration: false,
 })

Happy to provide more information if needed.

[jglien]

We are seeing this too, in both Node and Python Lambdas. Digging through older logs I did find this:

DD_EXTENSION | ERROR | Max retries exceeded, returning HTTP error status=413 Payload Too Large attempts=1

Perhaps trace payloads are too large sometimes in our environments, and this has just been ignored until now.

[dgolombek]

We're seeing this as well with v93. We also see it log DD_EXTENSION | ERROR | TRACES | Request failed: No requests sent immediately afterwards.

[jchrostek-dd]

Will start investigating.

[jchrostek-dd]

@AJMcKane Can you confirm that you weren't seeing these log messages at an info level before and now they are just at error level which makes it look like a sharp increase. I would like to know if this is indeed a regression or just a visibility of the logs.

Either way, we will look into the actual error messages being emitted.

[jchrostek-dd]

@AJMcKane Also, are there any other log messages that you see with details of why the request failed?

[AJMcKane]

@jchrostek-dd Thanks for the response:

Here's our grouped view of these log instances, you can quite clearly see where our deployment went out with the extension layer bump and then the greatly increased quantities of this log occurring.

I'm afraid this is all the context we get from the Ddog extension logs, a catch all error about retries exceeding limit, then the generic Request failed error.

[jchrostek-dd]

Found what might be the issue - between v92 and v93 we started pooling the clients. This isn't great for lambda due to freeze/thaw cycles. I merged a PR to disable this.

We are targeting getting a release out (v94) either end of this week or early next week. I'll post here when we do.

I don't think there are any other changes between v92 and v93 that would have impacted this. But if v94 doesn't fix the issue, I will continue to investigate.

[jchrostek-dd]

@dgolombek Did you start seeing DD_EXTENSION | ERROR | Max retries exceeded, returning HTTP error status=413 Payload Too Large attempts=1 because the log level was corrected or is this a regression/new error or you are seeing a spike in these (ignoring log level)?

[dgolombek]

@dgolombek Did you start seeing DD_EXTENSION | ERROR | Max retries exceeded, returning HTTP error status=413 Payload Too Large attempts=1 because the log level was corrected or is this a regression/new error or you are seeing a spike in these (ignoring log level)?

My apologies, we only saw that particular error a handful of times (3). Starting March 2nd when we updated to the new version, we've seen 1160 instances each of:

• DD_EXTENSION | ERROR | Max retries exceeded, returning request error
• DD_EXTENSION | ERROR | TRACES | Request failed: No requests sent