DD_EXTENSION | ERROR | Failed to extract request body: Failed to buffer the request body: length limit exceeded for large invocation payloads (>3 MB)

[oulusoy]

Description

Lambda functions that receive large invocation payloads (e.g. KafkaEvents >3 MB via MSK Event Source Mappings) produce the following error:

DD_EXTENSION | ERROR | Failed to extract request body: Failed to buffer the request body: length limit exceeded

The error originates from the Rust-based next-gen extension agent. The older Go-based agent does not have this issue.

Environment

• Runtime: Java 21 (public.ecr.aws/lambda/java:21)
• Extension version: v92 (public.ecr.aws/datadog/lambda-extension:92)
• Java tracer: dd-java-agent.jar (latest)
• Event source: MSK (Kafka) Event Source Mapping delivering KafkaEvent payloads of 3-6 MB

Reproduction

Invoke any Lambda function with a payload larger than ~3 MB while DD_TRACE_ENABLED=true:

aws lambda invoke --function-name my-function --payload file://large-kafka-event.json out.json

The function executes successfully, but the extension logs:

DD_EXTENSION | ERROR | Failed to extract request body: Failed to buffer the request body: length limit exceeded

Distinction from #899

Issue #899 was fixed in v89 by increasing the trace payload limit from 2 MiB to 50 MiB. That limit controls the communication between the language tracer and the extension's internal trace agent.

This issue is about a different limit: the extension's internal buffer for the Lambda invocation event body. This limit remains at ~3 MB and was not addressed by the v89 fix.

Investigation

Environment variables — none had any effect:

Variable	Value	Result
DD_UNIVERSAL_INSTRUMENTATION	false	Error persists
DD_CAPTURE_LAMBDA_PAYLOAD	false	Error persists
DD_EXPERIMENTAL_ENABLE_PROXY	true	Error persists
DD_SERVERLESS_APPSEC_ENABLED	true	Error persists

Extension versions — only Go-based agents are unaffected:

Version	Agent	Mode	5 MB payload
v65	Go	default	No error
v87	Go+Rust	DD_EXTENSION_VERSION=compatibility	No error
v87	Go+Rust	default (Rust)	Error
v89	Rust-only	default	Error
v92	Rust-only	default	Error

Why DD_LOGS_CONFIG_PROCESSING_RULES does not help:

DD_LOGS_CONFIG_PROCESSING_RULES only filters logs that flow through the extension's log collection pipeline (function stdout/stderr and platform events, collected via the Lambda Telemetry API). The DD_EXTENSION | ERROR messages are written directly to stdout/stderr by the extension binary itself and go straight to CloudWatch — they do not pass through the log collection pipeline where processing rules are applied.

Impact

• The error is non-blocking — Lambda execution and tracing via the Java Agent continue to work
• Invocation metadata extraction fails for affected invocations
• Produces significant log noise in production for any Lambda processing large Kafka batches
• Causes unnecessary costs (CloudWatch + Datadog ingestion) that cannot be filtered at the source

Request

Please consider either:

1. Increasing the invocation event body buffer limit to match the Lambda payload size limit (6 MB synchronous / 256 KB async)
2. Making it configurable via an environment variable
3. Gracefully skipping body extraction for oversized payloads without logging an ERROR

• The error is non-blocking — Lambda execution and tracing via the Java Agent continue to work
• Invocation metadata extraction fails for affected invocations
• Produces significant log noise in production for any Lambda processing large Kafka batches
• Causes unnecessary costs (CloudWatch + Datadog ingestion) that cannot be filtered at the source

[litianningdatadog]

@oulusoy Thank you for reporting this. We’ll investigate it promptly.