OpenTelemetry integration creates span ID mismatch between LLM_REQUEST and LLM_RESPONSE/LLM_ERROR (BigQuery Analytics Plugin)

[evekhm]

When OpenTelemetry tracing is enabled in the ADK, the span_id no longer matches between LLM_REQUEST and its corresponding LLM_RESPONSE (and LLM_ERROR) events.
This happens because before_model_callback runs before the framework's internal call_llm span begins. As a result, the BigQueryAgentAnalyticsPlugin grabs the parent span ID (e.g. from the execute_tool span) to log the LLM_REQUEST. Then, the model executes inside the child call_llm span. The on_model_error_callback and after_model_callback then execute within that child span.
Because of this, LLM_REQUEST.span_id == LLM_RESPONSE.parent_span_id.

This can break any downstream data warehouses or analysis tools that rightfully expect an atomic request/response LLM pair to share the identical span_id. Notably, without OpenTelemetry, span_id matches perfectly between all three events

When querying raw event tables with OTel enabled:
TOOL_STARTING to TOOL_COMPLETED still match perfectly on span_id.
AGENT_STARTING to AGENT_COMPLETED still match perfectly on span_id.
The issue is completely isolated to the LLM flow callbacks.

[surajksharma07]

@evekhm the mismatch comes down to timing. before_model_callback fires just before the call_llm span opens in _call_llm_async, which means LLM_REQUEST ends up picking up the parent span context while LLM_RESPONSE captures the child span.
That's what's causing the pairing to break.

One thing worth trying: move _handle_before_model_callback inside the with tracer.start_as_current_span('call_llm') block so all three callbacks are working within the same span context.
Could you test that on your end and see if it fixes the pairing for you?

If it holds up after some thorough testing, it'd be great to carry that forward into #4854!

[brucearctor]

@surajksharma07 Spot on with the diagnosis! PR #4854 implements exactly that — but found one additional subtlety worth noting:

Just moving the before_model_callback inside the call_llm span isn't enough on its own. The after_model_callback runs inside an async for loop that consumes yields from the error handler, which creates a child generate_content span via use_inference_span. Because async generators preserve their span context across yields, the after model callback ends up seeing the child span instead of call_llm.

The fix in #4854 addresses both sides:

1. Moves the before model callback inside the call_llm span
2. Wraps the after model callback with trace.use_span to re-activate the call_llm span