[OTEL] Add OpenTelemetry observability support

[royischoss]

Adds OTel-based observability to MLRun CE with automatic Python instrumentation, deployment-mode metrics collection, and Prometheus integration.
https://iguazio.atlassian.net/browse/CEML-685

Changes
OTel operator sub-chart

• Added opentelemetry-operator v0.78.1 as an optional dependency

New templates (templates/opentelemetry/)

• Pre-install hook to label/annotate the namespace for OTel webhook injection and namespace-wide Python auto-instrumentation
• Post-install hook that waits for OTel CRDs before creating the OpenTelemetryCollector and Instrumentation CRs, avoiding operator/CR race condition
• RBAC for hook jobs

Instrumentation CR

• Deployment-mode collector — single pod per namespace, exports metrics to Prometheus on port 8889
• Disabled aws_lambda OTel instrumentor to suppress irrelevant Lambda warnings
• Removed duplicate OTEL_RESOURCE_ATTRIBUTES_* env vars (auto-injected by the operator, caused hides previous definition warnings on every pod)

MLRun API crash fix

• Added mlrun.api.extraEnvKeyValue.PYTHONPATH — OTel operator injects PYTHONPATH=/otel-auto-instrumentation-python:$(PYTHONPATH) using K8s env var expansion, which can't see Docker image ENV vars. Without this explicit
  K8s env var, $(PYTHONPATH) resolves to empty, dropping the MLRun services package path and crashing the API

Admin/non-admin split

• Admin: installs OTel operator with namespace selector webhook, CRs disabled
• User namespace: operator disabled, collector + instrumentation CRs enabled

Other

• Azure Blob storage path helpers in _helpers.tpl (branching on storage.mode)
• Prometheus scrape config simplified to a single otel-collector job

[royischoss]

This is design limitation with no conditions on the values.yaml the scraping will run even if otel is disabled

[davesh0812]

Why do we need this extra scraping job and not just to add web.enable-otlp-receiver flag to the Prometheus deployment??

As you can see here