feat(opentelemetry sink): add native metric to OTLP conversion#24897
Open
szibis wants to merge 17 commits intovectordotdev:masterfrom
Open
feat(opentelemetry sink): add native metric to OTLP conversion#24897szibis wants to merge 17 commits intovectordotdev:masterfrom
szibis wants to merge 17 commits intovectordotdev:masterfrom
Conversation
Add encoding support for converting native Vector metrics to OTLP protobuf format (ExportMetricsServiceRequest). This enables Vector metrics from any source to be sent through OTLP sinks without requiring pre-formatted OTLP structure. Metric type mapping: - Counter → Sum (monotonic, Delta/Cumulative per MetricKind) - Gauge → Gauge - AggregatedHistogram → Histogram - AggregatedSummary → Summary - Distribution → Histogram (samples → buckets) - Set → Gauge (cardinality count) - Sketch → Gauge with warning (unsupported) Tag decomposition reverses the decode-path flattening: - resource.* tags → resource attributes - scope.name/version → InstrumentationScope - scope.* tags → scope attributes - All other tags → data point attributes Includes 36 tests covering type conversions, tag decomposition, roundtrip fidelity, and edge cases.
8 tasks
Update opentelemetry sink CUE requirements to reflect native metric auto-conversion support. Add comprehensive otlp-native-conversion docs with metric type mapping, tag decomposition reference, and use case examples. Add kvlist to spelling expect list.
This feature targets the 0.55.0 release, not 0.54.0.
This was referenced Mar 11, 2026
Contributor
|
Created DOCS-13660 for documentation team review |
… encode Apply review feedback patterns from PR vectordotdev#24621: - Replace `as u64` with `u64::try_from().ok()` for timestamp conversion - Replace `as u64`/`as f64` with `u64::from()`/`f64::from()` for sample.rate - Remove unwrap() in Distribution bucket overflow guard, use saturating index clamping instead
…ogram conversion NaN and Infinity sample values were not excluded from the accumulation loop in Distribution-to-Histogram conversion, even though they were already filtered from the boundary list. This caused NaN to poison the total_sum and misroute samples via binary_search_by. Non-finite samples are now skipped in the accumulation loop, matching the boundary filter.
Sketch metrics were converted to an empty gauge (value 0.0), causing silent data corruption downstream. Now Sketch metrics produce a Metric with no data field, which OTLP receivers will ignore. Also fixes the rate-limit key to use internal_log_rate_limit (Vector convention).
Add clippy::cast_precision_loss annotation on values.len() as f64 in Set conversion (usize→f64 is lossy for very large sets but acceptable). Fix test to use u64::try_from instead of bare 'as u64' cast, matching the production code pattern.
Add tests covering NaN/Infinity distribution samples, non-finite gauge and counter values, zero-rate samples, Sketch metric dropping, tag prefix collision routing, scope custom attributes, and non-finite histogram/summary sums.
Sketch metrics are now dropped with a warning instead of emitting an empty gauge, matching the code change.
5 tasks
Contributor
Author
|
Companion decode-side fix: #24905 adds decode for missing |
… tags Update decompose_metric_tags to handle 4 special tags as proto-level structural fields rather than generic attributes: - resource.dropped_attributes_count → Resource.dropped_attributes_count - resource.schema_url → ResourceMetrics.schema_url - scope.dropped_attributes_count → InstrumentationScope.dropped_attributes_count - scope.schema_url → ScopeMetrics.schema_url This ensures round-trip fidelity with fix/otlp-decode-missing-fields (vectordotdev#24905) once merged, while remaining backward-compatible (graceful defaults of 0 / empty string) before that PR merges.
pront
reviewed
Mar 12, 2026
Co-authored-by: Pavlos Rontidis <pavlos.rontidis@gmail.com>
…nversion - Use DataType::all_bits() instead of manual OR (reviewer nit) - Split OtlpSerializer doc into "Pre-formatted OTLP events" and "Native Vector events" sections for clarity - Sketch metrics now truly dropped (return None from native_metric_to_otlp_request instead of including empty metric) - Document resource.*/scope.* tag prefix reservation for OTLP mapping - Add sample config to changelog fragment - Remove orphaned docs/examples/otlp-native-conversion.md (content folded into encoder docstring)
Add explicit user-facing documentation about resource.*/scope.* tag prefix reservation in OTLP encoding: - CUE sink docs: new "Metric tag prefix conventions" section - Changelog: explicitly notes prefix reservation and behavior for non-OTLP sources using these prefixes coincidentally Addresses review feedback requesting explicit documentation for the implicit prefix reservation behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds encoding support for converting native Vector metrics to OTLP protobuf format (
ExportMetricsServiceRequest), enabling Vector metrics from any source to be sent through OTLP sinks without pre-formatted OTLP structure.This is the metrics counterpart to log/trace native conversion in #24621.
Related Issues
host_metrics→opentelemetrysink now works directly)codec: otlpnow properly encodes native metrics instead of requiring manual protobuf construction)User Impact
Before this PR, sending native Vector metrics to an OTLP collector required either:
use_otlp_decoding: trueon the source (passthrough only, no transforms possible)After this PR:
Conversion Architecture
graph LR subgraph "Vector Native Metrics" C[Counter] G[Gauge] H[AggregatedHistogram] S[AggregatedSummary] D[Distribution] Set[Set] end subgraph "OTLP Protobuf" Sum[Sum<br/>monotonic] OG[Gauge] OH[Histogram] OS[Summary] end C -->|"Incremental→Delta<br/>Absolute→Cumulative"| Sum G --> OG H -->|"Incremental→Delta<br/>Absolute→Cumulative"| OH S --> OS D -->|"samples→buckets"| OH Set -->|"cardinality count"| OGType Mapping
CounterSum(monotonic)Incremental→ Delta,Absolute→ CumulativeGaugeGaugeAggregatedHistogramHistogramIncremental→ Delta,Absolute→ CumulativeAggregatedSummarySummaryDistributionHistogramSetGaugeSketchTag Decomposition
Reverses the decode-path flattening from
build_metric_tags:resource.*Resource.attributes[](prefix stripped)scope.nameInstrumentationScope.namescope.versionInstrumentationScope.versionscope.*(other)InstrumentationScope.attributes[](prefix stripped)DataPoint.attributes[]Safety
From/TryFrom(no bareascasts for narrowing)internal_log_rate_limitused for rate-limited warnings (Vector convention)Test Plan
cargo clippyclean,cargo fmtclean