feat: output batching

[csatib02]

Adds support for output batching, as it is now the recommended approach.

Manually tested:

Deployed the Log-generator:

helm upgrade --install --wait --namespace example-tenant-ns log-generator oci://ghcr.io/kube-logging/helm-charts/log-generator

Deployed the following manifests:

apiVersion: v1
kind: Namespace
metadata:
  name: openobserve
---
apiVersion: v1
kind: Service
metadata:
  name: openobserve
  namespace: openobserve
spec:
  clusterIP: None
  selector:
    app: openobserve
  ports:
    - name: http
      port: 5080
      targetPort: 5080
---
apiVersion: v1
kind: Service
metadata:
  name: openobserve-otlp-grpc
  namespace: openobserve
spec:
  clusterIP: None
  selector:
    app: openobserve
  ports:
    - name: otlp-grpc
      port: 5081
      targetPort: 5081
---
# create statefulset
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openobserve
  namespace: openobserve
  labels:
    name: openobserve
spec:
  serviceName: openobserve
  replicas: 1
  selector:
    matchLabels:
      name: openobserve
      app: openobserve
  template:
    metadata:
      labels:
        name: openobserve
        app: openobserve
    spec:
      securityContext:
        fsGroup: 2000
        runAsUser: 10000
        runAsGroup: 3000
        runAsNonRoot: true
      # terminationGracePeriodSeconds: 0
      containers:
        - name: openobserve
          image: public.ecr.aws/zinclabs/openobserve:v0.14.0
          env:
            - name: ZO_ROOT_USER_EMAIL
              value: root@example.com
            - name: ZO_ROOT_USER_PASSWORD
              value: Complexpass#123
            - name: ZO_DATA_DIR
              value: /data
          # command: ["/bin/bash", "-c", "while true; do sleep 1; done"]
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 4096m
              memory: 2048Mi
            requests:
              cpu: 256m
              memory: 50Mi
          ports:
            - containerPort: 5080
              name: http
            - containerPort: 50801
              name: otlp-grpc
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        # storageClassName: default
        # NOTE: You can increase the storage size
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Namespace
metadata:
  name: collector
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    nsSelector: example-tenant
  name: example-tenant-ns
---
apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Collector
metadata:
  name: example-collector
spec:
  controlNamespace: collector
  tenantSelector:
    matchLabels:
      collectorLabel: example-collector
---
apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Tenant
metadata:
  labels:
    collectorLabel: example-collector
  name: example-tenant
spec:
  subscriptionNamespaceSelectors:
    - matchLabels:
        nsSelector: example-tenant
  logSourceNamespaceSelectors:
    - matchLabels:
        nsSelector: example-tenant
---
apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Subscription
metadata:
  name: subscription-sample-1
  namespace: example-tenant-ns
spec:
  condition: "true"
  outputs:
    - name: otlp-test-output-1
      namespace: example-tenant-ns
---
apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Subscription
metadata:
  name: subscription-sample-2
  namespace: example-tenant-ns
spec:
  condition: "true"
  outputs:
    - name: otlp-test-output-2
      namespace: example-tenant-ns
---
apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Output
metadata:
  name: otlp-test-output-1
  namespace: example-tenant-ns
spec:
  otlp:
    endpoint: openobserve-otlp-grpc.openobserve.svc.cluster.local:5081
    sending_queue:
      batch:
        flush_timeout: 60s
        min_size: 1048576
    headers:
      Authorization: "Basic cm9vdEBleGFtcGxlLmNvbTpJb1RrQ1daUEZRcUIwMkxh"
      organization: default
      stream-name: default
    tls:
      insecure: true
---
apiVersion: telemetry.kube-logging.dev/v1alpha1
kind: Output
metadata:
  name: otlp-test-output-2
  namespace: example-tenant-ns
spec:
  otlp:
    endpoint: openobserve-otlp-grpc.openobserve.svc.cluster.local:5081
    sending_queue:
      batch:
        flush_timeout: 60s
        min_size: 1048576
    headers:
      Authorization: "Basic cm9vdEBleGFtcGxlLmNvbTpJb1RrQ1daUEZRcUIwMkxh"
      organization: default
      stream-name: default
    tls:
      insecure: true

Resulting config:

exporters:
  otlp/example-tenant-ns_otlp-test-output-1:
    endpoint: openobserve-otlp-grpc.openobserve.svc.cluster.local:5081
    headers:
      Authorization: Basic cm9vdEBleGFtcGxlLmNvbTpJb1RrQ1daUEZRcUIwMkxh
      organization: default
      stream-name: default
    retry_on_failure:
      enabled: true
      max_elapsed_time: 0
    sending_queue:
      batch:
        flush_timeout: 60s
        min_size: 1048576
      enabled: true
      queue_size: 1000
    tls:
      insecure: true

[Copilot]

Pull request overview

This PR adds support for output batching by extending sending_queue configuration for exporters, aligning with the OpenTelemetry Collector’s recommended approach.

Changes:

• Extend QueueSettings / exporter config generation to support sending_queue.batch plus new queue behavior fields (sizer, wait_for_result, block_on_overflow) and update the default queue_size to 1000.
• Update CRDs and configuration-generation tests/fixtures to reflect the new queue defaults and batching config.
• Update lint/format configuration and docs to describe output batching as the recommended approach.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/resources/otel_conf_gen/pipeline/components/exporter/common.go	Adds mapping for new sending queue fields and sending_queue.batch; updates default queue size to 1000.
pkg/resources/otel_conf_gen/pipeline/components/exporter/otlp_exporter_test.go	Updates expectations for queue defaults and adds a test case for queue batch settings.
pkg/resources/otel_conf_gen/pipeline/components/exporter/otlphttp_exporter_test.go	Updates expected default queue_size from 100 → 1000.
pkg/resources/otel_conf_gen/pipeline/components/exporter/fluent_forward_exporter_test.go	Updates expected default queue_size from 100 → 1000.
pkg/resources/otel_conf_gen/otel_col_conf_test_fixtures/complex.yaml	Updates fixture exporter queue sizes to 1000.
api/telemetry/v1alpha1/otlp_config.go	Introduces QueueBatch and updates QueueSettings fields (replacing old blocking).
api/telemetry/v1alpha1/zz_generated.deepcopy.go	Adds deepcopy support for QueueBatch and updates deepcopy for new QueueSettings fields.
config/crd/bases/telemetry.kube-logging.dev_outputs.yaml	CRD schema updates for new queue/batch fields and updated queue defaults.
charts/telemetry-controller/crds/telemetry.kube-logging.dev_outputs.yaml	Helm-synced CRD schema updates matching config/crd/bases.
README.md	Documents output batching (sending_queue.batch) as recommended; clarifies batch processor is not recommended for new configs.
.golangci.yml	Enables formatters (gci/gofmt/gofumpt/goimports), enables revive/gocyclo, and removes staticcheck.
controllers/telemetry/suite_test.go	Removes //nolint: revive from dot imports.
controllers/telemetry/controller_integration_test.go	Removes //nolint: revive from dot imports; adjusts scaffold marker and adds //nolint: gci.
cmd/main.go	Adds //nolint: gci and retains the kubebuilder scaffold marker (now with a leading space).
.gitignore	Ignores .envrc.
CLAUDE.md	Adds repository guidance for Claude Code usage and project structure/commands.
.claude/settings.json	Adds Claude Code project settings and sandbox/permissions configuration.

Comments suppressed due to low confidence (5)

controllers/telemetry/suite_test.go:25

• revive is enabled in .golangci.yml, but these dot-imports no longer have //nolint: revive. Unless revive is configured to allow dot imports, this will fail lint. Either restore the //nolint: revive suppression here (or at file level) or adjust the revive rule set to permit dot imports in Ginkgo test files.

	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"

controllers/telemetry/controller_integration_test.go:22

• These dot-imports will likely be flagged now that revive is enabled and the prior //nolint: revive suppressions were removed. Consider re-adding the suppression for these lines (or switching to named imports) to keep CI lint passing.

	. "github.com/onsi/ginkgo/v2"
	. "github.com/onsi/gomega"

controllers/telemetry/controller_integration_test.go:32

• The kubebuilder scaffold marker is now // +kubebuilder:scaffold:imports (note the space). Kubebuilder’s scaffolding tools typically look for the exact //+kubebuilder:scaffold:imports marker; adding a space may prevent future scaffolding from inserting imports correctly. Suggest reverting to the exact marker text and, if needed, keep gci suppression on a separate line.

	//nolint: gci
	// +kubebuilder:scaffold:imports

cmd/main.go:38

• Same as above: the scaffold marker has a leading space (// +kubebuilder:scaffold:imports). If kubebuilder expects //+kubebuilder:scaffold:imports exactly, this may break future scaffold operations. Consider restoring the exact marker string and keeping //nolint: gci separate.

	//nolint: gci
	// +kubebuilder:scaffold:imports

.golangci.yml:45

• staticcheck was previously enabled but is no longer listed under linters.enable. If this was accidental, re-enable it to keep catching correctness issues that other linters typically miss. If intentional, consider documenting the rationale in this config to avoid future reintroductions.

  enable:
    - bodyclose
    - errcheck
    - ineffassign
    - misspell
    - nolintlint
    - revive
    - gocyclo
    - unconvert
    - unparam
    - unused
    - whitespace

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.