Skip to content

fix(chain): truncate ChainStepStatus.Output to prevent etcd limit breach #2586

@dapperdivers

Description

@dapperdivers

Problem

ChainStepStatus.Output in chain_types.go:96 is an unbounded string stored in CR status. Knights returning large reports (common in morning briefing with 8 parallel steps) can exceed etcd's 1.5MB object limit, causing status updates to fail and chains to hang.

Impact

  • Chain controller hangs when it can't update status
  • Morning briefing with 8 verbose knight reports is the most likely trigger
  • Silent failure — no error surfaced to the user

Proposed Fix

  • Truncate output in status to 32KB with a [truncated] marker
  • Store full output in a ConfigMap or NATS KV bucket
  • Add outputRef field to ChainStepStatus pointing to the full output location

References

  • Architecture review: drafts/roundtable-architecture-review.md (C4)
  • File: roundtable-operator/api/v1alpha1/chain_types.go

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions