Skip to content

Fix log fanout delivery#20528

Open
isagi-y22 wants to merge 1 commit intoerigontech:mainfrom
isagi-y22:fanout
Open

Fix log fanout delivery#20528
isagi-y22 wants to merge 1 commit intoerigontech:mainfrom
isagi-y22:fanout

Conversation

@isagi-y22
Copy link
Copy Markdown

A failed Send in SubscribeLogs currently skips the rest of subscribers for the same log (continue outerLoop), which causes observable log loss for healthy subscribers. This patch keeps broadcasting the current log to remaining subscribers, still removes broken subscribers after dispatch, and adds TestLogsFilter_SendFailure_DoesNotSkipHealthySubscribers to reproduce the bug and lock in the correct behavior.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes log fanout behavior in the private API logs subscription so that a single subscriber send failure does not prevent healthy subscribers from receiving the same log.

Changes:

  • Update distributeLogs to continue broadcasting a log to remaining subscribers even if one Send fails.
  • Add a regression test that simulates failing subscribers and asserts healthy subscribers still receive all logs and failing subscribers are removed.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
node/privateapi/logsfilter.go Adjusts send-failure control flow so one broken subscriber doesn’t abort per-log fanout.
node/privateapi/logsfilter_test.go Adds a failing server stub and a regression test covering send-failure fanout behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 194 to 197
if err := filter.sender.Send(proto); err != nil {
filtersToDelete[filterId] = filter
continue outerLoop
continue
}
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Send error, this code schedules the filter for deletion inside distributeLogs, but the corresponding subscribeLogs handler still has a deferred removeLogsFilter(filterId, filter). If distributeLogs deletes/subtracts first, the deferred removeLogsFilter will later subtract the same filter again, potentially driving aggLogsFilter counters negative and leaving EmptyLogSubscription state incorrect. Consider making removal idempotent (e.g., in removeLogsFilter: check that logsFilters[filterId] still exists and matches the same filter pointer before subtracting/deleting), or avoid subtracting/deleting here and instead mark/close the subscriber so the deferred cleanup is the single removal path.

Copilot uses AI. Check for mistakes.
Comment on lines 194 to 197
if err := filter.sender.Send(proto); err != nil {
filtersToDelete[filterId] = filter
continue outerLoop
continue
}
Copy link

Copilot AI Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once a subscriber Send fails, the filter stays in a.logsFilters until the end of distributeLogs, so subsequent logs in the same batch will keep attempting Send (and failing) for that subscriber. Since you already track filtersToDelete, you can skip Send for filterIds already marked to reduce repeated failures/overhead when logs contains many entries.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants