CNV-80608: PR15 - k8s: add orphan AlertRelabelConfig garbage collection

[sradco]

When an operator or manual action removes rules from a PrometheusRule (or deletes the CR entirely),
the ARCs created by the plugin for
classification/drop/stamp become orphaned.

This adds a GC pass to sync() that detects and deletes orphaned ARCs by comparing live rule IDs against each ARC's alertRuleId annotation. GitOps-managed ARCs
are never auto-deleted; a warning is logged instead.

[machadovilaca]

we should try to filter by some label in the query instead fetching all

[sradco]

Hi,
The GC workload is inherently "scan all, check each".
There's no way to pre-filter for orphans without already knowing which ones are orphans.
It's an informer cache read, not an API query.

[machadovilaca]

but then we are filtering the results by liveRuleIDs
are we not?

[sradco]

Updated

[machadovilaca]

not much use as we are already logging the deletion with the same level

[sradco]

+1. Removed. Thanks.

[machadovilaca]

this does not belong here

we want to run when PrometheusRule is updated or deleted
that should be handled in pkg/k8s/prometheus_rule.go
where you would add event handlers for those actions

logic for the garbage collection and respective unit test should also live in a dedicated file with matching names
pkg/k8s/relabeled_rules_gc_test.go is probably not convetional

[sradco]

this does not belong here

we want to run when PrometheusRule is updated or deleted that should be handled in pkg/k8s/prometheus_rule.go where you would add event handlers for those actions

I disagree.
There are two separate PrometheusRule informers in this codebase, and they serve different purposes:
prometheus_rule.go is a CRUD manager. It wraps the Get/List/Update/Delete/AddRule operations.
It has no workqueue, no reconciliation loop, no event handlers.
relabeled_rules.go is a controller, It has a workqueue, event handlers, and a reconciliation loop.

Running GC on secret changes is unnecessary but harmless. It's an local cache scan that's a no-op when nothing is orphaned. Running on initial-sync and the 15-minute resync is actually important.
It catches orphans left behind from before the plugin started (for example: plugin crash, missed events). A pure event-driven approach would miss those.

logic for the garbage collection and respective unit test should also live in a dedicated file with matching names pkg/k8s/relabeled_rules_gc_test.go is probably not convetional

+1. Moved gcOrphanedARCs to alert_relabel_config_gc.go and renamed the test file to alert_relabel_config_gc_test.go

[machadovilaca]

the cache receives the event handlers for all the prometheus rule resources on startup
so thats not an issue

and although the garbage collector performs operations on ARC resources
those operations are necessary only due to operations on related prometheus rule resources
so the handler should be somewhere related

it actually can still be in this file, because we track the relationship between those resources
but should still track prometheus rule events, given by rrm.queue.Add("prometheus-rule-sync")

[sradco]

@machadovilaca thank you for the review. Please see my comments