Add option to include exclude alerts via their labels

[martialblog]

Fixes #113

See #111

[martialblog]

@gianmarco-mameli @mattttv

This PR introduces --include-label and --exclude-label flags for alerts. Feel free to test this and give feedback.

[gianmarco-mameli]

Hi @martialblog thank you, I'll give a try and I get back to you with some results

[mattttv]

hello @martialblog. thanks for your efforts - awesome!
I tested this in a cluster 4.19.10 without any user prometheus (because the filter in openshift metrics is somewhat limited and gives wrong amounts of checks because it mixes user with core)
i checked with one --include-label severity=MYSEVERITYwhere severity was critical,info,warning,none and everything worked great! nice!
then I combined two params --include-label severity=none --include-label severity=info and it looks like only the last parameter is used - so there is a problem with the logic

check_prometheus alert --include-label severity=none
yields 1 alert
check_prometheus alert --include-label severity=info
yields 23 alerts
check_prometheus alert --include-label severity=none --include-label severity=info
yields 23 alerts
check_prometheus alert --include-label severity=info --include-label severity=none
yields 1 alerts

single exclude-label also seem to work but multiple params seem only to take the first one into account (so the other way round than with include).

Cheers,
Matt

[martialblog]

@mattttv Thanks for testing. Yeah right now it won't check for either, I'll have a look at it.

[martialblog]

@mattttv I updated the implementation. Should work now

[mattttv]

hello @martialblog / Markus!
Thanks again for the quick patch - my previous tests are now flawless. I tried shifting the parameters and there was no change.

so if two labels of the same domain are being used their results are combined with OR, for instance severity:

check_prometheus alert --include-label severity=none
yields 1 alert
check_prometheus alert --include-label severity=info
yields 23 alerts
check_prometheus alert --include-label severity=none --include-label severity=info
yields 24 alerts - WORKS
check_prometheus alert --include-label severity=info --include-label severity=none
yields 24 alerts - WORKS

maybe the help should say something about multiple include-labels - one could assume that all conditions must hold (AND) and not just one of them?
check_prometheus alert --include-label namespace=openshift-insights --include-label severity=info -> yields all the alerts from severity=info not only the alerts where namespace=openshift-insights AND severity=info ... but this is fine behaviour in my opinion

using include and exclude at the same time worked perfectly even with different labels:

--include-label namespace=openshift-insights
yields 2 alerts
--include-label namespace=openshift-insights --exclude-label severity=info
yields 0 alerts - WORKS

and including and excluding at the same time also worked: --include-label namespace=openshift-insights --exclude-label namespace=openshift-insights -> 0 alerts

I think this looks great! thanks again,
Matt

[gianmarco-mameli]

Thanks @mattttv for the tests, I'm a little full this period and I'm trying to find a little time to try that new functions.
But for now I see that your tests are more complete than the ones that I've planned to try myself

[martialblog]

@mattttv Thanks for the feedback.