[Feature]: Feeding kafka messages through the same broker instance

[PawelPlesniak]

Description

In an attempt to reduce the number of kafka handlers we have per application, we can feed all the messages through a single broker, going to separate topics.

In the case of drunc, this will reduce the number of kafka brokers from 3 (broadcast, opmon, ers through logging) to 1 (broadcast, opmon, and ers ALL through logging handlers). This will also reduce the requirement for the broadcasted decorator, simplifying signature checking.

Potential impact radius

Small/Isolated

Reason for change

No response

Suggested implementations

No response

Testing suggestions

No response

Anything else?

No response

[emmuhamm]

Personal notes on the kafka stuff

So basically we want to avoid having multiple kafkaers, kafkaopmon, kafkaother, etc and rather have a single kafka log handler that can send it to multiple places

th crux of the scenario is this

from the logger side, you can have a couple of different protobuf messages that are constructed. For example issue chain (which is constructed in the publish method), opmon entry (constructed elsewhere), and string (?).

At the end of the day, all that gets sent to the same kafka broker. wahoo

so all you need is to be able to make a kafka publisher first that accepts any protobuf message, and then make a logging handler that wraps it. that logging handler must be flexible enough to choose which one to go to.

A lot of this will be basically making the publisher first and then playing with the different topics streams and stuff before you can even think about how to make a configuration for it.

Some other notes
https://github.com/DUNE-DAQ/drunc/blob/95ad48769bbd674a161ae265144218fe3bd8d8cf/src/drunc/controller/decorators.py#L34 is where the definition of metric publishing for opmon is. Also note that opmon is time series, while ers is instance

In the same file, in the obj.controller_publisher, the commandTime is from protobuf. The type of message (protobuf) itself is defined in https://github.com/DUNE-DAQ/opmonlib/blob/develop/schema/opmonlib/opmon_entry.proto

for opmon, its slightly different
no opmon handler, doesnt exist, but it will 
a standalone opmon no, but a hook thru a logging handler yes 

The opmon version is here
https://github.com/DUNE-DAQ/kafkaopmon/blob/develop/python/kafkaopmon/OpMonPublisher.py

User will construct their protobuf message and then pass it to the kafkapublisher

the end goal is basically a kafka publisher wrapped in a log handler

broker = running instance of kafka, this is what distributes the message

=publisher=producer , this sends the messages (terhcnially interchangabel but not really)

so at the end goal we want a kafkahandler that can publish to any topic
if string or exception go to ers
otherwise if protobuf msg go to opmon
else throw