Maestro V5 - Kafka integration

[leoraba]

Part of #279

Summary of request

Integrate Kafka to use event driven indexing.

Details

Indexing messages. These messages are responsible for triggering the indexing process for a specific repository, study, or analysis. The event message must include key fields such as repositoryCode, studyId, analysisId, and removeAnalysis to specify the nature and type of event being processed.

The repositoryCode identifies the repository being targeted, studyId specifies the study associated with the event, analysisId denotes the unique analysis identifier, and removeAnalysis is a boolean flag indicating whether the analysis should be indexed, if callers set this flag it will do a remove instead of added.

For example:
1 - Index the analysis ID "EGAZ00001254247" use:

{ "repositoryCode" : "collab", "studyId" : "PEME-CA", "analysisId" : "EGAZ00001254247" }

2. Remove index fro analysis ID "EGAZ00001254247" use:

{ "repositoryCode" : "collab", "studyId" : "PEME-CA", "analysisId" : "EGAZ00001254247", "removeAnalysis": false }

3. Index the study "PEME-CA":

{ "repositoryCode" : "collab", "studyId" : "PEME-CA" }

4. Index the whole repository "colab":

{ "repositoryCode" : "collab" }

Features:

• DLQ: Use Dead Letter Queue to provide a mechanism to capture problematic messages for further inspection, debugging, or reprocessing.
• Dynamic configuration: Option configure Kafka via environment variable parameters. i.e.disable kafka, set kafka host, set topic name, disable DLQ, connection max attempts, etc.