Health Checks

[ghjm]

Use the operator's credentials to connect to the ClickHouse instance and run health checks:

• Access point is available (use chi level service)
• Distributed query check:
  SELECT count() FROM cluster('all-sharded', cluster('all-sharded', system.one))
• Zookeeper check -- run only if zookeeper is a part of CHI spec.configuration:
  SELECT count() FROM system.zookeeper WHERE path = '/'
• No readonly replicas:
  SELECT max(value) FROM cluster('{cluster}', system.metrics) WHERE metric = 'ReadonlyReplica'
• No delayed inserts:
  SELECT value FROM system.metrics WHERE metric = 'DelayedInserts'
• Healthy schema: MaxPartCountForPartition: >150 yellow, >300 red:
  select value from system.asynchronous_metrics where metric='MaxPartCountForPartition'

Provide some kind of warning if the health checks are failing.