[DPE-8772] Stereo mode: add watcher role to the PostgreSQL charm

[marceloneppel]

Summary

This PR turns the earlier stereo-mode watcher proof of concept into a deployment mode of the main postgresql charm. The same charm can now be deployed either as a full PostgreSQL unit or as a lightweight watcher/witness by setting role=watcher at deploy time.

This keeps stereo mode inside a single charm codebase while also expanding watcher support for multi-cluster and async-replication scenarios.

The implementation includes:

• An immutable deploy-time role config with postgresql and watcher modes
• Unified watcher relation support in the main charm via watcher-offer and watcher
• Patroni-compatible per-relation raft_controller services with dynamic port allocation
• Watcher-side get-cluster-status and trigger-health-check actions
• Richer relation data for timeline, per-member lag, TLS status, standby clusters, and watcher voting
• Clear failure messages for PostgreSQL-only actions run against watcher units
• New unit, integration, and spread coverage for stereo mode, async replication, and Juju spaces

How It Works

┌──────────────────────────┐      watcher-offer / watcher      ┌──────────────────────────┐
│ PostgreSQL cluster       │◄─────────────────────────────────►│ postgresql               │
│ role=postgresql          │                                    │ role=watcher            │
│ - Patroni + PostgreSQL   │                                    │ - Raft witness only     │
│ - Raft voter             │                                    │ - No PostgreSQL server  │
└──────────────────────────┘                                    └──────────────────────────┘

• The default deployment keeps the normal PostgreSQL workload and exposes watcher-offer.
• A deployment with role=watcher skips the PostgreSQL workload and runs only the watcher logic.
• Each watcher relation gets its own Patroni-compatible Raft controller instance, data directory, and port.
• The watcher is dynamically excluded from voting when the PostgreSQL unit count is odd, avoiding an even-sized Raft membership that would reduce partition tolerance.
• The watcher publishes and consumes enough relation data to report cluster topology, health, and async-replication context without requiring separate manual Patroni queries.

Deployment

For a local or single-host LXD demo, deploy the watcher with profile=testing so the AZ co-location safety check does not block the watcher when all units land in the same availability zone.

# Deploy a 2-node PostgreSQL cluster
juju deploy postgresql pg --channel 16/edge --base ubuntu@24.04 --num-units 2

# Deploy the same charm as a watcher
# Use profile=testing for local demos where all units may share the same AZ.
juju deploy postgresql pg-watcher --channel 16/edge --base ubuntu@24.04 \
  --config role=watcher \
  --config profile=testing

# Relate PostgreSQL to the watcher
juju integrate pg:watcher-offer pg-watcher:watcher

# Inspect watcher-reported status
juju run pg-watcher/0 get-cluster-status
juju run pg-watcher/0 trigger-health-check

For production deployments, keep profile=production and place the watcher in a different availability zone from the PostgreSQL units. If the watcher is co-located in the same AZ, it will enter blocked with an AZ co-location status by design.

A single watcher deployment can also relate to more than one PostgreSQL cluster, which is covered by the async-replication stereo-mode tests added in this branch.

Test Coverage

• Unit tests for role validation, watcher relation handling, watcher requirer behaviour, and Raft controller service generation
• Integration tests for stereo mode deployment, failover, watcher actions, odd/even voting behaviour, async replication, and cross-space networking
• Spread coverage for stereo mode

Checklist

• I have added or updated any relevant documentation.
• I have cleaned any remaining cloud resources from my accounts.

Supercedes #1401.

[codecov]

Codecov Report

❌ Patch coverage is 45.48311% with 694 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.79%. Comparing base (95a5784) to head (2a88644).

Files with missing lines	Patch %	Lines
src/relations/watcher_requirer.py	42.79%	238 Missing and 16 partials ⚠️
src/relations/watcher.py	54.39%	141 Missing and 20 partials ⚠️
src/raft_controller.py	50.00%	107 Missing and 15 partials ⚠️
src/watcher_health.py	28.20%	55 Missing and 1 partial ⚠️
src/charm.py	45.16%	37 Missing and 14 partials ⚠️
src/cluster.py	6.00%	47 Missing ⚠️
src/relations/async_replication.py	40.00%	1 Missing and 2 partials ⚠️

❌ Your project check has failed because the head coverage (64.79%) is below the target coverage (70.00%). You can increase the head coverage or adjust the target coverage.

❗ There is a different number of reports uploaded between BASE (95a5784) and HEAD (2a88644). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (95a5784)	HEAD (2a88644)
	2	1

Additional details and impacted files

@@             Coverage Diff             @@
##           16/edge    #1516      +/-   ##
===========================================
- Coverage    70.43%   64.79%   -5.64%     
===========================================
  Files           15       19       +4     
  Lines         4282     5545    +1263     
  Branches       694      889     +195     
===========================================
+ Hits          3016     3593     +577     
- Misses        1057     1675     +618     
- Partials       209      277      +68     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.