Cluster edit and asdb management scripts

.gitignore

@@ -13,3 +13,4 @@ data/
 .charon
 prometheus/prometheus.yml
 commit-boost/config.toml
+

CLAUDE.md

@@ -0,0 +1,151 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This repository contains Docker Compose configurations for running a Charon Distributed Validator Node (CDVN), which coordinates multiple operators to run Ethereum validators. A distributed validator node runs four main components:
+- Execution client (EL): Processes Ethereum transactions
+- Consensus client (CL/beacon node): Participates in Ethereum's proof-of-stake consensus
+- Charon: Obol Network's distributed validator middleware that coordinates between operators
+- Validator client (VC): Signs attestations and proposals through Charon
+
+## Architecture & Multi-Client System
+
+The repository uses a **profile-based multi-client architecture** where different Ethereum client implementations can be swapped via `.env` configuration:
+
+- **Compose file structure**: `compose-el.yml` (execution), `compose-cl.yml` (consensus), `compose-vc.yml` (validator), `compose-mev.yml` (MEV), and `docker-compose.yml` (main/monitoring)
+- **Client selection**: Set via environment variables `EL`, `CL`, `VC`, `MEV` in `.env` (e.g., `EL=el-nethermind`, `CL=cl-lighthouse`, `VC=vc-lodestar`, `MEV=mev-mevboost`)
+- **Profiles**: Docker Compose profiles automatically activate the selected clients via `COMPOSE_PROFILES=${EL},${CL},${VC},${MEV}`
+- **Service naming**: Client services use prefixed names (e.g., `el-nethermind`, `cl-lighthouse`, `vc-lodestar`) while the main compose file uses unprefixed names for backward compatibility
+
+### Supported Clients
+
+- **Execution Layer**: `el-nethermind`, `el-reth`, `el-none`
+- **Consensus Layer**: `cl-lighthouse`, `cl-grandine`, `cl-teku`, `cl-lodestar`, `cl-none`
+- **Validator Clients**: `vc-lodestar`, `vc-nimbus`, `vc-prysm`, `vc-teku`
+- **MEV Clients**: `mev-mevboost`, `mev-commitboost`, `mev-none`
+
+### Key Integration Points
+
+- Charon connects to the consensus layer at `http://${CL}:5052` (beacon node API)
+- Validator clients connect to Charon at `http://charon:3600` (distributed validator middleware API)
+- Consensus layer connects to execution layer at `http://${EL}:8551` (Engine API with JWT auth)
+- MEV clients expose builder API at port `18550`
+
+## Common Commands
+
+### Starting/Stopping the Cluster
+
+```bash
+# Start the full cluster (uses profile from .env)
+docker compose up -d
+
+# Stop specific services
+docker compose down <service-name>
+
+# Stop all services
+docker compose down
+
+# View logs
+docker compose logs -f <service-name>
+
+# Restart after config changes
+docker compose restart <service-name>
+```
+
+### Switching Clients
+
+```bash
+# 1. Stop the old client
+docker compose down cl-lighthouse
+
+# 2. Update .env to change CL variable (e.g., CL=cl-grandine)
+
+# 3. Start new client
+docker compose up cl-grandine -d
+
+# 4. Restart charon to use new beacon node
+docker compose restart charon
+
+# 5. Optional: clean up old client data
+rm -rf ./data/lighthouse
+```
+
+### Testing
+
+```bash
+# Verify containers can be created
+docker compose up --no-start
+
+# Test with debug profile
+docker compose -f docker-compose.yml -f compose-debug.yml up --no-start
+```
+
+## Configuration
+
+### Environment Setup
+
+1. Copy the appropriate sample file: `.env.sample.mainnet` or `.env.sample.hoodi` → `.env`
+2. Set `NETWORK` (mainnet, hoodi)
+3. Select clients by uncommenting the desired `EL`, `CL`, `VC`, `MEV` variables
+4. Configure optional settings (ports, external hostnames, monitoring tokens, etc.)
+
+### Important Environment Variables
+
+- `NETWORK`: Ethereum network (mainnet, hoodi)
+- `EL`, `CL`, `VC`, `MEV`: Client selection (determines which Docker profiles activate)
+- `CHARON_BEACON_NODE_ENDPOINTS`: Override default beacon node (defaults to selected CL client)
+- `CHARON_FALLBACK_BEACON_NODE_ENDPOINTS`: Fallback beacon nodes for redundancy
+- `BUILDER_API_ENABLED`: Enable/disable MEV-boost integration
+- `CLUSTER_NAME`, `CLUSTER_PEER`: Required for monitoring with Alloy/Prometheus
+- `ALERT_DISCORD_IDS`: Discord IDs for Obol Agent monitoring alerts
+
+### Key Directories
+
+- `.charon/`: Cluster configuration and validator keys (created by DKG or add-validators)
+- `data/`: Persistent data for all clients (execution, consensus, validator databases)
+- `jwt/`: JWT secret for execution<->consensus authentication
+- `grafana/`: Monitoring dashboards and configuration
+- `prometheus/`: Metrics collection configuration
+- `scripts/`: Automation scripts for cluster operations
+
+## Cluster Edit Scripts
+
+Located in `scripts/edit/`, these automate complex cluster modification operations. Each has its own README with full usage details:
+
+- **[Add Validators](scripts/edit/add-validators/README.md)** - Add new validators to an existing cluster
+- **[Add Operators](scripts/edit/add-operators/README.md)** - Expand the cluster by adding new operators
+- **[Remove Operators](scripts/edit/remove-operators/README.md)** - Remove operators from the cluster
+- **[Replace Operator](scripts/edit/replace-operator/README.md)** - Replace a single operator in the cluster
+- **[Recreate Private Keys](scripts/edit/recreate-private-keys/README.md)** - Refresh private key shares while keeping the same validator public keys
+- **[Anti-Slashing DB (vc/)](scripts/edit/vc/README.md)** - Export/import/update anti-slashing databases (EIP-3076)
+
+## Monitoring Stack
+
+- **Grafana** (port 3000): Dashboards for cluster health, validator performance
+- **Prometheus**: Metrics collection from all services
+- **Loki**: Log aggregation (optional, via `CHARON_LOKI_ADDRESSES`)
+- **Tempo**: Distributed tracing (debug profile)
+- **Alloy**: Log and metric forwarding (uses `alloy-monitored` labels on services)
+
+Access Grafana at `http://localhost:3000` (or `${MONITORING_PORT_GRAFANA}`).
+
+## Development Workflow
+
+When modifying this repository:
+
+1. **Test container creation** before committing changes to compose files
+2. **Preserve backward compatibility** for existing node operators (data paths, service names)
+3. **Update all sample .env files** when adding new configuration options
+4. **Test client switching** if modifying compose file structure
+5. **Update version defaults** to tested/stable releases
+
+## Important Notes
+
+- **Never commit `.env` files** - they contain operator-specific configuration
+- **JWT secret** in `jwt/jwt.hex` must be shared between EL and CL clients
+- **Cluster lock** in `.charon/cluster-lock.json` is critical - back it up before any edit operations
+- **Validator keys** in `.charon/validator_keys/` must be kept secure and never committed
+- **Data directory compatibility**: When switching VCs, verify the new client can handle existing key state
+- **Slashing protection**: Always export/import ASDB when switching VCs or replacing operators

scripts/README.md

@@ -0,0 +1,27 @@
+# Cluster Edit Automation Scripts
+
+Automation scripts for Charon distributed validator cluster editing operations.
+
+## Documentation
+
+- [Charon Edit Commands](https://docs.obol.org/next/advanced-and-troubleshooting/advanced/)
+- [EIP-3076 Slashing Protection Interchange Format](https://eips.ethereum.org/EIPS/eip-3076)
+
+## Scripts
+
+| Directory | Description |
+|-----------|-------------|
+| [edit/add-validators/](edit/add-validators/README.md) | Add new validators to an existing cluster |
+| [edit/recreate-private-keys/](edit/recreate-private-keys/README.md) | Refresh private key shares while keeping the same validator public keys |
+| [edit/add-operators/](edit/add-operators/README.md) | Expand the cluster by adding new operators |
+| [edit/remove-operators/](edit/remove-operators/README.md) | Remove operators from the cluster |
+| [edit/replace-operator/](edit/replace-operator/README.md) | Replace a single operator in a cluster |
+| [edit/vc/](edit/vc/README.md) | Export/import/update anti-slashing databases (EIP-3076) |
+| [edit/test/](edit/test/README.md) | E2E integration tests for all edit scripts |
+
+## Prerequisites
+
+- `.env` file with `NETWORK` and `VC` variables
+- Docker and `docker compose`
+- `jq`
+

scripts/edit/add-operators/README.md

@@ -0,0 +1,99 @@
+# Add-Operators Scripts
+
+Scripts to automate the [add-operators ceremony](https://docs.obol.org/next/advanced-and-troubleshooting/advanced/add-operators) for Charon distributed validators.
+
+## Overview
+
+These scripts help operators expand an existing distributed validator cluster by adding new operators. This is useful for:
+
+- **Cluster expansion**: Adding more operators for increased redundancy
+- **Decentralization**: Distributing validator duties across more parties
+- **Resilience**: Expanding the operator set while maintaining the same validators
+
+**Important**: This is a coordinated ceremony. All operators (existing AND new) must run their respective scripts simultaneously to complete the process.
+
+> **Warning**: This is an alpha feature in Charon and is not yet recommended for production use.
+
+There are two scripts for the two roles involved:
+
+- **`existing-operator.sh`** - For operators already in the cluster
+- **`new-operator.sh`** - For new operators joining the cluster
+
+## Prerequisites
+
+- `.env` file with `NETWORK` and `VC` variables set
+- Docker running
+- `jq` installed
+- **Existing operators**: `.charon` directory with `cluster-lock.json` and `validator_keys`
+- **New operators**: Charon ENR private key (generated via `--generate-enr`)
+
+## For Existing Operators
+
+Automates the complete workflow for operators already in the cluster:
+
+```bash
+./scripts/edit/add-operators/existing-operator.sh \
+    --new-operator-enrs "enr:-..."
+```
+
+### Options
+
+| Option | Required | Description |
+|--------|----------|-------------|
+| `--new-operator-enrs <enrs>` | Yes | Comma-separated ENRs of new operators |
+| `--dry-run` | No | Preview without executing |
+| `-h, --help` | No | Show help message |
+
+### Workflow
+
+1. **Export ASDB** - Export anti-slashing database from running VC
+2. **Run ceremony** - P2P coordinated add-operators ceremony with all operators
+3. **Update ASDB** - Replace pubkeys in exported ASDB to match new cluster-lock
+4. **Stop containers** - Stop charon and VC
+5. **Backup and replace** - Backup current `.charon/` to `./backups/`, install new configuration
+6. **Import ASDB** - Import updated anti-slashing database
+7. **Restart containers** - Start charon and VC with new configuration
+
+## For New Operators
+
+Two-step workflow for new operators joining the cluster.
+
+**Step 1:** Generate ENR and share with existing operators:
+
+```bash
+./scripts/edit/add-operators/new-operator.sh --generate-enr
+```
+
+**Step 2:** After receiving the existing cluster-lock, run the ceremony:
+
+```bash
+./scripts/edit/add-operators/new-operator.sh \
+    --new-operator-enrs "enr:-...,enr:-..." \
+    --cluster-lock ./received-cluster-lock.json
+```
+
+### Options
+
+| Option | Required | Description |
+|--------|----------|-------------|
+| `--new-operator-enrs <enrs>` | For ceremony | Comma-separated ENRs of ALL new operators |
+| `--cluster-lock <path>` | For ceremony | Path to existing cluster-lock.json |
+| `--generate-enr` | No | Generate new ENR private key |
+| `--dry-run` | No | Preview without executing |
+| `-h, --help` | No | Show help message |
+
+## Current Limitations
+
+- The new cluster configuration will not be reflected on the Obol Launchpad
+- The cluster will have a new cluster hash (different observability identifier)
+- All operators (existing and new) must participate; no partial participation option
+- Cluster threshold remains unchanged
+
+## Related
+
+- [Add-Validators Workflow](../add-validators/README.md)
+- [Remove-Operators Workflow](../remove-operators/README.md)
+- [Recreate-Private-Keys Workflow](../recreate-private-keys/README.md)
+- [Replace-Operator Workflow](../replace-operator/README.md)
+- [Anti-Slashing DB Scripts](../vc/README.md)
+- [Obol Documentation](https://docs.obol.org/next/advanced-and-troubleshooting/advanced/add-operators)