feat: leanpoint upstreams sync and convert-validator-config

README.md

@@ -24,6 +24,7 @@ A single command line quickstart to spin up lean node(s)
 3. **yq**: YAML processor for automated configuration parsing
    - Install on macOS: `brew install yq`
    - Install on Linux: See [yq installation guide](https://github.com/mikefarah/yq#install)
+4. **Python 3 + PyYAML** (optional, for leanpoint upstreams sync): Required only if you use the automatic leanpoint upstreams sync (tooling server). Install with `pip install pyyaml` or `uv add pyyaml`.
 
 ## Quick Start
 
@@ -95,6 +96,54 @@ NETWORK_DIR=local-devnet ./spin-node.sh --node all --generateGenesis --aggregato
 # is updated in validator-config.yaml before nodes are started
 ```
 
+### Leanpoint deployment
+
+After validator nodes are spun up, leanpoint is deployed so it can monitor them. Behavior depends on deployment mode:
+
+- **Local deployment** (`NETWORK_DIR=local-devnet`, `deployment_mode: local`): Leanpoint runs **locally**. `sync-leanpoint-upstreams.sh` generates `upstreams.json` (with `--docker` so the container can reach host validators at `host.docker.internal`), writes it to `<NETWORK_DIR>/data/upstreams.json`, pulls the latest image, and starts a local Docker container. UI at http://localhost:5555. The container is removed on Ctrl+C cleanup or when you run with `--stop`.
+- **Ansible/remote deployment**: Leanpoint is updated on the **tooling server**. The script rsyncs `upstreams.json` to the server, pulls the latest image there, and recreates the remote container.
+
+**What runs:**
+1. `convert-validator-config.py` reads `validator-config.yaml` and generates `upstreams.json` (validator URLs for health checks).
+2. `sync-leanpoint-upstreams.sh` either deploys leanpoint locally (local devnet) or syncs to the tooling server and recreates the remote container (Ansible).
+
+**Remote defaults:** Tooling server `46.225.10.32`, user `root`, remote path `/etc/leanpoint/upstreams.json`, container name `leanpoint`. Override with env vars (see script header in `sync-leanpoint-upstreams.sh`).
+
+**SSH key for remote sync:** When using Ansible deployment, the tooling server may require a specific SSH key. Pass `--sshKey ~/.ssh/id_ed25519_github` (or `--private-key`) so the sync can succeed.
+
+**Skip via flag:** Pass `--skip-leanpoint` to `spin-node.sh` to skip leanpoint deployment (local and remote). Alternatively set `LEANPOINT_SYNC_DISABLED=1`, or the step is skipped when the convert script or validator config is missing.
+
+**Standalone use of convert script:** You can generate `upstreams.json` for local leanpoint without the tooling server:
+
+```sh
+# From lean-quickstart root
+python3 convert-validator-config.py local-devnet/genesis/validator-config.yaml upstreams.json
+# With --docker for leanpoint in Docker reaching a host devnet:
+python3 convert-validator-config.py local-devnet/genesis/validator-config.yaml upstreams-local-docker.json --docker
+```
+
+Requires Python 3 and PyYAML (`pip install pyyaml`).
+
+### Remote Observability Stack
+
+Every Ansible deployment automatically deploys an observability stack alongside each lean node on remote hosts. No additional flags are needed.
+
+**What gets deployed on each remote host:**
+- **cadvisor** - Container metrics
+- **node-exporter** - System metrics
+- **prometheus** - Scrape local targets, remote_write to central
+- **promtail** - Collect lean node container logs, push to Loki
+
+**How it works:**
+- The local prometheus on each host scrapes the lean node (at its `metricsPort`), cadvisor, node-exporter, and itself, then forwards all data to central prometheus via `remote_write`
+- Promtail discovers the lean node container via Docker socket and pushes logs to central Loki
+
+**Key properties:**
+- **Idempotent**: cadvisor and node-exporter are only started if not already running; prometheus and promtail only restart when their config files change
+- **Persistent**: observability containers are not stopped when lean nodes are stopped — they run independently
+- **Configurable**: central endpoints, images, and ports can be overridden in `ansible/roles/observability/defaults/main.yml`
+- **Remote config path**: `/opt/lean-quickstart/observability/` on each host
+
 ## Args
 
 1. `NETWORK_DIR` is an env to specify the network directory. Should have a `genesis` directory with genesis config. A `data` folder will be created inside this `NETWORK_DIR` if not already there.
@@ -570,6 +619,7 @@ For more details, see the [Docker Desktop host networking documentation](https:/
 This quickstart includes automated configuration parsing:
 
 - **Official Genesis Generation**: Uses PK's `eth-beacon-genesis` docker tool from [PR #36](https://github.com/ethpandaops/eth-beacon-genesis/pull/36)
+- **Leanpoint upstreams sync**: After nodes are spun up, `convert-validator-config.py` and `sync-leanpoint-upstreams.sh` generate `upstreams.json` from `validator-config.yaml`, rsync it to the tooling server, and restart the leanpoint container (see [Leanpoint upstreams sync](#leanpoint-upstreams-sync-tooling-server))
 - **Complete File Set**: Generates `validators.yaml`, `nodes.yaml`, `genesis.json`, `genesis.ssz`, and `.key` files
 - **QUIC Port Detection**: Automatically extracts QUIC ports from `validator-config.yaml` using `yq`
 - **Node Detection**: Dynamically discovers available nodes from the validator configuration

ansible-devnet/genesis/validator-config.yaml

@@ -14,6 +14,7 @@ validators:
       ip: "46.224.123.223"
       quic: 9001
     metricsPort: 9095
+    apiPort: 5055
     isAggregator: false
     count: 1 # number of indices for this node
 
@@ -26,6 +27,7 @@ validators:
       ip: "77.42.27.219"
       quic: 9001
     metricsPort: 9095
+    apiPort: 5055
     isAggregator: false
     devnet: 1
     count: 1
@@ -39,6 +41,7 @@ validators:
       ip: "46.224.123.220"
       quic: 9001
     metricsPort: 9095
+    apiPort: 5055
     isAggregator: false
     count: 1
 
@@ -51,6 +54,7 @@ validators:
       ip: "46.224.135.177"
       quic: 9001
     metricsPort: 9095
+    httpPort: 5055
     isAggregator: false
     count: 1
 
@@ -63,6 +67,7 @@ validators:
       ip: "46.224.135.169"
       quic: 9001
     metricsPort: 9095
+    apiPort: 5055
     isAggregator: false
     count: 1
 
@@ -75,6 +80,7 @@ validators:
       ip: "37.27.250.20"
       quic: 9001
     metricsPort: 9095
+    apiPort: 9095
     isAggregator: false
     count: 1
 
@@ -87,5 +93,6 @@ validators:
       ip: "78.47.44.215"
       quic: 9001
     metricsPort: 9095
+    apiPort: 9095
     isAggregator: false
     count: 1

ansible/playbooks/deploy-nodes.yml

@@ -14,6 +14,9 @@
   hosts: localhost
   connection: local
   gather_facts: yes
+  tags:
+    - deploy
+    - observability
   vars:
     validator_config_file: "{{ genesis_dir }}/validator-config.yaml"
 
@@ -205,6 +208,17 @@
         - setup
         - deploy
 
+    - name: Deploy observability stack
+      include_role:
+        name: observability
+        apply:
+          tags:
+            - observability
+            - deploy
+      tags:
+        - observability
+        - deploy
+
     - name: Deploy this node
       include_tasks: helpers/deploy-single-node.yml
       tags:

ansible/playbooks/site.yml

@@ -3,7 +3,7 @@
 # 1. Clean data directories (if clean_data=true)
 # 2. Generate genesis files locally (including .key files)
 # 3. Copy genesis files to remote hosts
-# 4. Deploy nodes
+# 4. Deploy nodes (observability stack + lean node per host)
 #
 # Usage:
 #   ansible-playbook -i inventory/hosts.yml playbooks/site.yml \

ansible/roles/ethlambda/tasks/main.yml

@@ -38,14 +38,16 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - apiPort
     - isAggregator
   when: node_name is defined
 
 - name: Set node ports and aggregator flag
   set_fact:
     ethlambda_quic_port: "{{ ethlambda_node_config.results[0].stdout }}"
     ethlambda_metrics_port: "{{ ethlambda_node_config.results[1].stdout }}"
-    ethlambda_is_aggregator: "{{ 'true' if (ethlambda_node_config.results[2].stdout | default('') | trim) == 'true' else 'false' }}"
+    ethlambda_api_port: "{{ ethlambda_node_config.results[2].stdout }}"
+    ethlambda_is_aggregator: "{{ 'true' if (ethlambda_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
   when: ethlambda_node_config is defined
 
 - name: Ensure node key file exists
@@ -97,7 +99,8 @@
         --gossipsub-port {{ ethlambda_quic_port }}
         --node-id {{ node_name }}
         --node-key /config/{{ node_name }}.key
-        --metrics-address 0.0.0.0
+        --http-address 0.0.0.0
+        --api-port {{ ethlambda_api_port }}
         --metrics-port {{ ethlambda_metrics_port }}
         {{ '--is-aggregator' if (ethlambda_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}

ansible/roles/grandine/tasks/main.yml

@@ -38,6 +38,7 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - apiPort
     - privkey
     - isAggregator
   when: node_name is defined
@@ -46,8 +47,9 @@
   set_fact:
     grandine_quic_port: "{{ grandine_node_config.results[0].stdout }}"
     grandine_metrics_port: "{{ grandine_node_config.results[1].stdout }}"
-    grandine_privkey: "{{ grandine_node_config.results[2].stdout }}"
-    grandine_is_aggregator: "{{ 'true' if (grandine_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
+    grandine_api_port: "{{ grandine_node_config.results[2].stdout }}"
+    grandine_privkey: "{{ grandine_node_config.results[3].stdout }}"
+    grandine_is_aggregator: "{{ 'true' if (grandine_node_config.results[4].stdout | default('') | trim) == 'true' else 'false' }}"
   when: grandine_node_config is defined
 
 - name: Ensure node key file exists
@@ -101,11 +103,12 @@
         --node-id {{ node_name }}
         --node-key /config/{{ node_name }}.key
         --port {{ grandine_quic_port }}
-        --address 0.0.0.0
         --hash-sig-key-dir /config/hash-sig-keys
-        --metrics
         --http-address 0.0.0.0
-        --http-port {{ grandine_metrics_port }}
+        --http-port {{ grandine_api_port }}
+        --metrics
+        --metrics-address 0.0.0.0
+        --metrics-port {{ grandine_metrics_port }}
         {{ '--is-aggregator' if (grandine_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
       register: grandine_container

ansible/roles/lantern/tasks/main.yml

@@ -29,6 +29,7 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - apiPort
     - privkey
     - isAggregator
   when: node_name is defined
@@ -37,8 +38,9 @@
   set_fact:
     lantern_quic_port: "{{ lantern_node_config.results[0].stdout }}"
     lantern_metrics_port: "{{ lantern_node_config.results[1].stdout }}"
-    lantern_privkey: "{{ lantern_node_config.results[2].stdout }}"
-    lantern_is_aggregator: "{{ 'true' if (lantern_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
+    lantern_api_port: "{{ lantern_node_config.results[2].stdout }}"
+    lantern_privkey: "{{ lantern_node_config.results[3].stdout }}"
+    lantern_is_aggregator: "{{ 'true' if (lantern_node_config.results[4].stdout | default('') | trim) == 'true' else 'false' }}"
   when: lantern_node_config is defined
 
 - name: Ensure node key file exists
@@ -90,7 +92,7 @@
         --node-key-path /config/{{ node_name }}.key
         --listen-address /ip4/0.0.0.0/udp/{{ lantern_quic_port }}/quic-v1
         --metrics-port {{ lantern_metrics_port }}
-        --http-port 5055
+        --http-port {{ lantern_api_port }}
         --hash-sig-key-dir /config/hash-sig-keys
         {{ '--is-aggregator' if (lantern_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}

ansible/roles/lighthouse/tasks/main.yml

@@ -37,6 +37,7 @@
   loop:
     - enrFields.quic
     - metricsPort
+    - apiPort
     - privkey
     - isAggregator
   when: node_name is defined
@@ -45,8 +46,9 @@
   set_fact:
     lighthouse_quic_port: "{{ lighthouse_node_config.results[0].stdout }}"
     lighthouse_metrics_port: "{{ lighthouse_node_config.results[1].stdout }}"
-    lighthouse_privkey: "{{ lighthouse_node_config.results[2].stdout }}"
-    lighthouse_is_aggregator: "{{ 'true' if (lighthouse_node_config.results[3].stdout | default('') | trim) == 'true' else 'false' }}"
+    lighthouse_api_port: "{{ lighthouse_node_config.results[2].stdout }}"
+    lighthouse_privkey: "{{ lighthouse_node_config.results[3].stdout }}"
+    lighthouse_is_aggregator: "{{ 'true' if (lighthouse_node_config.results[4].stdout | default('') | trim) == 'true' else 'false' }}"
   when: lighthouse_node_config is defined
 
 - name: Ensure node key file exists
@@ -104,6 +106,7 @@
         --metrics
         --metrics-address 0.0.0.0
         --metrics-port {{ lighthouse_metrics_port }}
+        --api-port {{ lighthouse_api_port }}
         {{ '--is-aggregator' if (lighthouse_is_aggregator | default('false')) == 'true' else '' }}
         {{ ('--checkpoint-sync-url ' + checkpoint_sync_url) if (checkpoint_sync_url is defined and checkpoint_sync_url | length > 0) else '' }}
       register: lighthouse_container

ansible/roles/observability/defaults/main.yml

@@ -0,0 +1,12 @@
+---
+observability_dir: "/opt/lean-quickstart/observability"
+cadvisor_image: "gcr.io/cadvisor/cadvisor:latest"
+node_exporter_image: "prom/node-exporter:latest"
+prometheus_image: "prom/prometheus:latest"
+promtail_image: "grafana/promtail:latest"
+prometheus_port: 9090
+promtail_port: 9080
+cadvisor_port: 9098
+node_exporter_port: 9100
+remote_write_url: "http://46.225.10.32:9090/api/v1/write"
+loki_push_url: "http://46.225.10.32:3100/loki/api/v1/push"