Node data persistence in persistent volume claims

.github/workflows/test.yml

@@ -42,7 +42,7 @@ jobs:
           - graph_test.py
           - logging_test.py
           - ln_basic_test.py
-          - ln_test.py
+          - ln_graph_test.py
           - onion_test.py
           - plugin_test.py
           - rpc_test.py

docs/persistence.md

@@ -0,0 +1,72 @@
+# Data Persistence
+
+By default, Warnet nodes use ephemeral storage, meaning all data is lost when a pod is deleted or restarted. This document describes how to enable persistent storage to be able to use warnet for persistent development environment, such that blockchain data, wallet information, and other node state can survive pod restarts and network redeployments. This is done with Kubernetes Persistent Volume Claims (PVCs), which persist independently of pod lifecycle.
+
+Persistence is available for:
+- **Bitcoin Core** nodes
+- **LND** nodes
+- **CLN** nodes
+
+## Enabling Persistence
+
+Persistence is configured per-node in the network graph definition. Add a `persistence` section to any node's configuration. This creates a new PVC for that node, which is then mounted to the appropriate data directory inside the container.
+
+Also add `restartPolicy: Always` to the node's configuration to ensure that the pod is restarted if it is deleted or crashes. This is important to ensure proper restart after restart of the kubernetes cluster. If there is a risk of hard resets of the cluster, add `reindex=1` to bitcoin_core config to reindex the blockchain on startup and fix potential corrupted chain state.
+
+### Bitcoin Core Node
+
+```yaml
+bitcoin_core:
+  image: bitcoincore-27.1:latest
+  restartPolicy: Always
+  persistence:
+    enabled: true
+    size: 20Gi # optional, default is 20Gi
+    storageClass: "" # optional, default is cluster default storage class
+    accessMode: ReadWriteOncePod # optional, default is ReadWriteOncePod. For compatibility with older Kubernetes versions, you may need to set this to ReadWriteOnce
+  config: |
+    reindex=1
+```
+
+### Lightning Node
+
+```yaml
+<lnd or cln>:
+  image:
+    tag: <node-version-tag>
+  restartPolicy: Always
+  persistence:
+    enabled: true
+    size: 10Gi # optional, default is 10Gi
+    storageClass: "" # optional, default is cluster default storage class
+    accessMode: ReadWriteOncePod # optional, default is ReadWriteOncePod. For compatibility with older Kubernetes versions, you may need to set this to ReadWriteOnce
+```
+
+## Existing PVCs
+
+To use custom made PVC or PVC from previous deployment, use the `existingClaim` field to reference an existing PVC by name. If the network configuration or namespace did not change, there is no need to explicitly set the `existingClaim`. The existing PVC is used by default, since its generated name matches the default pattern. To explicitly use a PVC set the name like this:
+
+```yaml
+persistence:
+  enabled: true
+  existingClaim: "tank-0001.default-bitcoincore-data"
+```
+
+The generated PVC names follow the pattern:
+`<pod-name>.<namespace>-<node-type>-data`
+
+For example for a bitcoin core node:
+`tank-0001.default-bitcoincore-data`
+
+And for a LND node:
+`tank-0001-ln.default-lnd-data`
+
+Get the list of PVCs in the cluster with `kubectl get pvc -A` and delete any PVCs that are no longer needed with `kubectl delete pvc <pvc-name> -n <namespace>`.
+
+## Mount Paths
+
+When persistence is enabled, the following directories are persisted in the PVCs:
+
+- **Bitcoin Core:** `/root/.bitcoin/`
+- **LND:** `/root/.lnd/`
+- **CLN:** `/root/.lightning/`

resources/charts/bitcoincore/charts/cln/templates/pod.yaml

@@ -56,6 +56,8 @@ spec:
         - mountPath: /root/.lightning/config
           name: config
           subPath: config
+        - mountPath: /root/.lightning
+          name: cln-data
     {{- with .Values.extraContainers }}
     {{- toYaml . | nindent 4 }}
     {{- end }}
@@ -80,6 +82,18 @@ spec:
     - configMap:
         name: {{ include "cln.fullname" . }}
       name: config
+    - name: cln-data
+      {{- if .Values.persistence.enabled }}
+      {{- if .Values.persistence.existingClaim }}
+      persistentVolumeClaim:
+        claimName: {{ .Values.persistence.existingClaim }}
+      {{- else }}
+      persistentVolumeClaim:
+        claimName: {{ include "cln.fullname" . }}.{{ .Release.Namespace }}-cln-data
+      {{- end }}
+      {{- else }}
+      emptyDir: {}
+      {{- end }}
   {{- with .Values.nodeSelector }}
   nodeSelector:
     {{- toYaml . | nindent 4 }}

resources/charts/bitcoincore/charts/cln/templates/pvc.yaml

@@ -0,0 +1,19 @@
+{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "cln.fullname" . }}.{{ .Release.Namespace }}-cln-data
+  labels:
+    {{- include "cln.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/resource-policy": keep
+spec:
+  accessModes:
+    - {{ .Values.persistence.accessMode }}
+  resources:
+    requests:
+      storage: {{ .Values.persistence.size }}
+  {{- if .Values.persistence.storageClass }}
+  storageClassName: {{ .Values.persistence.storageClass }}
+  {{- end }}
+{{- end }}

resources/charts/bitcoincore/charts/cln/values.yaml

@@ -82,6 +82,15 @@ startupProbe:
       - "-c"
       - "lightning-cli createrune > /working/rune.json"
 
+# Node data persistence configuration. Create persistent volume claim, or use an existing one.
+persistence:
+  enabled: false
+  storageClass: ""
+  accessMode: ReadWriteOncePod
+  size: 10Gi
+  # Use existing persistent volume claim instead of creating a new one.
+  existingClaim: ""
+
 # Additional volumes on the output Deployment definition.
 volumes:
   - name: working

resources/charts/bitcoincore/charts/lnd/templates/pod.yaml

@@ -53,7 +53,15 @@ spec:
             - /bin/sh
             - -c
             - |
-              rm -rf /root/.lnd/data/chain
+              MACAROON_PATH="/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
+
+              # If wallet is already initialized, unlock it and exit
+              if [ -f "$MACAROON_PATH" ]; then
+                until [ "$(curl --silent --insecure https://localhost:8080/v1/unlockwallet --data "{\"wallet_password\":\"AAAAAAAAAAA=\"}")" == "{}" ]; do
+                  sleep 5
+                done
+                exit 0
+              fi
 
               until curl --silent --insecure https://localhost:8080/v1/genseed > /tmp/genseed.json; do
                 sleep 5
@@ -79,7 +87,7 @@ spec:
         - mountPath: /root/.lnd/tls.cert
           name: config
           subPath: tls.cert
-        - name: shared-volume
+        - name: lnd-data
           mountPath: /root/.lnd/
     {{- with .Values.extraContainers }}
     {{- toYaml . | nindent 4 }}
@@ -95,7 +103,7 @@ spec:
         - "--macaroonpath=/root/.lnd/data/chain/bitcoin/{{ .Values.global.chain }}/admin.macaroon"
         - "--httplisten=0.0.0.0:{{ .Values.circuitbreaker.httpPort }}"
       volumeMounts:
-        - name: shared-volume
+        - name: lnd-data
           mountPath: /root/.lnd/
         - name: config
           mountPath: /tls.cert
@@ -108,8 +116,18 @@ spec:
     - configMap:
         name: {{ include "lnd.fullname" . }}
       name: config
-    - name: shared-volume
+    - name: lnd-data
+      {{- if .Values.persistence.enabled }}
+      {{- if .Values.persistence.existingClaim }}
+      persistentVolumeClaim:
+        claimName: {{ .Values.persistence.existingClaim }}
+      {{- else }}
+      persistentVolumeClaim:
+        claimName: {{ include "lnd.fullname" . }}.{{ .Release.Namespace }}-lnd-data
+      {{- end }}
+      {{- else }}
       emptyDir: {}
+      {{- end }}
   {{- with .Values.nodeSelector }}
   nodeSelector:
     {{- toYaml . | nindent 4 }}

resources/charts/bitcoincore/charts/lnd/templates/pvc.yaml

@@ -0,0 +1,19 @@
+{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "lnd.fullname" . }}.{{ .Release.Namespace }}-lnd-data
+  labels:
+    {{- include "lnd.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/resource-policy": keep
+spec:
+  accessModes:
+    - {{ .Values.persistence.accessMode }}
+  resources:
+    requests:
+      storage: {{ .Values.persistence.size }}
+  {{- if .Values.persistence.storageClass }}
+  storageClassName: {{ .Values.persistence.storageClass }}
+  {{- end }}
+{{- end }}

resources/charts/bitcoincore/charts/lnd/values.yaml

@@ -9,13 +9,13 @@ image:
   repository: lightninglabs/lnd
   pullPolicy: IfNotPresent
   # Overrides the image tag whose default is the chart appVersion.
-  tag: "v0.19.0-beta"
+  tag: "v0.20.1-beta"
 
 imagePullSecrets: []
 nameOverride: ""
 fullnameOverride: ""
 
-podLabels: 
+podLabels:
   app: "warnet"
   mission: "lightning"
 
@@ -65,12 +65,11 @@ resources: {}
   #   cpu: 100m
   #   memory: 128Mi
 
-
 livenessProbe:
   exec:
     command:
-    - pidof
-    - lnd
+      - pidof
+      - lnd
   failureThreshold: 3
   initialDelaySeconds: 60
   periodSeconds: 5
@@ -85,6 +84,15 @@ readinessProbe:
     port: 10009
   timeoutSeconds: 1
 
+# Node data persistence configuration. Create persistent volume claim, or use an existing one.
+persistence:
+  enabled: false
+  storageClass: ""
+  accessMode: ReadWriteOncePod
+  size: 10Gi
+  # Use existing persistent volume claim instead of creating a new one.
+  existingClaim: ""
+
 # Additional volumes on the output Deployment definition.
 volumes: []
 # - name: foo
@@ -128,6 +136,6 @@ defaultConfig: ""
 channels: []
 
 circuitbreaker:
-  enabled: false  # Default to disabled
+  enabled: false # Default to disabled
   image: carlakirkcohen/circuitbreaker:attackathon-test
-  httpPort: 9235
+  httpPort: 9235

resources/charts/bitcoincore/templates/pod.yaml

@@ -109,7 +109,17 @@ spec:
       {{- toYaml . | nindent 4 }}
     {{- end }}
     - name: data
+      {{- if .Values.persistence.enabled }}
+      {{- if .Values.persistence.existingClaim }}
+      persistentVolumeClaim:
+        claimName: {{ .Values.persistence.existingClaim }}
+      {{- else }}
+      persistentVolumeClaim:
+        claimName: {{ include "bitcoincore.fullname" . }}.{{ .Release.Namespace }}-bitcoincore-data
+      {{- end }}
+      {{- else }}
       emptyDir: {}
+      {{- end }}
     - name: config
       configMap:
         name: {{ include "bitcoincore.fullname" . }}

resources/charts/bitcoincore/templates/pvc.yaml

@@ -0,0 +1,19 @@
+{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: {{ include "bitcoincore.fullname" . }}.{{ .Release.Namespace }}-bitcoincore-data
+  labels:
+    {{- include "bitcoincore.labels" . | nindent 4 }}
+  annotations:
+    "helm.sh/resource-policy": keep
+spec:
+  accessModes:
+    - {{ .Values.persistence.accessMode }}
+  resources:
+    requests:
+      storage: {{ .Values.persistence.size }}
+  {{- if .Values.persistence.storageClass }}
+  storageClassName: {{ .Values.persistence.storageClass }}
+  {{- end }}
+{{- end }}

resources/charts/bitcoincore/values.yaml

@@ -15,7 +15,7 @@ imagePullSecrets: []
 nameOverride: ""
 fullnameOverride: ""
 
-podLabels: 
+podLabels:
   app: "warnet"
   mission: "tank"
 
@@ -61,12 +61,11 @@ resources: {}
   #   cpu: 100m
   #   memory: 128Mi
 
-
 livenessProbe:
   exec:
     command:
-    - pidof
-    - bitcoind
+      - pidof
+      - bitcoind
   failureThreshold: 12
   initialDelaySeconds: 5
   periodSeconds: 5
@@ -78,6 +77,14 @@ readinessProbe:
   successThreshold: 1
   timeoutSeconds: 10
 
+# Node data persistence configuration. Create persisten volume claim, or use an existing one.
+persistence:
+  enabled: false
+  storageClass: ""
+  accessMode: ReadWriteOncePod
+  size: 20Gi
+  # Use existing persistent volume claim instead of creating a new one.
+  existingClaim: ""
 
 # Additional volumes on the output Deployment definition.
 volumes: []

resources/scenarios/commander.py

@@ -173,8 +173,13 @@ def run_test(self):
     @staticmethod
     def ensure_miner(node):
         wallets = node.listwallets()
+
         if "miner" not in wallets:
-            node.createwallet("miner", descriptors=True)
+            allwallets = node.listwalletdir()
+            if "'miner'" in str(allwallets):
+                node.loadwallet("miner")
+            else:
+                node.createwallet("miner", descriptors=True)
         return node.get_wallet_rpc("miner")
 
     @staticmethod

resources/scenarios/ln_init.py

@@ -63,6 +63,15 @@ def gen(n):
             mining_tank.rpc_timeout = 6000
             return self.generatetoaddress(mining_tank, n, miner_addr, sync_fun=self.no_op)
 
+        ##
+        # Run only once per network, do this after p2p connections
+        # so everyone is on the same chain if there is a restart
+        # with inconsistent persistence.
+        ##
+        if mining_tank.getblockcount() > 0:
+            self.log.info("Chain already exists, exiting")
+            return
+
         self.log.info("Locking out of IBD...")
         gen(1)