Channels are not forgotten because "Owning subdaemon onchaind died"

[btweenthebars]

I'm on v25.12.1

I have a handful of channels that

the status on the listpeerchannels showing "Owning subdaemon onchaind died (61952)".
Upon investigation, the lnlog contains events of:

2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: FATAL SIGNAL 11 (version v25.12.1-modded)
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: common/daemon.c:46 (send_backtrace) 0x5b686293d382
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: common/daemon.c:83 (crashdump) 0x5b686293d3d1
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x74456f04532f
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:999 (handle_htlc_onchain_fulfill) 0x5b6862929673
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:1262 (output_spent) 0x5b6862929bf5
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:1596 (handle_onchaind_spent) 0x5b6862929ea9
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:1643 (wait_for_resolved) 0x5b686292a59b
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:2467 (handle_our_unilateral) 0x5b686292bb96
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:3497 (main) 0x5b686292dc12
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x74456f02a1c9
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: ../csu/libc-start.c:360 (__libc_start_main_impl) 0x74456f02a28a
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: (null):0 ((null)) 0x5b6862927654
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: (null):0 ((null)) 0xffffffffffffffff
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: FATAL SIGNAL (version v25.12.1-modded)
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: common/daemon.c:46 (send_backtrace) 0x5b686293d382
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: common/status.c:207 (status_failed) 0x5b68629418c2
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: common/subdaemon.c:18 (status_backtrace_exit) 0x5b6862941a00
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: common/daemon.c:87 (crashdump) 0x5b686293d3df
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x74456f04532f
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:999 (handle_htlc_onchain_fulfill) 0x5b6862929673
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:1262 (output_spent) 0x5b6862929bf5
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:1596 (handle_onchaind_spent) 0x5b6862929ea9
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:1643 (wait_for_resolved) 0x5b686292a59b
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:2467 (handle_our_unilateral) 0x5b686292bb96
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: onchaind/onchaind.c:3497 (main) 0x5b686292dc12
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: ../sysdeps/nptl/libc_start_call_main.h:58 (__libc_start_call_main) 0x74456f02a1c9
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: ../csu/libc-start.c:360 (__libc_start_main_impl) 0x74456f02a28a
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: (null):0 ((null)) 0x5b6862927654
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: backtrace: (null):0 ((null)) 0xffffffffffffffff
2026-03-20T09:15:49.911Z **BROKEN** 02xx-onchaind-chan#23242: STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL
2026-03-20T09:15:49.911Z **BROKEN** 02xx-chan#23242: Owning subdaemon onchaind died (61952)

and the STDERR shows:

lightning_onchaind: FATAL SIGNAL 11 (version v25.12.1-modded)
0x5b686293d335 send_backtrace
       common/daemon.c:38
0x5b686293d3d1 crashdump
       common/daemon.c:83
0x74456f04532f ???
       ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x5b6862929673 handle_htlc_onchain_fulfill
       onchaind/onchaind.c:999
0x5b6862929bf5 output_spent
       onchaind/onchaind.c:1262
0x5b6862929ea9 handle_onchaind_spent
       onchaind/onchaind.c:1596
0x5b686292a59b wait_for_resolved
       onchaind/onchaind.c:1643
0x5b686292bb96 handle_our_unilateral
       onchaind/onchaind.c:2467
0x5b686292dc12 main
       onchaind/onchaind.c:3497
0x74456f02a1c9 __libc_start_call_main
       ../sysdeps/nptl/libc_start_call_main.h:58
0x74456f02a28a __libc_start_main_impl
       ../csu/libc-start.c:360
0x5b6862927654 ???
       _start+0x24:0
0xffffffffffffffff ???
       ???:0
lightning_onchaind: FATAL SIGNAL (version v25.12.1-modded)
0x5b686293d335 send_backtrace
       common/daemon.c:38
0x5b68629418c2 status_failed
       common/status.c:207
0x5b6862941a00 status_backtrace_exit
       common/subdaemon.c:18
0x5b686293d3df crashdump
       common/daemon.c:87
0x74456f04532f ???
       ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0
0x5b6862929673 handle_htlc_onchain_fulfill
       onchaind/onchaind.c:999
0x5b6862929bf5 output_spent
       onchaind/onchaind.c:1262
0x5b6862929ea9 handle_onchaind_spent
       onchaind/onchaind.c:1596
0x5b686292a59b wait_for_resolved
       onchaind/onchaind.c:1643
0x5b686292bb96 handle_our_unilateral
       onchaind/onchaind.c:2467
0x5b686292dc12 main
       onchaind/onchaind.c:3497
0x74456f02a1c9 __libc_start_call_main
       ../sysdeps/nptl/libc_start_call_main.h:58
0x74456f02a28a __libc_start_main_impl
       ../csu/libc-start.c:360
0x5b6862927654 ???
       _start+0x24:0
0xffffffffffffffff ???
       ???:0

and beyond the BROKEN lines there are these events happening:

2026-03-20T09:15:26.019Z INFO 02xx-chan#23242: Restarting onchaind (ONCHAIN): closed in block 859196
2026-03-20T09:15:26.019Z UNUSUAL 02xx-chan#23242: Peer permanent failure in ONCHAIN: Funding transaction spent: onchain txid caxx (reason=unknown)
2026-03-20T09:15:26.019Z UNUSUAL 02xx-chan#23242: Not dropping our unilateral close onchain since we already saw caxx confirm.
2026-03-20T09:15:26.019Z INFO 02xx-chan#23242: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2026-03-20T09:15:33.180Z INFO 02xx-chan#23242: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2026-03-20T09:16:03.692Z INFO 02xx-chan#23242: RBF onchain txid c3xx (fee 134sat) with txid b0xx (fee 267sat)

All funds appear to be recovered.

Also, I have to mention that, these are old channels that have been sitting zombied on my nodes prior before upgrading to "v25.12.1".

[enaples]

Any chance you are running a very old version of bitcoind? It seems that some retrieved blocks have not any witness

[btweenthebars]

You're right. I'm running pruned node with btc-rpc-proxy on top of it. When requesting an old block that doesn't exist on a pruned node, btc-rpc-proxy is supposed to go retrieve the block from the network, and maybe it doesn't retrieve the witness data.

[Schnema1]

@btweenthebars interesting! I did clone my old systemdisc with clonezilla. The node booted an ran like before. So I did upgrade CLN to 25.12 too, but kept Bitcoin Core v29 for some hours. If I remember correct, the CLN started without issue. But the next day I discovered dozends of crash logs.

With debug enabled:
INFO lightningd: Scanning for missed UTXOs from block 908329
.....
DEBUG lightningd: Adding block 921174: 00000000000000000000a8e0eefb0f6e3ed9ca0bb64fa97aecc9c65c64a91718
DEBUG plugin-bcli: Log pruned 593 entries (mem 10528925 -> 6954560)
DEBUG plugin-bcli: Log pruned 264 entries (mem 10496687 -> 4148574)
BROKEN lightningd: FATAL SIGNAL 11 (version v25.12.1)
BROKEN lightningd: backtrace: common/daemon.c:46 (send_backtrace) 0x55e79c2c1497
BROKEN lightningd: backtrace: common/daemon.c:83 (crashdump) 0x55e79c2c14e6
BROKEN lightningd: backtrace: (null):0 ((null)) 0x7f170682d51f
BROKEN lightningd: backtrace: bitcoin/block.c:213 (bitcoin_block_from_hex) 0x55e79c2ebfea
BROKEN lightningd: backtrace: lightningd/bitcoind.c:496 (getrawblockbyheight_callback) 0x55e79c20ffd7
BROKEN lightningd: backtrace: lightningd/plugin.c:701 (plugin_response_handle) 0x55e79c263bc6
BROKEN lightningd: backtrace: lightningd/plugin.c:790 (plugin_read_json) 0x55e79c269175
BROKEN lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x55e79c2fbed9
BROKEN lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x55e79c2fc3b8
BROKEN lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x55e79c2fc475
BROKEN lightningd: backtrace: ccan/ccan/io/poll.c:470 (io_loop) 0x55e79c2fdf14
BROKEN lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x55e79c237f87
BROKEN lightningd: backtrace: lightningd/lightningd.c:1495 (main) 0x55e79c23daec
BROKEN lightningd: backtrace: (null):0 ((null)) 0x7f1706814d8f
BROKEN lightningd: backtrace: (null):0 ((null)) 0x7f1706814e3f
BROKEN lightningd: backtrace: (null):0 ((null)) 0x55e79c20dd64
BROKEN lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff

Upgrade CLN to 25.12.1, Bitcoin Core to v30.2 and back to v29 did not help. Nor to delete the gossip store. I need to come back online as soon as passible.

Did you manage to get your node back running?

@anyone can I provide more information?

[btweenthebars]

Doesn't look like the same issue. My node is running fine, just some closed channels are not cleared/forgotten. I switched to pruned node almost a year ago while running 24.11 which had some other bugs relating to closing channels, however funds were recovered so I didn't bother upgrading. Then I recently upgrade to 25.12.1 which addressed those bugs, then many of the unresolved channels were closed and forgotten. Only these ones are still hanging around. I'm still trying to verify that it's the btc-rpc-proxy that does not get the witness data along with block information. @enaples do you know which RPC methods CLN call to get the info ? Is it the "getblock" with the verbosity=1 ?

[Schnema1]

Ok, thanks for this information. I forgot to mention, after V25.12 booted, I then changed the config file and added autoclean options. Therefore I had the documentation next and carefully added the options. After thinking about my issue, and seen your one, I thought autoclean did "clean" something CLN was not ready or should not have been cleaned (just yet) and resulting in this crash. As the block 921174 is 5 months old. This sounded similar to me, so I added it to your issue / question.

So I will open a new issue then. Thank you!