Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 124 additions & 69 deletions cometbft/next/docs/experimental/lib-p2p.mdx
Original file line number Diff line number Diff line change
@@ -1,58 +1,74 @@
---
order: 1
title: Lib-P2P
title: "Lib-P2P Networking"
description: "Performant and scalable peer-to-peer networking for CometBFT"

Check warning on line 4 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L4

Did you really mean 'Performant'?
icon: "waypoints"

Check warning on line 5 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L5

Did you really mean 'waypoints'?
---

# Experimental Lib-P2P Support
<Warning>
This is an experimental feature.
It has been tested under a range of network conditions and perturbations, but you should validate it
for your specific workload before enabling it on a production mainnet chain.

Check warning on line 11 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L11

Did you really mean 'mainnet'?
</Warning>

CometBFT includes an experimental networking layer based on [go-libp2p](https://libp2p.io/).
<Warning>
Use only alongside [AdaptiveSync](/cometbft/next/docs/core/block-sync#adaptivesync)
</Warning>

This is an experimental networking layer based on [go-libp2p](https://libp2p.io/).
It adds a new transport and connection-management layer for peer-to-peer communication,
while keeping the reactor-facing CometBFT API unchanged.

Actors still use the same core p2p concepts (`Switch`, `Peer`, `PeerSet`, `Reactor`, envelopes).
Actors still use the same core p2p concepts (`Switch`, `Peer`, `PeerSet`, `Reactor`, `Envelope`, ...).
The transport implementation under those abstractions is `lib-p2p` instead of `comet-p2p`.

`lib-p2p` is a widely used networking stack with production-ready peer-to-peer features,
lib-p2p is a widely used networking stack with production-ready peer-to-peer features,
and implementations across many languages and transport protocols (TCP, QUIC, WebSockets, and more).

This feature is implemented alongside `combined_mode`, which blends blocksync with consensus reactors.
That improves network liveness by letting slower nodes continue ingesting blocks while consensus is running.

You can refer to the implementation in the CometBFT codebase here:

- [`lp2p`](https://github.com/cometbft/cometbft/tree/main/lp2p)
- [`internal/autopool`](https://github.com/cometbft/cometbft/tree/main/internal/autopool)
- [lp2p](https://github.com/cometbft/cometbft/tree/main/lp2p)
- [internal/autopool](https://github.com/cometbft/cometbft/tree/main/internal/autopool)

### Performance and Liveness
## Performance and Liveness

Check warning on line 33 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L33

Did you really mean 'Liveness'?

In high-load conditions, legacy `comet-p2p` can become a networking bottleneck for modern blockchain workloads:

Check warning on line 35 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L35

Did you really mean 'blockchain'?

- It's more prone to congestion under concurrent message pressure,
- Stream/message handling is less effective at scaling with traffic spikes,
- It is more prone to congestion under concurrent message pressure.
- Stream/message handling is less effective at scaling with traffic spikes.
- This limits end-to-end throughput when the rest of the stack is optimized.

The `lib-p2p` integration addresses this with native stream-oriented transport, concurrent receive pipelines,
and autoscaled worker pools per reactor, which helps reduce queue pressure and improve message flow under load.

Check warning on line 42 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L42

Did you really mean 'autoscaled'?
Beyond raw throughput, this also improves network liveness by making peer communication and block propagation

Check warning on line 43 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L43

Did you really mean 'liveness'?
more resilient under sustained congestion and sudden load spikes.

In our benchmarks, together with additional performance improvements across the stack, we reached over 2000 TPS,
and `lib-p2p` has been one of the key unblockers enabling that result.

Check warning on line 47 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L47

Did you really mean 'unblockers'?

## Differences in transport and peer identification
## Differences in Transport and Peer IDs

`lib-p2p` uses its own [peer ID format](https://github.com/libp2p/specs/blob/master/peer-ids/peer-ids.md),
which is different from `comet-p2p`. The two formats are not compatible.

```
```text
# comet-p2p peer ID format
539ffc12a0ac78970dab31ae8cdcfdd4285b2162@10.186.73.3:26656

# lib-p2p peer ID format
{ host = "10.186.73.3:26656", id = "12D3KooWRuTppVZGE7qhanfsHfmzkWUZnnRbTxbgWYvKibij9niy" }
```

`lib-p2p` uses QUIC instead of TCP in this integration, so you need UDP open in your firewall.
To print your node's lib-p2p peer ID from the CLI:

```bash
cometbft show-node-id --libp2p
# e.g. 12D3KooWJwoqHMXukQGFg425582Jr2Cq9VLE6MtbRt21hRrudjqM
```

<Warning>
lib-p2p uses QUIC instead of TCP, ensure UDP communication is allowed in your firewall.
</Warning>

## Configuration

Expand All @@ -61,111 +77,150 @@

By default, the node listens on UDP port `26656`.

<Info>
To validate successful configuration, check logs for: <br/>
**"EXPERIMENTAL: go-libp2p transport is enabled."**
</Info>


```toml
[p2p.libp2p]

# Enabled set true to use go-libp2p for networking instead of CometBFT's p2p.
enabled = true

# Disables resource manager.
# Warning! This might consume all of the system's resources.
disable_resource_manager = false

# Bootstrap peers to connect to
# format: { host, id, private (opt), persistent (opt), unconditional (opt) }
# dns resolution is also supported (e.g. "example.com:26656")
# DNS resolution is also supported (e.g. "example.com:26656")
bootstrap_peers = [
{ host = "10.186.73.3:26656", id = "12D3KooWRuTppVZGE7qhanfsHfmzkWUZnnRbTxbgWYvKibij9niy", persistent = true },
{ host = "10.186.73.5:26656", id = "12D3KooWHjC8SJFVpAvY3qpM5PPeXSpQZvLxcZb7Tjr1kHMLEFtS", persistent = true },
{ host = "10.186.73.6:26656", id = "12D3KooWJFbLcqdPpNP7E1EXC4tDiPtxEGDM6K7RXpDsdWmVnjSu", persistent = true },
]

# Options for scaling concurrent p2p message queues.
[p2p.libp2p.scaler]
min_workers = 4

Check warning on line 103 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L103

Did you really mean 'min_workers'?
max_workers = 32

Check warning on line 104 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L104

Did you really mean 'max_workers'?

# downscale concurrency if P90 latency of message processing
# is longer than the threshold
threshold_latency = "100ms"

Check warning on line 108 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L108

Did you really mean 'threshold_latency'?

## Optional per-reactor override.
# [[p2p.libp2p.scaler.overrides]]
# reactor = "MEMPOOL"
# min_workers = 8

Check warning on line 113 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L113

Did you really mean 'min_workers'?
# max_workers = 512

Check warning on line 114 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L114

Did you really mean 'max_workers'?
# threshold_latency = "500ms"

Check warning on line 115 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L115

Did you really mean 'threshold_latency'?

# Resource limits mode:
# - disabled: no limits (unsafe on untrusted/public networks)
# - default: lib-p2p defaults
# - custom: enforce explicit peer/stream caps
[p2p.libp2p.limits]
mode = "default"

# Used only in custom mode.
# max_peers = 200

Check warning on line 125 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L125

Did you really mean 'max_peers'?
# max_peer_streams = 16

Check warning on line 126 in cometbft/next/docs/experimental/lib-p2p.mdx

View check run for this annotation

Mintlify / Mintlify Validation (cosmos-docs) - vale-spellcheck

cometbft/next/docs/experimental/lib-p2p.mdx#L126

Did you really mean 'max_peer_streams'?
```

Each host, similar to `comet-p2p`, supports the following options:
- `persistent` - ensures peer is always (re)connected (even after removal)
- `unconditional` - not affected by the max number of peers limit (see notes on limitations)
- `private` - the peer will not be gossiped to other peers (see notes on limitations)
Each bootstrap peer supports these options:

To validate successful configuration, check the logs for the following message:
- `persistent`: ensures the peer is always (re)connected.
- `unconditional`: not affected by the max number of peers limit.
- `private`: peer is not gossiped to other peers.

```text
Using go-libp2p transport!
```
### Queue Scaler

Using `lib-p2p` with `combined_mode` enabled is recommended:
The queue scaler controls reactor receive concurrency using a throughput/latency feedback loop.

```toml
[blocksync]
- `min_workers`, `max_workers`: lower/upper worker bounds per reactor.
- `threshold_latency`: target processing-latency threshold.
- `overrides`: per-reactor values (case-insensitive reactor name).

version = "v0"
In most deployments, the defaults are enough and should be used first.
Tune only when metrics show persistent queue growth, elevated receive latency, or poor throughput.

# Experimental Combined mode (bool):
#
# Run both BLOCKSYNC and CONSENSUS for improved liveness, connectivity, and performance.
combined_mode = true
```
### Resource Manager

## Implementation details
Resource manager mode determines connection and stream limits:

- `default`: uses libp2p autoscaled limits and sane built-in protocol caps.
- `custom`: disables most cometbft p2p limits but enforces explicit `max_peers` and `max_peer_streams` caps.
- `disabled`: no limits, useful for controlled benchmarking and local testing.

Recommended tuning flow if defaults are not enough:

1. Start with `mode = "default"` and observe metrics under representative load.
2. If limits are still unclear, run short, controlled tests with `mode = "disabled"` to discover required headroom.
3. Move to `mode = "custom"` and set conservative `max_peers` / `max_peer_streams` caps based on measurements.
4. Re-test and keep safety margin; avoid running public networks long-term in `disabled` mode.

You can find more details about lib-p2p resource manager here:
- [libp2p/go-libp2p: p2p/host/resource-manager/README.md](https://github.com/libp2p/go-libp2p/blob/master/p2p/host/resource-manager/README.md)
- [libp2p/go-libp2p: p2p/host/resource-manager/limit_defaults.go](https://github.com/libp2p/go-libp2p/blob/062200be7aa1d18a0f54eefb17b0dbe2e96f0a79/p2p/host/resource-manager/limit_defaults.go#L663)

## Implementation Details

From the CometBFT actor-model perspective, the API stays the same:
`Reactor`, `Peer`, `PeerSet`, `Switch`, and envelope flow remain compatible,
so existing reactors can run without protocol-level rewrites.

At the connection layer, lib-p2p replaces CometBFT secret connection with lib-p2p native
identity and [secure handshake mechanisms](https://github.com/libp2p/specs/blob/master/tls/tls.md).
This means peer session establishment, encryption negotiation, and remote identification are handled
identity and [secure handshake mechanisms](https://github.com/libp2p/specs/blob/master/tls/tls.md).
This means peer session establishment, encryption negotiation, and remote identification are handled
by the lib-p2p transport stack.

CometBFT channel traffic is mapped to lib-p2p protocol handlers:

- Each CometBFT channel is exposed under a lib-p2p `protocol.ID` namespace (e.g., `/p2p/cometbft/1.0.0/...`).
- Each CometBFT channel is exposed under a lib-p2p `protocol.ID` namespace (for example `/p2p/cometbft/1.0.0/...`).
- Messages are exchanged over lib-p2p streams bound to those protocol handlers.
- Inbound handling is concurrent, with a priority FIFO queue and worker pool to process
reactor traffic in parallel under load.

The worker pool is autoscaled by `autopool`:

- It tracks per-message processing durations and computes decisions from throughput EWMA,
queue pressure, and latency percentile.
queue pressure, and latency percentile (P90).
- High throughput growth or queue pressure scales workers up; high P90 latency above the
configured threshold triggers shrink to avoid overload.
- Default limits are 4-32 workers per reactor; mempool uses a wider range (8-512) to
absorb bursty transaction traffic.
- Priority ordering is preserved before dispatch (`Receive()` pushes by priority, then workers consume in parallel).

## Combined Mode

`combined_mode` runs blocksync and consensus together, instead of treating them
as separate phases. This improves catch-up liveness because a slower node can keep ingesting
verified blocks while consensus is already active.

If enabled, blocksync continuously fetches candidate blocks from peers and
passes them into consensus through an ingestion path:

1. Blocksync takes two consecutive blocks from its pool (fetched from other peers).
2. It builds an ingest candidate for the first block, using commit data from
the next block.
3. It verifies the candidate against the latest state (light client verification).
4. Then it calls consensus, which applies the block as the next height.
## Comparison and Limitations

Important behavior in practice:
<Warning>
The current release does **not** include peer exchange (PEX).
Nodes must use explicit bootstrap peers/static topology; PEX will be added in a future release.
</Warning>

- Blocksync loads the latest state before verification, and both reactors effectively track the same chain progression.
- If consensus already included a block concurrently, blocksync skips it.
- Ingestion enforces strict invariants and keeps the state machine consistent.


# Comparison and Limitations

> **Important:** The current release does **NOT** include peer exchange (PEX).
> Nodes must use explicit bootstrap peers/static topology; automatic peer exchange will be added in the next release.

| Area | `comet-p2p` | `lib-p2p` |
| Area | comet-p2p | lib-p2p |
| -------------------- | ---------------------------------- | ------------------------------------- |
| Transport | TCP | QUIC |
| Peer identity | Comet peer IDs (`<hex>@host:port`) | lib-p2p peer IDs (`12D3Koo...`) |
| Connection handshake | Comet secret connection | lib-p2p identity and secure handshake |
| Peer exchange | PEX + address book flow | **No PEX in this release** |

Because peer identification formats differ, you cannot run a mixed `comet-p2p` / `lib-p2p` network.

## Metrics

Note, because peer identification formats differ, you cannot run a mixed `comet-p2p` / `lib-p2p` network.
Key metrics for `lib-p2p` queue-scaler and resource tuning:

```text
cometbft_p2p_message_reactor_queue_concurrency
cometbft_p2p_messages_reactor_in_flight
cometbft_p2p_messages_reactor_pending_duration
cometbft_p2p_message_reactor_receive_duration
cometbft_p2p_messages_received
cometbft_p2p_peer_receive_bytes_total
cometbft_p2p_peer_send_bytes_total
cometbft_p2p_message_receive_bytes_total
cometbft_p2p_message_send_bytes_total
cometbft_p2p_peer_send_queue_size
cometbft_p2p_peers
```
3 changes: 2 additions & 1 deletion docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -5210,7 +5210,8 @@
{
"group": "Experimental",
"pages": [
"cometbft/next/docs/experimental/krakatoa-mempool"
"cometbft/next/docs/experimental/krakatoa-mempool",
"cometbft/next/docs/experimental/lib-p2p"
]
},
{
Expand Down