OEV-1150: add latest rpc nonce metric by augustbleeds · Pull Request #402 · smartcontractkit/chainlink-evm

augustbleeds · 2026-03-26T21:20:23Z

Adds a ~~local txm nonce metric and~~ rpc nonce metric (~~pending vs~~ latest)

github-actions · 2026-03-26T21:20:36Z

👋 augustbleeds, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

github-actions · 2026-03-26T21:22:44Z

✅ API Diff Results - `github.com/smartcontractkit/chainlink-evm`

✅ Compatible Changes (1)

`pkg/txm.(*txmMetrics)` (1)

SetRPCNonce — ➕ Added

📄 View full apidiff report

Copilot

Pull request overview

Adds nonce-related instrumentation to the EVM TXM so operators can observe the TXM’s local nonce and the RPC-reported nonce (pending vs latest).

Changes:

Add Prometheus gauges (and Beholder/OTel gauges) for txm_local_nonce and txm_rpc_nonce (with source=pending|latest).
Record RPC nonce at several RPC fetch points and record local nonce whenever the TXM nonce map is updated.
Add unit tests validating the new Prometheus gauges’ behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
pkg/txm/txm.go	Emits local nonce and RPC nonce metrics during TXM nonce initialization, broadcast flow, and backfill.
pkg/txm/metrics.go	Defines/registers new nonce gauges and exposes `SetLocalNonce`/`SetRPCNonce` helpers.
pkg/txm/metrics_test.go	Adds tests for the new nonce gauge setters (Prometheus side).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-26T21:24:06Z

pkg/txm/metrics.go

+	}, []string{"chainID", "address"})
+	promRPCNonce = promauto.NewGaugeVec(prometheus.GaugeOpts{
+		Name: "txm_rpc_nonce",
+		Help: "The latest nonce reported by the RPC node for a given address.",


The help text for txm_rpc_nonce says it is "The latest nonce reported by the RPC node", but the metric is also used for the pending nonce (distinguished by the source label). Update the help text to reflect that it can represent either pending or latest based on source, so dashboards/alerts interpret it correctly.

Suggested change

Help: "The latest nonce reported by the RPC node for a given address.",

Help: "The nonce reported by the RPC node for a given address (either 'pending' or 'latest', as indicated by the 'source' label).",

Copilot · 2026-03-26T21:24:06Z

pkg/txm/txm.go

 			if e != nil {
 				return false, e
 			}
+			t.Metrics.SetRPCNonce(ctx, address, pendingNonce, "pending")


BroadcastTransaction now calls t.Metrics.SetRPCNonce(...) unconditionally. Txm.Metrics is only initialized in Start(), and several unit tests (and any direct use of NewTxm without starting) call BroadcastTransaction without setting Metrics, which will panic with a nil-pointer dereference. Consider guarding this call with if t.Metrics != nil { ... } (consistent with initializeNonce / SetNonce), or ensure Metrics is always non-nil after NewTxm (e.g., a noop metrics implementation).

Suggested change

t.Metrics.SetRPCNonce(ctx, address, pendingNonce, "pending")

if t.Metrics != nil {

t.Metrics.SetRPCNonce(ctx, address, pendingNonce, "pending")

}

Copilot · 2026-03-26T21:24:06Z

pkg/txm/txm.go

 		pendingNonce, pErr := t.client.PendingNonceAt(ctx, fromAddress)
 		if pErr != nil {
 			return pErr
 		}
+		t.Metrics.SetRPCNonce(ctx, fromAddress, pendingNonce, "pending")
 		if pendingNonce <= *tx.Nonce {


sendTransactionWithError now calls t.Metrics.SetRPCNonce(...) without a nil check. Since Metrics is only set in Start() (and can be nil in tests/when Txm is used without starting), this introduces a potential nil-pointer panic on the error path. Add a nil guard here or ensure Metrics is always initialized before this method can run.

Copilot · 2026-03-26T21:24:07Z

pkg/txm/txm.go

@@ -361,6 +369,7 @@
 	if err != nil {
 		return err
 	}
+	t.Metrics.SetRPCNonce(ctx, address, latestNonce, "latest")


BackfillTransactions now records the RPC nonce via t.Metrics.SetRPCNonce(...) unconditionally. Because Metrics is initialized in Start() and is nil on a freshly constructed Txm, calling BackfillTransactions directly (as some tests do) will panic. Guard this call with if t.Metrics != nil { ... } or make Metrics non-nil by construction (e.g., noop metrics).

Copilot · 2026-03-26T21:24:07Z

pkg/txm/metrics.go

+	m.localNonce.Record(ctx, int64(nonce))
+}
+
+func (m *txmMetrics) SetRPCNonce(ctx context.Context, address common.Address, nonce uint64, source string) {
+	promRPCNonce.WithLabelValues(m.chainID.String(), address.String(), source).Set(float64(nonce))
+	m.rpcNonce.Record(ctx, int64(nonce))
+}
+
 func (m *txmMetrics) EmitTxMessage(ctx context.Context, txHash common.Hash, fromAddress common.Address, tx *types.Transaction) error {


SetLocalNonce/SetRPCNonce accept address (and source) and the Prometheus gauges are labeled accordingly, but the Beholder/OTel gauges are recorded without any attributes. In a multi-address TXM this produces a single time series that will effectively flap/overwrite between different addresses (and between pending/latest), making the Beholder metric ambiguous or incorrect. Consider either (a) recording with attributes for chainID, address and source (using the Labeler), or (b) dropping the Beholder gauges entirely if high-cardinality dimensions are intentionally avoided.

Suggested change

m.localNonce.Record(ctx, int64(nonce))

}

func (m *txmMetrics) SetRPCNonce(ctx context.Context, address common.Address, nonce uint64, source string) {

promRPCNonce.WithLabelValues(m.chainID.String(), address.String(), source).Set(float64(nonce))

m.rpcNonce.Record(ctx, int64(nonce))

}

func (m *txmMetrics) EmitTxMessage(ctx context.Context, txHash common.Hash, fromAddress common.Address, tx *types.Transaction) error {

}

func (m *txmMetrics) SetRPCNonce(ctx context.Context, address common.Address, nonce uint64, source string) {

promRPCNonce.WithLabelValues(m.chainID.String(), address.String(), source).Set(float64(nonce))

}

func (m *txmMetrics) EmitTxMessage(ctx context.Context, txHash common.Hash, fromAddress common.Address, tx *types.Transaction) error {

func (m *txmMetrics) EmitTxMessage(ctx context.Context, txHash common.Hash, fromAddress common.Address, tx *types.Transaction) error {

dimriou · 2026-03-27T09:40:10Z

pkg/txm/txm.go

+		if t.Metrics != nil {
+			t.Metrics.SetRPCNonce(ctxWithTimeout, address, pendingNonce, "pending")
+		}
 		t.lggr.Debugf("Set initial nonce for address: %v to %d", address, pendingNonce)


Can you upgrade this log message to info? This should log once and it's quite valuable.

dimriou · 2026-03-27T09:51:41Z

pkg/txm/metrics.go

 		Name: "txm_time_until_tx_confirmed",
 		Help: "The amount of time elapsed from a transaction being broadcast to being included in a block.",
 	}, []string{"chainID"})
+	promLocalNonce = promauto.NewGaugeVec(prometheus.GaugeOpts{


Do we actually need prom metrics?

Just keeping the pattern, but I think it's good practice if NOPs want to see the metrics

dimriou · 2026-03-27T09:55:04Z

pkg/txm/txm.go

IMHO, I don't find much value in tracking the local nonce. It is initialized once and gets incremented sequentially. You can also infer that information by looking at the transactions of the node. Similar case for the pending nonce, which is used once for initialization and then there are clear log errors if you're getting throttled. I would keep the latest nonce value to understand which transactions are being confirmed and which are not.

I see what you're saying, in theory the latest nonce is going to be much more valuable. Yeah, I'll remove it then.

add local/rpc nonce metric

1f33adb

Copilot AI review requested due to automatic review settings March 26, 2026 21:20

augustbleeds requested review from a team and dimriou as code owners March 26, 2026 21:20

product-security-plaid-production bot requested a review from fernandezlautaro March 26, 2026 21:20

Copilot started reviewing on behalf of augustbleeds March 26, 2026 21:20 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

dimriou reviewed Mar 27, 2026

View reviewed changes

remove local nonce metrics and pending rpc nonce metrics

10326f4

augustbleeds changed the title ~~OEV-1150: add local/rpc nonce metric~~ OEV-1150: add latest rpc nonce metric Mar 27, 2026

nil check

2a764a2

	Help: "The latest nonce reported by the RPC node for a given address.",
	Help: "The nonce reported by the RPC node for a given address (either 'pending' or 'latest', as indicated by the 'source' label).",

Conversation

augustbleeds commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 26, 2026

Uh oh!

github-actions bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ API Diff Results - github.com/smartcontractkit/chainlink-evm

✅ Compatible Changes (1)

pkg/txm.(*txmMetrics) (1)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

dimriou Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

dimriou Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

augustbleeds Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

dimriou Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

augustbleeds Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

augustbleeds commented Mar 26, 2026 •

edited

Loading

github-actions bot commented Mar 26, 2026 •

edited

Loading

✅ API Diff Results - `github.com/smartcontractkit/chainlink-evm`

`pkg/txm.(*txmMetrics)` (1)