DurableEmitter (PoC) Integration Testing

[DylanTinianov]

DurableEmitter

Integrates DurableEmitter (PoC) with local CRE environment. Creates ORM for durable event storage, and verifies the emitter persists events and deletes them once accepted by Chip server.

Also adds load testing on DurableEmitter to scale TPS and payload sizes.

DurableEmitter (chainlink-common)

Metrics & Grafana Dashboard

Here are metrics from running a single Cron workflow emitting an event every second.

Load Testing

Can use a mock Chip Server or connect to a local Chip server by passing an environment variable.
Endpoint: Local Chip Server
DB: Local Postgres

Ramp Up

Results with 1 minute runtime:

Q max represents the max number of persisted DB entries (queued up events)

Publish mean 57.293 ms
Takeaway: Chip Publish seems to take ~50ms which is the bottleneck

Sustained Throughput

Sends 100000 events with 10 concurrent emitters.

Results:
--- Emit Phase ---
Events emitted: 100000
Emit errors: 0
Elapsed: 37.314s
Emit rate: 2680 events/sec
--- Delivery Phase ---
Total elapsed: 37.538s
End-to-end rate: 2664 events/sec

Sustained 1k TPS

Currently 1k TPS is not quite sustainable with current configurations. This could be due to multiple factors.

Payload Size Scaling

We can see here that we're not currently bound by payload size.

Local CRE Testing

When running locally I'm using workflow-gateway-capabilities-don.toml

CRE_V2_DurableEmitterTest

Verifies the DurableEmitter is active and functioning by deploying a cron workflow that emits events, then checking that chip_durable_events sees sustained insert+delete activity over time.

Local test output:
2:27PM INF Waiting for sustained durable event activity... v2_durable_emitter_test.go:122: chip_durable_events: +0 inserts, +0 deletes, 5 pending v2_durable_emitter_test.go:122: chip_durable_events: +5 inserts, +13 deletes, 5 pending v2_durable_emitter_test.go:122: chip_durable_events: +6 inserts, +13 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +6 inserts, +13 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +7 inserts, +14 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +26 inserts, +33 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +26 inserts, +33 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +27 inserts, +34 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +27 inserts, +34 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +27 inserts, +34 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +28 inserts, +35 deletes, 6 pending v2_durable_emitter_test.go:122: chip_durable_events: +50 inserts, +53 deletes, 6 pending v2_durable_emitter_test.go:129: pending durable events at end of test: 6 v2_durable_emitter_test.go:135: final chip_durable_events stats: inserts=58 (+50) deletes=56 (+53) 2:29PM INF Durable emitter test completed successfully

We can see that DurableEmitter is successfully persisting events, sending them to Chip and then deleting the events from persistence on the Local CRE environment.

Test_CRE_V2_DurableEmitter_Load

Deploys 20 workflows with 1s (minimum possible) cron jobs to stress the durable emitter pipeline. This measures the maximum sustained throughput of the persist → publish → delete cycle against real Postgres.