Skip to content

telemetry: consolidate clickhouse schema migrations#3460

Open
snormore wants to merge 7 commits intomainfrom
snor/telemetry-migrations
Open

telemetry: consolidate clickhouse schema migrations#3460
snormore wants to merge 7 commits intomainfrom
snor/telemetry-migrations

Conversation

@snormore
Copy link
Copy Markdown
Contributor

@snormore snormore commented Apr 6, 2026

Summary of Changes

  • Introduce telemetry/migrations package: a shared goose migration runner (embedded SQL, slog adapter, RunMigrations func) used by all telemetry services that write to ClickHouse
  • Consolidate flow-enricher and gnmi-writer ClickHouse schemas into versioned goose migrations: flows (with enrichment columns), device_ifindex, and all 7 gnmi-writer tables
  • Add CLICKHOUSE_RUN_MIGRATIONS env var / --clickhouse-run-migrations flag to flow-enricher and gnmi-writer for on-startup migration execution
  • Update both integration tests to apply schemas via RunMigrations — tests now run against the same migrations as production
  • Add setup.sql for one-time admin setup of telemetry_mainnet_beta, telemetry_testnet, and telemetry_devnet databases with writer users
  • Remove per-service clickhouse/ schema directories and the flow_schema_migrations.yml CI workflow (migration CI moves to the infra repo)

Notes

flows table engine: The migration specifies ENGINE = MergeTree() rather than SharedMergeTree. ClickHouse Cloud automatically promotes MergeTree (and its variants) to the equivalent SharedMergeTree engine transparently — specifying SharedMergeTree explicitly is unnecessary and breaks single-node deployments such as the test containers.

Diff Breakdown

Category Files Lines (+/-) Net
Core logic 2 +352 / -1 +351
Scaffolding 12 +58 / -222 -164
Tests 2 +36 / -18 +18
Fixtures 2 +9 / -66 -57
Config/build 2 +25 / -85 -60
Generated 1 +53 / -0 +53
Total 21 +526 / -392 +134

Most net-new lines are goose migration SQL; the rest is deletion of old scattered schema files.

Key files (click to expand)
  • telemetry/migrations/20250303200213_gnmi_init.sql — new goose migration containing all 7 gnmi-writer tables and their _latest views
  • telemetry/migrations/migrations.goRunMigrations func: opens a ClickHouse database/sql connection, sets the embedded FS, runs goose up; slog adapter for goose logging
  • telemetry/migrations/20250303200212_flows_init.sql — moved from flow-enricher; adds enrichment columns (src_device_code, dst_device_code, src_location, dst_location, src_exchange, dst_exchange)
  • telemetry/migrations/20250303200214_device_ifindex_init.sql — new migration for the device_ifindex lookup table read by flow-enricher
  • telemetry/gnmi-writer/cmd/gnmi-writer/main.go — adds ClickhouseRunMigrations to Config, wired from CLICKHOUSE_RUN_MIGRATIONS / --clickhouse-run-migrations
  • telemetry/flow-enricher/cmd/flow-enricher/main.go — same CLICKHOUSE_RUN_MIGRATIONS wiring in the non-stdout output path
  • telemetry/migrations/setup.sql — one-time manual setup: creates three databases and per-database writer users

Testing Verification

  • go build and go vet pass on all affected packages
  • gnmi-writer integration test updated to apply schemas via RunMigrations against a testcontainer
  • flow-enricher integration test updated to apply schemas via RunMigrations and seed device_ifindex from the insert fixture; test now writes to flows (production table name) instead of flows_integration

@snormore snormore force-pushed the snor/telemetry-migrations branch from 3efe6c5 to 6f847b5 Compare April 6, 2026 21:26
@snormore snormore marked this pull request as ready for review April 6, 2026 21:41
@snormore snormore requested a review from packethog April 6, 2026 21:41
@snormore snormore force-pushed the snor/telemetry-migrations branch from 6f847b5 to c46e3c7 Compare April 6, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants