telemetry: consolidate clickhouse schema migrations#3460
Open
telemetry: consolidate clickhouse schema migrations#3460
Conversation
3efe6c5 to
6f847b5
Compare
packethog
approved these changes
Apr 6, 2026
Move flow-enricher and gnmi-writer ClickHouse schemas into a shared telemetry/migrations package using goose. Add CLICKHOUSE_RUN_MIGRATIONS support to both services for on-startup migration execution.
…ation, cover in tests
…promotes to SharedMergeTree)
… avoid TTL expiry
6f847b5 to
c46e3c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
telemetry/migrationspackage: a shared goose migration runner (embedded SQL, slog adapter,RunMigrationsfunc) used by all telemetry services that write to ClickHouseflows(with enrichment columns),device_ifindex, and all 7 gnmi-writer tablesCLICKHOUSE_RUN_MIGRATIONSenv var /--clickhouse-run-migrationsflag to flow-enricher and gnmi-writer for on-startup migration executionRunMigrations— tests now run against the same migrations as productionsetup.sqlfor one-time admin setup oftelemetry_mainnet_beta,telemetry_testnet, andtelemetry_devnetdatabases with writer usersclickhouse/schema directories and theflow_schema_migrations.ymlCI workflow (migration CI moves to the infra repo)Notes
flowstable engine: The migration specifiesENGINE = MergeTree()rather thanSharedMergeTree. ClickHouse Cloud automatically promotesMergeTree(and its variants) to the equivalentSharedMergeTreeengine transparently — specifyingSharedMergeTreeexplicitly is unnecessary and breaks single-node deployments such as the test containers.Diff Breakdown
Most net-new lines are goose migration SQL; the rest is deletion of old scattered schema files.
Key files (click to expand)
telemetry/migrations/20250303200213_gnmi_init.sql— new goose migration containing all 7 gnmi-writer tables and their_latestviewstelemetry/migrations/migrations.go—RunMigrationsfunc: opens a ClickHousedatabase/sqlconnection, sets the embedded FS, runsgoose up; slog adapter for goose loggingtelemetry/migrations/20250303200212_flows_init.sql— moved from flow-enricher; adds enrichment columns (src_device_code,dst_device_code,src_location,dst_location,src_exchange,dst_exchange)telemetry/migrations/20250303200214_device_ifindex_init.sql— new migration for thedevice_ifindexlookup table read by flow-enrichertelemetry/gnmi-writer/cmd/gnmi-writer/main.go— addsClickhouseRunMigrationsto Config, wired fromCLICKHOUSE_RUN_MIGRATIONS/--clickhouse-run-migrationstelemetry/flow-enricher/cmd/flow-enricher/main.go— sameCLICKHOUSE_RUN_MIGRATIONSwiring in the non-stdout output pathtelemetry/migrations/setup.sql— one-time manual setup: creates three databases and per-database writer usersTesting Verification
go buildandgo vetpass on all affected packagesRunMigrationsagainst a testcontainerRunMigrationsand seeddevice_ifindexfrom the insert fixture; test now writes toflows(production table name) instead offlows_integration