Skip to content

freshness metrics#1695

Draft
leoyvens wants to merge 3 commits intomainfrom
leo/add-timestamp
Draft

freshness metrics#1695
leoyvens wants to merge 3 commits intomainfrom
leo/add-timestamp

Conversation

@leoyvens
Copy link
Collaborator

@leoyvens leoyvens commented Feb 4, 2026

This adds freshness tracking for dump jobs via two new metrics:

  • latest_segment_timestamp: Unix timestamp of the end block in the most recently
    written segment
  • latest_segment_delay_seconds: Difference between latest_segment_timestamp and wall
    clock time at write

For derived datasets, freshness is recorded inline when a segment is written, since writes are sequential.

For raw datasets it's more involved: multiple writers dump disjoint block ranges in parallel, so recording freshness at write time would give inconsistent values depending on which writer finishes last. Instead, a background task subscribes to table change notifications and reports freshness based on synced_range.

Commits:

  1. Add timestamp field to BlockRange, propagated from block data through segment
    metadata.
  2. Add the two freshness metrics and record them for derived datasets (inline at write
    time).
  3. Add background freshness tracking for raw datasets via notification subscription.

@leoyvens leoyvens requested a review from Theodus February 4, 2026 18:25
@leoyvens leoyvens marked this pull request as draft February 4, 2026 18:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant