-
Notifications
You must be signed in to change notification settings - Fork 44
Open
Description
Context
Many existing extractors already have access to relationship data beyond lineage but don't surface it. Once the typed relationship model is in place, extractors should emit everything they can.
Scope
Audit and update existing extractors to capture available relationships:
- BigQuery — table read/write patterns from access logs, dataset ownership
- Tableau — dashboard-to-datasource relationships, workbook ownership
- Optimus — job ownership, job-to-table read/write patterns
- Kafka — topic ownership, consumer group relationships
- Postgres/MySQL — view dependencies, role-based access patterns
- Grafana/Metabase/Superset — dashboard ownership, datasource connections
Approach
For each extractor:
- Identify what relationship data is available at the source
- Map source relationships to typed edges from the new relationship model
- Emit relationships as raw observations alongside existing asset data — Compass handles entity resolution
- Update tests and documentation
Why
The richest graph comes from extracting relationships that already exist at the source. This is low-hanging fruit — the data is there, we just aren't capturing it.
Dependencies
- Typed relationship model (Expand relationship model with typed edges beyond lineage #499)
- Proto schema changes in
raystack/proton
References
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels