Skip to content

Update existing extractors to emit richer relationships #500

@ravisuhag

Description

@ravisuhag

Context

Many existing extractors already have access to relationship data beyond lineage but don't surface it. Once the typed relationship model is in place, extractors should emit everything they can.

Scope

Audit and update existing extractors to capture available relationships:

  • BigQuery — table read/write patterns from access logs, dataset ownership
  • Tableau — dashboard-to-datasource relationships, workbook ownership
  • Optimus — job ownership, job-to-table read/write patterns
  • Kafka — topic ownership, consumer group relationships
  • Postgres/MySQL — view dependencies, role-based access patterns
  • Grafana/Metabase/Superset — dashboard ownership, datasource connections

Approach

For each extractor:

  1. Identify what relationship data is available at the source
  2. Map source relationships to typed edges from the new relationship model
  3. Emit relationships as raw observations alongside existing asset data — Compass handles entity resolution
  4. Update tests and documentation

Why

The richest graph comes from extracting relationships that already exist at the source. This is low-hanging fruit — the data is there, we just aren't capturing it.

Dependencies

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions