Evolve Compass sink protocol for rich graph delivery #501
Description
Context
The Compass sink currently pushes individual asset records via HTTP PATCH with flat payloads. As Meteor's extraction becomes richer, the delivery protocol to Compass needs to evolve.
Scope
Upgrade the Compass sink to support:
- Source-annotated observations — Each observation carries its source identity, extraction timestamp, and enough context for Compass to perform entity resolution against its existing graph
- Typed relationships — Deliver the full set of relationships discovered during extraction, not just upstream/downstream lineage
- Cross-source hints — Annotations with shared identifiers, matching URNs, and compatible schemas that help Compass infer cross-source edges during entity resolution
- Change deltas — When change detection is available, push only what changed rather than full snapshots
Boundary
Meteor delivers raw observations. Entity resolution, deduplication, and graph construction happen in Compass. The sink protocol should carry enough metadata (source, timestamps, identifiers) for Compass to make resolution decisions, but Meteor does not need to know what's already in the graph.
Approach
- Design the enriched payload format in coordination with Compass ingestion API changes
- Maintain backward compatibility — the sink should work with both old and new Compass versions
- Support batched delivery of related observations and their relationships in a single request
Why
The Compass sink is the primary delivery mechanism. The sophistication should be in what Meteor delivers, not in how many places it delivers to. A richer protocol means a richer graph in Compass.
Dependencies
- Typed relationship model in Meteor
- Corresponding ingestion API changes in Compass