feat: entity system and temporal graph with Postgres-native search by ravisuhag · Pull Request #249 · raystack/compass

ravisuhag · 2026-03-29T07:13:21Z

Summary

Complete rewrite of Compass from a static asset catalog to a temporal entity graph with Postgres-native search. No Elasticsearch dependency.

Companion PR: raystack/proton#461

What's New

Core Domain (`core/entity/`)

Entity: open type system, temporal (valid_from/valid_to), properties JSONB
Edge: typed, directed, temporal relationships between entities
Chunk: pgvector embeddings for semantic search
HybridSearch: keyword + semantic fusion via Reciprocal Rank Fusion
Service: context subgraph assembly, downstream impact analysis

Storage (`store/postgres/`)

EntityRepository: CRUD with temporal support
EdgeRepository: recursive CTE graph traversal (upstream/downstream)
ChunkRepository: pgvector cosine similarity search
EntitySearchRepository: tsvector full-text + pg_trgm fuzzy matching

Search (all Postgres, no ES)

Mode	Engine	Purpose
keyword	tsvector + pg_trgm	Exact matches, fuzzy/typo tolerance
semantic	pgvector cosine	Conceptual/natural language
hybrid	Both + RRF fusion	Best of both

API (12 new RPCs)

Entity: GetAll, GetByID, Upsert, Delete, Search, Suggest, GetTypes
Context: GetEntityContext, GetEntityImpact
Edge: Upsert, Get, Delete

MCP Tools

search_entities: hybrid search across the entity graph
get_context: assembled context subgraph (entity + edges + related)
impact: downstream blast radius analysis

CLI

compass entity list/view/upsert/delete/search/types/context/impact

What's Removed

core/asset/ — entire old asset package
core/discussion/, core/tag/ — discussions, comments, tags, tag templates
store/elasticsearch/ — ES client and all discovery repos
All old handlers, CLI commands, migrations
Elasticsearch from docker-compose, CI, config

Schema (6 tables)

namespaces → users → entities → edges → chunks → stars

Single fresh migration: 000001_init_schema.up.sql

Test Plan

go build ./... passes
go vet ./... passes
11 unit tests for entity domain (type validation, RRF fusion, service CRUD)
Existing namespace/user/validator tests pass
Integration tests with real Postgres (follow-up)

New entity system built as an independent package alongside the existing asset system. Zero changes to existing asset code. All search is Postgres-native — no Elasticsearch dependency for v2. Core domain (core/entity/): - Entity: open type system, temporal (valid_from/valid_to), properties JSONB - Edge: typed, directed, temporal relationships - Chunk: pgvector embeddings for semantic search - Service: context assembly, impact analysis, search orchestration - HybridSearch: keyword + semantic fusion via Reciprocal Rank Fusion - SearchRepository: Postgres tsvector + pg_trgm (replaces ES) Storage (store/postgres/): - EntityRepository: CRUD with temporal support - EdgeRepository: recursive CTE graph traversal - ChunkRepository: pgvector cosine similarity search - EntitySearchRepository: tsvector full-text + pg_trgm fuzzy matching - Migration 000019: entities, edges, chunks tables with search_vector generated column, GIN indexes, trigram indexes Search architecture (all Postgres): - Keyword: tsvector with weighted fields (URN/name=A, desc=B, source=C) - Fuzzy: pg_trgm similarity with automatic fallback - Semantic: pgvector cosine distance on chunk embeddings - Hybrid: RRF fusion of keyword + semantic results - Suggest: pg_trgm similarity-based autocomplete MCP tools (internal/mcp/): - search_entities: hybrid search across the entity graph - get_context: assembled context subgraph - impact: downstream blast radius analysis

Generated Go code from proton proto definitions for entity v2 RPCs. Created handler/entity.go implementing all 12 new RPC methods: Entity CRUD: GetAllEntities, GetEntityByID, UpsertEntity, DeleteEntity Search: SearchEntities, SuggestEntities, GetEntityTypes Context: GetEntityContext, GetEntityImpact Edges: UpsertEdge, GetEdges, DeleteEdge Handler accepts entity and edge services via WithEntityService/ WithEdgeService options — no changes to existing handler constructor signature. Serve function passes options through to handler.

Deleted: - core/asset/ — entire old asset package - core/discussion/ — discussions and comments - core/tag/ — tags and tag templates - store/elasticsearch/ — ES client and discovery repos - store/postgres/asset_*, lineage_*, discussion_*, comment_*, tag_* repos - handler/asset.go, discussion.go, comment.go, tag.go, search.go, lineage.go - All old CLI commands (assets, discussions, lineage, search) - All old migrations (replaced with single 000001_init_schema.up.sql) Updated: - Proto regenerated from proton (entity-only service definition) - handler.go: simplified — only namespace, star, user, entity, edge services - bootstrap.go: removed ES, removed all old service wiring - config.go: removed Elasticsearch config - star package: updated to use entity.Entity instead of asset.Asset - CLI: entity commands (list, view, upsert, delete, search, types, context, impact) - Migration: single fresh schema with namespaces, users, entities, edges, chunks, stars Schema (6 tables): - namespaces, users, entities, edges, chunks, stars - tsvector + pg_trgm + pgvector for search (no ES) - RLS on all tables

- go mod tidy: removed unused ES and other stale dependencies - Namespace service: nil-safe DiscoveryRepository (ES removed) - Deleted stale integration tests (postgres, user, namespace repo tests that depended on deleted test infrastructure) - Fixed last 2 "asset" string references in star errors and CLI - Added entity unit tests (11 tests): - Type validation (open type system) - Entity.IsCurrent temporal check - Reciprocal Rank Fusion (empty, single, multi-list) - Service: Upsert, GetByURN, Delete, GetAll, GetTypes, Search

- Removed elasticsearch from docker-compose.yaml (Postgres-only) - Removed elasticsearch from config.yaml and config.example.yaml - Removed elasticsearch service from .github/workflows/test.yml - Deleted stale guide docs (ingestion, querying, starring, telemetry) - Deleted stale handler mocks (no tests import them) - Removed empty test/ directory - Updated Postgres to v16 in docker-compose - go mod tidy (cleaned unused indirect deps)

vercel · 2026-03-29T07:13:26Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
compass	Ready	Preview, Comment	Mar 29, 2026 7:12pm

coderabbitai · 2026-03-29T07:13:28Z

Important

Review skipped

Too many files!

This PR contains 216 files, which is 66 over the limit of 150.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0a4fa10a-e7a9-448b-ba39-45d1da01cdcc

📥 Commits

Reviewing files that changed from the base of the PR and between 709bf5b and e4988b2.

⛔ Files ignored due to path filters (3)

gen/raystack/compass/v1beta1/compassv1beta1connect/service.connect.go is excluded by !**/gen/**
gen/raystack/compass/v1beta1/service.pb.go is excluded by !**/*.pb.go, !**/gen/**
go.sum is excluded by !**/*.sum

📒 Files selected for processing (216)

.github/workflows/test.yml
.gitignore
Makefile
cli/assets.go
cli/discussions.go
cli/entities.go
cli/lineage.go
cli/root.go
cli/search.go
cli/utils.go
core/asset/asset.go
core/asset/asset_test.go
core/asset/discovery.go
core/asset/discovery_test.go
core/asset/errors.go
core/asset/filter.go
core/asset/filter_test.go
core/asset/lineage.go
core/asset/mocks/asset_repository.go
core/asset/mocks/discovery_repository.go
core/asset/mocks/lineage_repository.go
core/asset/patch.go
core/asset/probe.go
core/asset/probes_filter.go
core/asset/service.go
core/asset/service_test.go
core/asset/type.go
core/asset/type_test.go
core/asset/unique_strings.go
core/asset/version.go
core/asset/version_test.go
core/discussion/comment.go
core/discussion/comment_test.go
core/discussion/discussion.go
core/discussion/discussion_test.go
core/discussion/errors.go
core/discussion/filter.go
core/discussion/filter_test.go
core/discussion/mocks/discussion_repository.go
core/discussion/service.go
core/discussion/service_test.go
core/discussion/state.go
core/discussion/state_test.go
core/discussion/type_test.go
core/discussion/types.go
core/entity/chunk.go
core/entity/edge.go
core/entity/entity.go
core/entity/entity_test.go
core/entity/search.go
core/entity/search_test.go
core/entity/service.go
core/entity/service_test.go
core/namespace/service.go
core/star/errors.go
core/star/mocks/star_repository.go
core/star/service.go
core/star/star.go
core/tag/errors.go
core/tag/mocks/tag_repository.go
core/tag/mocks/tag_template_repository.go
core/tag/service.go
core/tag/service_test.go
core/tag/tag.go
core/tag/tag_template.go
core/tag/tag_template_service.go
core/tag/tag_template_service_test.go
core/tag/utils.go
core/tag/validator.go
core/tag/validator/builder.go
core/tag/validator/error.go
core/tag/validator/validator.go
docker-compose.yaml
docs/docs/concepts/asset.mdx
docs/docs/concepts/overview.mdx
docs/docs/concepts/type.md
docs/docs/concepts/user.md
docs/docs/configuration.md
docs/docs/guides/discussion.md
docs/docs/guides/ingestion.md
docs/docs/guides/querying.md
docs/docs/guides/starring.md
docs/docs/guides/tagging.md
docs/docs/guides/telemetry.md
docs/docs/introduction.md
docs/docs/reference/configuration.md
docs/docs/tour/1-my-first-asset.md
docs/docs/tour/2-querying-assets.mdx
docs/docs/tour/3-asset-lineage.mdx
docs/docs/tour/introduction.md
docs/sidebars.js
go.mod
handler/asset.go
handler/asset_test.go
handler/comment.go
handler/comment_test.go
handler/discussion.go
handler/discussion_test.go
handler/entity.go
handler/handler.go
handler/lineage.go
handler/lineage_test.go
handler/mocks/asset_service.go
handler/mocks/discussion_service.go
handler/mocks/namespace_service.go
handler/mocks/star_service.go
handler/mocks/tag_service.go
handler/mocks/tag_template_service.go
handler/mocks/user_service.go
handler/namespace_test.go
handler/option.go
handler/search.go
handler/search_test.go
handler/star.go
handler/tag.go
handler/tag_template.go
handler/tag_template_test.go
handler/tag_test.go
handler/type.go
handler/type_test.go
handler/user.go
handler/user_test.go
internal/config/config.example.yaml
internal/config/config.go
internal/mcp/entity_handlers.go
internal/mcp/entity_tools.go
internal/mcp/format.go
internal/mcp/handlers.go
internal/mcp/handlers_test.go
internal/mcp/server.go
internal/mcp/server_test.go
internal/mcp/tools.go
internal/server/bootstrap.go
internal/server/server.go
store/elasticsearch/discovery_repository.go
store/elasticsearch/discovery_repository_test.go
store/elasticsearch/discovery_search_repository.go
store/elasticsearch/discovery_search_repository_test.go
store/elasticsearch/es.go
store/elasticsearch/es_test.go
store/elasticsearch/init_test.go
store/elasticsearch/options.go
store/elasticsearch/schema.go
store/elasticsearch/testdata/search-test-fixture.json
store/elasticsearch/testdata/suggest-test-fixture.json
store/elasticsearch/testutil/elastic_search.go
store/postgres/asset_model.go
store/postgres/asset_model_test.go
store/postgres/asset_repository.go
store/postgres/asset_repository_test.go
store/postgres/chunk_repository.go
store/postgres/comment_model.go
store/postgres/discussion_comment_repository.go
store/postgres/discussion_comment_repository_test.go
store/postgres/discussion_model.go
store/postgres/discussion_repository.go
store/postgres/discussion_repository_test.go
store/postgres/edge_repository.go
store/postgres/entity_repository.go
store/postgres/entity_search_repository.go
store/postgres/jsonmap.go
store/postgres/lineage_model.go
store/postgres/lineage_repository.go
store/postgres/lineage_repository_test.go
store/postgres/migrations/000001_create_tags_templates_fields_table.down.sql
store/postgres/migrations/000001_create_tags_templates_fields_table.up.sql
store/postgres/migrations/000001_init_schema.down.sql
store/postgres/migrations/000001_init_schema.up.sql
store/postgres/migrations/000002_create_users_table.down.sql
store/postgres/migrations/000002_create_users_table.up.sql
store/postgres/migrations/000003_create_assets_table.down.sql
store/postgres/migrations/000003_create_assets_table.up.sql
store/postgres/migrations/000004_create_stars_table.down.sql
store/postgres/migrations/000004_create_stars_table.up.sql
store/postgres/migrations/000005_create_assets_versions_table.down.sql
store/postgres/migrations/000005_create_assets_versions_table.up.sql
store/postgres/migrations/000006_create_lineage_graph_table.down.sql
store/postgres/migrations/000006_create_lineage_graph_table.up.sql
store/postgres/migrations/000007_create_discussions_table.down.sql
store/postgres/migrations/000007_create_discussions_table.up.sql
store/postgres/migrations/000008_create_comments_table.down.sql
store/postgres/migrations/000008_create_comments_table.up.sql
store/postgres/migrations/000009_update_tags_templates_assetid.down.sql
store/postgres/migrations/000009_update_tags_templates_assetid.up.sql
store/postgres/migrations/000010_drop_assets_idx_urn_type_service.down.sql
store/postgres/migrations/000010_drop_assets_idx_urn_type_service.up.sql
store/postgres/migrations/000011_create_idx_assets_urn.down.sql
store/postgres/migrations/000011_create_idx_assets_urn.up.sql
store/postgres/migrations/000012_create_asset_probes_table.down.sql
store/postgres/migrations/000012_create_asset_probes_table.up.sql
store/postgres/migrations/000013_alter_assets_table.down.sql
store/postgres/migrations/000013_alter_assets_table.up.sql
store/postgres/migrations/000014_create_namespace_table_and_add_namespace_in_tables.down.sql
store/postgres/migrations/000014_create_namespace_table_and_add_namespace_in_tables.up.sql
store/postgres/migrations/000015_enable_row_level_security_all_tables.down.sql
store/postgres/migrations/000015_enable_row_level_security_all_tables.up.sql
store/postgres/migrations/000016_update_assets_versions.down.sql
store/postgres/migrations/000016_update_assets_versions.up.sql
store/postgres/migrations/000017_add_soft_deletion.down.sql
store/postgres/migrations/000017_add_soft_deletion.up.sql
store/postgres/migrations/000018_index_asset_probes.down.sql
store/postgres/migrations/000018_index_asset_probes.up.sql
store/postgres/namespace_repository_test.go
store/postgres/postgres_test.go
store/postgres/star_repository.go
store/postgres/star_repository_test.go
store/postgres/tag_model.go
store/postgres/tag_model_test.go
store/postgres/tag_repository.go
store/postgres/tag_repository_test.go
store/postgres/tag_template_repository.go
store/postgres/tag_template_repository_test.go
store/postgres/user_model_test.go
store/postgres/user_repository_test.go
test/asset_test.go
test/helper_test.go

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

ravisuhag added 5 commits March 29, 2026 01:00

build: update PROTON_COMMIT to eefb04d (merged entity proto)

7a08437

vercel bot had a problem deploying to Preview March 29, 2026 07:16 Failure

ravisuhag changed the title ~~feat: v2 entity system — temporal graph with Postgres-native search~~ feat: entity system — temporal graph with Postgres-native search Mar 29, 2026

ravisuhag changed the title ~~feat: entity system — temporal graph with Postgres-native search~~ feat: entity system and temporal graph with Postgres-native search Mar 29, 2026

ravisuhag force-pushed the feat/entity-system branch from 17c6f0e to 7a08437 Compare March 29, 2026 07:23

vercel bot had a problem deploying to Preview March 29, 2026 07:23 Failure

chore: gitignore docs/_drafts

677ede5

vercel bot had a problem deploying to Preview March 29, 2026 07:24 Failure

fix: resolve all lint issues

5e532b0

vercel bot had a problem deploying to Preview March 29, 2026 13:34 Failure

fix: remove stale docs and fix sidebars.js

5366c97

vercel bot had a problem deploying to Preview March 29, 2026 19:07 Failure

fix: resolve broken doc links and update concepts overview

e4988b2

vercel bot deployed to Preview March 29, 2026 19:12 View deployment

ravisuhag merged commit 8c34130 into main Mar 29, 2026
5 checks passed

ravisuhag deleted the feat/entity-system branch March 29, 2026 19:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: entity system and temporal graph with Postgres-native search#249

feat: entity system and temporal graph with Postgres-native search#249
ravisuhag merged 10 commits intomainfrom
feat/entity-system

ravisuhag commented Mar 29, 2026

Uh oh!

vercel bot commented Mar 29, 2026 •

edited

Loading

Uh oh!

coderabbitai bot commented Mar 29, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ravisuhag commented Mar 29, 2026

Summary

What's New

Core Domain (core/entity/)

Storage (store/postgres/)

Search (all Postgres, no ES)

API (12 new RPCs)

MCP Tools

CLI

What's Removed

Schema (6 tables)

Test Plan

Uh oh!

vercel bot commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai bot commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Core Domain (`core/entity/`)

Storage (`store/postgres/`)

vercel bot commented Mar 29, 2026 •

edited

Loading

coderabbitai bot commented Mar 29, 2026 •

edited

Loading