feat: replace LevelDB with DuckDB as persistence backend#20402
feat: replace LevelDB with DuckDB as persistence backend#20402
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
d183c0c to
edb4b2d
Compare
|
Closes APP-67 |
|
This PR has conflicts. You need to rebase the PR before it can be merged. |
|
This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed. |
|
This PR has conflicts. You need to rebase the PR before it can be merged. |
e8cb8c1 to
f479de1
Compare
f479de1 to
3e357d9
Compare
|
This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed. |
erri120
left a comment
There was a problem hiding this comment.
I pushed some small fixes to keep the PR feedback loop small on this. Only issue is the generate-query-types script which I'm not sure when or how we want to use it and also it has import failures.
|
This PR has conflicts. You need to rebase the PR before it can be merged. |
Replace the LevelDB persistence layer (levelup/encoding-down) with DuckDB's level_pivot extension via @duckdb/node-api. The level_pivot raw mode reads existing LevelDB databases directly with no migration needed. - Add @duckdb/node-api dependency (1.4.4-r.1) - Rewrite LevelPersist.ts internals: DuckDB in-memory instance with LevelDB ATTACHed via level_pivot, SQL operations instead of streams - Remove levelup, encoding-down, @types/levelup, @types/encoding-down - Keep leveldown for repairDB, keep same class/export shape and IPersistor interface
3bc8bab to
a4a0185
Compare
|
This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed. |
|
This PR has conflicts. You need to rebase the PR before it can be merged. |
|
Looks really cool! |
…script - Remove #restackingFunc from LevelPersist, convert wrapped methods to plain async - Switch level_pivot install to community repo in DuckDBSingleton and script - Fix broken import/output paths in generate-query-types script - Use createRequire to resolve @duckdb/node-api from @vortex/main workspace - Fix temp file cleanup ordering (detach before close, cleanup after shutdown) - Add pnpm generate:query-types alias - Generate initial queryTypes.ts
Resolve conflict in ipc.ts: keep persist:push and app:init channels
|
This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed. |
|
I'm getting this error when launching Vortex: Stack shows |
- Change pivot row types from interface to type alias so they satisfy the Record<string, unknown> constraint on Table<T> (fixes build) - Replace INSERT ... ON CONFLICT upsert with transactional UPDATE-first, INSERT-if-missing pattern since level_pivot tables don't support UNIQUE indexes (fixes runtime persistence crashes) - Track transaction state so setItem wraps in its own transaction when called standalone but skips when already inside one
What this PR does
Vortex is gradually moving business logic out of the renderer process (React/Redux) and into the main process. This PR replaces the LevelDB persistence backend with DuckDB (via the `level_pivot` extension), and adds the infrastructure for main-process code to write state and sync it back to the renderer.
How it works
Persistence backend
DuckDB attaches a LevelDB file as a typed relational database via the `level_pivot` extension. State is stored as `path -> JSON value` pairs, so the renderer's Redux store and hydration flow are unchanged. A single `DuckDBSingleton` owns the DuckDB instance and connection pool; each `LevelPersist` database attaches to it under a unique alias so all hives share one engine.
Bidirectional sync
State previously only flowed renderer -> Redux -> `persist:diff` IPC -> main -> LevelDB. Main-process code can now also initiate writes:
SQL query files and the parser
Named SQL queries live in `src/queries/` as plain `.sql` files annotated with a comment-based DSL:
The `@type` tag at the top of each file controls how the query is handled. `setup` queries create pivot tables at startup; `view` queries create SQL views; `select` queries are the named, parameterised queries called by name at runtime.
`queryParser.ts` scans the `src/queries/` tree at runtime, parses each file extracting annotations and SQL body, validates name uniqueness across all files, and returns a list of `ParsedQuery` objects.
Code generation
`scripts/generate-query-types.ts` is a build-time script that:
SQL schema changes propagate to TypeScript automatically with a single script run.
Live query updates
After each write transaction commits, `LevelPersist` reads `level_pivot_dirty_tables()` to find which tables were modified. Those table names are forwarded to `QueryInvalidator`, which debounces rapid writes (~16 ms) and batches them before acting.
`QueryRegistry` maintains a reverse index built at startup by calling `getTableNames(sql)` on each registered query, producing a map of `tableName -> Set`. `QueryInvalidator` uses this index to resolve dirty tables into affected query names.
`QueryWatcher` holds a set of active subscriptions. When `QueryInvalidator` flushes, it calls `QueryWatcher.onQueriesInvalidated(affectedQueries)`. For each matching subscription the watcher re-executes the query, JSON-compares the result to the previous snapshot, and fires the callback only if something changed, delivering `{ previous, current }` diffs to subscribers.
This is all main-process only. The renderer continues using Redux; live query results feed back to Redux via `pushStateToRenderer` when main-process code decides to act on them.
Why DuckDB
As we move code to the main thread, we want main-process logic to query and write application state using real SQL. DuckDB gives us joins, aggregations, window functions, and typed schemas on top of the same LevelDB files already on disk.