Skip to content

feat: replace LevelDB with DuckDB as persistence backend#20402

Merged
erri120 merged 43 commits intomasterfrom
duckdb-integration
Mar 25, 2026
Merged

feat: replace LevelDB with DuckDB as persistence backend#20402
erri120 merged 43 commits intomasterfrom
duckdb-integration

Conversation

@halgari
Copy link
Copy Markdown
Contributor

@halgari halgari commented Feb 17, 2026

What this PR does

Vortex is gradually moving business logic out of the renderer process (React/Redux) and into the main process. This PR replaces the LevelDB persistence backend with DuckDB (via the `level_pivot` extension), and adds the infrastructure for main-process code to write state and sync it back to the renderer.

How it works

Persistence backend

DuckDB attaches a LevelDB file as a typed relational database via the `level_pivot` extension. State is stored as `path -> JSON value` pairs, so the renderer's Redux store and hydration flow are unchanged. A single `DuckDBSingleton` owns the DuckDB instance and connection pool; each `LevelPersist` database attaches to it under a unique alias so all hives share one engine.

Bidirectional sync

State previously only flowed renderer -> Redux -> `persist:diff` IPC -> main -> LevelDB. Main-process code can now also initiate writes:

  1. Call `pushStateToRenderer(hive, operations)` with a list of diff operations
  2. Main writes them to DuckDB and waits for the write to flush
  3. Main sends a `persist:push` IPC message to all renderer windows
  4. Renderer applies the operations via a `__persist_push` Redux action
  5. `persistDiffMiddleware` skips `__persist_push` actions to avoid a feedback loop

SQL query files and the parser

Named SQL queries live in `src/queries/` as plain `.sql` files annotated with a comment-based DSL:

-- @type select
-- @name recently_managed_games
-- @description Games the user has managed, sorted by last activation date
-- @param profile_id VARCHAR
SELECT game_id, last_activated FROM ...

The `@type` tag at the top of each file controls how the query is handled. `setup` queries create pivot tables at startup; `view` queries create SQL views; `select` queries are the named, parameterised queries called by name at runtime.

`queryParser.ts` scans the `src/queries/` tree at runtime, parses each file extracting annotations and SQL body, validates name uniqueness across all files, and returns a list of `ParsedQuery` objects.

Code generation

`scripts/generate-query-types.ts` is a build-time script that:

  1. Spins up a temporary DuckDB instance and attaches an empty LevelDB for schema-only work
  2. Runs all `setup` queries, then `DESCRIBE`s them to read column types
  3. Runs all `view` queries, then `PREPARE`s each `select` query to read result column metadata
  4. Maps DuckDB column types to TypeScript types (`VARCHAR -> string`, `INTEGER -> number`, etc.)
  5. Writes `src/main/store/generated/queryTypes.ts` with typed `Params` and `Row` interfaces for every query, a `Models` interface with typed `Table`/`View` accessors, and a `createModels(db)` factory

SQL schema changes propagate to TypeScript automatically with a single script run.

Live query updates

After each write transaction commits, `LevelPersist` reads `level_pivot_dirty_tables()` to find which tables were modified. Those table names are forwarded to `QueryInvalidator`, which debounces rapid writes (~16 ms) and batches them before acting.

`QueryRegistry` maintains a reverse index built at startup by calling `getTableNames(sql)` on each registered query, producing a map of `tableName -> Set`. `QueryInvalidator` uses this index to resolve dirty tables into affected query names.

`QueryWatcher` holds a set of active subscriptions. When `QueryInvalidator` flushes, it calls `QueryWatcher.onQueriesInvalidated(affectedQueries)`. For each matching subscription the watcher re-executes the query, JSON-compares the result to the previous snapshot, and fires the callback only if something changed, delivering `{ previous, current }` diffs to subscribers.

This is all main-process only. The renderer continues using Redux; live query results feed back to Redux via `pushStateToRenderer` when main-process code decides to act on them.

Why DuckDB

As we move code to the main thread, we want main-process logic to query and write application state using real SQL. DuckDB gives us joins, aggregations, window functions, and typed schemas on top of the same LevelDB files already on disk.

@halgari halgari self-assigned this Feb 17, 2026
@halgari halgari added the don't-merge Don't merge this yet, we're still discussing the code label Feb 17, 2026
@halgari halgari changed the title Duckdb integration Datamodel Query system built on DuckDB Feb 17, 2026
@halgari

This comment was marked as outdated.

Comment thread src/shared/types/ipc.ts Outdated
Comment thread src/shared/types/ipc.ts Outdated
Comment thread src/main/store/DuckDBSingleton.ts Outdated
Comment thread src/main/store/DuckDBSingleton.ts Outdated
Comment thread src/main/store/LevelPersist.ts Outdated
Comment thread src/main/store/QueryInvalidator.ts Outdated
@halgari halgari removed the don't-merge Don't merge this yet, we're still discussing the code label Feb 18, 2026
@halgari halgari force-pushed the duckdb-integration branch from d183c0c to edb4b2d Compare March 17, 2026 14:27
@insomnious
Copy link
Copy Markdown
Contributor

Closes APP-67

@github-actions
Copy link
Copy Markdown

This PR has conflicts. You need to rebase the PR before it can be merged.

@github-actions
Copy link
Copy Markdown

This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed.

@github-actions
Copy link
Copy Markdown

This PR has conflicts. You need to rebase the PR before it can be merged.

@halgari halgari changed the title Datamodel Query system built on DuckDB feat: replace LevelDB with DuckDB as persistence backend Mar 23, 2026
@halgari halgari force-pushed the duckdb-integration branch from e8cb8c1 to f479de1 Compare March 23, 2026 18:23
@halgari halgari requested a review from a team as a code owner March 23, 2026 18:23
@halgari halgari force-pushed the duckdb-integration branch from f479de1 to 3e357d9 Compare March 23, 2026 18:32
@github-actions
Copy link
Copy Markdown

This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed.

Comment thread src/main/src/store/LevelPersist.ts Outdated
erri120
erri120 previously approved these changes Mar 24, 2026
Copy link
Copy Markdown
Member

@erri120 erri120 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed some small fixes to keep the PR feedback loop small on this. Only issue is the generate-query-types script which I'm not sure when or how we want to use it and also it has import failures.

@github-actions
Copy link
Copy Markdown

This PR has conflicts. You need to rebase the PR before it can be merged.

Replace the LevelDB persistence layer (levelup/encoding-down) with DuckDB's
level_pivot extension via @duckdb/node-api. The level_pivot raw mode reads
existing LevelDB databases directly with no migration needed.

- Add @duckdb/node-api dependency (1.4.4-r.1)
- Rewrite LevelPersist.ts internals: DuckDB in-memory instance with LevelDB
  ATTACHed via level_pivot, SQL operations instead of streams
- Remove levelup, encoding-down, @types/levelup, @types/encoding-down
- Keep leveldown for repairDB, keep same class/export shape and IPersistor interface
@github-actions
Copy link
Copy Markdown

This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed.

erri120
erri120 previously approved these changes Mar 24, 2026
@github-actions
Copy link
Copy Markdown

This PR has conflicts. You need to rebase the PR before it can be merged.

@Aragas
Copy link
Copy Markdown
Member

Aragas commented Mar 24, 2026

Looks really cool!

…script

- Remove #restackingFunc from LevelPersist, convert wrapped methods to plain async
- Switch level_pivot install to community repo in DuckDBSingleton and script
- Fix broken import/output paths in generate-query-types script
- Use createRequire to resolve @duckdb/node-api from @vortex/main workspace
- Fix temp file cleanup ordering (detach before close, cleanup after shutdown)
- Add pnpm generate:query-types alias
- Generate initial queryTypes.ts
Resolve conflict in ipc.ts: keep persist:push and app:init channels
@github-actions
Copy link
Copy Markdown

This PR doesn't have conflicts anymore. It can be merged after all status checks have passed and it has been reviewed.

@erri120
Copy link
Copy Markdown
Member

erri120 commented Mar 25, 2026

I'm getting this error when launching Vortex:

Failed to store application state: Binder Error: The specified columns as conflict target are not referenced by UNIQUE/PRIMARY KEY CONSTRAINT or INDEX

Stack shows ReduxPersistorIPC.ts:253:11.

- Change pivot row types from interface to type alias so they satisfy
  the Record<string, unknown> constraint on Table<T> (fixes build)
- Replace INSERT ... ON CONFLICT upsert with transactional UPDATE-first,
  INSERT-if-missing pattern since level_pivot tables don't support
  UNIQUE indexes (fixes runtime persistence crashes)
- Track transaction state so setItem wraps in its own transaction when
  called standalone but skips when already inside one
@erri120 erri120 merged commit 9dc5851 into master Mar 25, 2026
5 checks passed
@erri120 erri120 deleted the duckdb-integration branch March 25, 2026 14:35
@erri120 erri120 mentioned this pull request Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants