Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions .cursor/rules/benchmarking.mdc
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Use this mapping when deciding which React benchmark scenarios are relevant to a
- Relevant for: `@data-client/react` hooks, `@data-client/core` store initialization
- All libraries

- **Update propagation** (`update-single-entity`, `update-shared-user-500-mounted`, `update-shared-user-10000-mounted`)
- **Update propagation** (`update-entity`, `update-user`, `update-user-10000`)
- Exercises: store update → React rerender → DOM mutation
- Relevant for: `@data-client/core` dispatch/reducer, `@data-client/react` subscription/selector
- All libraries (normalization advantage shows with shared user at scale)
Expand All @@ -41,7 +41,12 @@ Use this mapping when deciding which React benchmark scenarios are relevant to a
- Relevant for: `@data-client/normalizr` denormalize memoization, Entity identity
- All libraries (data-client should show fewest changed refs)

- **Sorted/derived view** (`sorted-view-mount-500`, `sorted-view-update-entity`)
- **Multi-view entity update** (`update-entity-multi-view`)
- Exercises: cross-query entity propagation — one update to a shared entity reflected in list, detail panel, and pinned cards
- Relevant for: `@data-client/normalizr` normalized cache, `@data-client/core` subscription fan-out
- All libraries (normalization advantage: one store write vs. multiple query invalidations + refetches)

- **Sorted/derived view** (`getlist-500-sorted`, `update-entity-sorted`)
- Exercises: `Query` schema memoization via `useQuery` (data-client) vs `useMemo` sort (competitors)
- Relevant for: `@data-client/endpoint` Query, `@data-client/normalizr` MemoCache, `@data-client/react` useQuery
- All libraries
Expand All @@ -58,8 +63,8 @@ Use this mapping when deciding which React benchmark scenarios are relevant to a

| Category | Scenarios | Typical run-to-run spread |
|---|---|---|
| **Stable** | `getlist-*`, `update-single-entity`, `ref-stability-*`, `sorted-view-mount-*` | 2–5% |
| **Moderate** | `update-shared-user-*`, `sorted-view-update-*` | 5–10% |
| **Stable** | `getlist-*`, `update-entity`, `ref-stability-*` | 2–5% |
| **Moderate** | `update-user-*`, `update-entity-sorted` | 5–10% |
| **Volatile** | `memory-mount-unmount-cycle`, `startup-*`, `(react commit)` suffixes | 10–25% |

Regressions >5% on stable scenarios or >15% on volatile scenarios are worth investigating.
Expand Down
2 changes: 1 addition & 1 deletion examples/benchmark-react/.babelrc.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
const options = { polyfillMethod: false };
if (process.env.REACT_COMPILER === 'true') {
if (process.env.REACT_COMPILER !== 'false') {
options.reactCompiler = {};
}

Expand Down
70 changes: 46 additions & 24 deletions examples/benchmark-react/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,12 @@ The repo has two benchmark suites:
**Hot path (CI)**

- **Get list** (`getlist-100`, `getlist-500`) — Time to show a ListView component that auto-fetches 100 or 500 issues from the list endpoint, then renders (unit: ms). Exercises the full fetch + normalization + render pipeline.
- **Update single entity** (`update-single-entity`) — Time to update one issue and propagate to the UI (unit: ms).
- **Update shared user (scaling)** (`update-shared-user-500-mounted`, `update-shared-user-10000-mounted`) — Update one shared user with 500 or 10,000 mounted issues to test subscriber scaling. Normalized cache: one store update, all views of that user update.
- **Get list sorted** (`getlist-500-sorted`) — Mount 500 issues through a sorted/derived view. data-client uses `useQuery(sortedIssuesQuery)` with `Query` schema memoization; competitors use `useMemo` + sort.
- **Update entity** (`update-entity`) — Time to update one issue and propagate to the UI (unit: ms).
- **Update entity sorted** (`update-entity-sorted`) — After mounting a sorted view, update one entity. data-client's `Query` memoization avoids re-sorting when sort keys are unchanged.
- **Update entity multi-view** (`update-entity-multi-view`) — Update one issue that appears simultaneously in a list, a detail panel, and a pinned-cards strip. Exercises cross-query entity propagation: normalized cache updates once and all three views reflect the change; non-normalized libraries must invalidate and refetch each query independently.
- **Update user (scaling)** (`update-user`, `update-user-10000`) — Update one shared user with 1,000 or 10,000 mounted issues to test subscriber scaling. Normalized cache: one store update, all views of that user update.
- **Ref-stability** (`ref-stability-issue-changed`, `ref-stability-user-changed`) — Count of components that received a **new** object reference after an update (unit: count; smaller is better). Normalization keeps referential equality for unchanged entities.
- **Sorted view mount** (`sorted-view-mount-500`) — Mount 500 issues through a sorted/derived view. data-client uses `useQuery(sortedIssuesQuery)` with `Query` schema memoization; competitors use `useMemo` + sort.
- **Sorted view update** (`sorted-view-update-entity`) — After mounting a sorted view, update one entity. data-client's `Query` memoization avoids re-sorting when sort keys are unchanged.
- **Invalidate and resolve** (`invalidate-and-resolve`) — data-client only; invalidates a cached endpoint and immediately re-resolves. Measures Suspense boundary round-trip.

**With network (local comparison)**
Expand All @@ -50,22 +51,43 @@ The repo has two benchmark suites:

## Expected results

These are approximate values to help calibrate expectations. Exact numbers vary by machine and CPU throttling.
Illustrative **relative** results with **SWR = 100%** (baseline). For **duration** rows, each value is (library median ms ÷ SWR median ms) × 100 — **lower is faster**. For **ref-stability** rows, the same idea uses the “refs changed” count — **lower is fewer components that saw a new object reference**. Figures are rounded from the **Latest measured results** table below (network simulation on); absolute milliseconds will vary by machine, but **library-to-library ratios** are usually similar.

| Scenario | data-client | tanstack-query | swr |
|---|---|---|---|
| `getlist-100` | ~similar | ~similar | ~similar |
| `update-shared-user-500-mounted` | Low (one store write propagates) | Higher (list refetch) | Higher (list refetch) |
| `ref-stability-issue-changed` (100 mounted) | ~1 changed | ~100 changed (list refetch) | ~100 changed (list refetch) |
| `ref-stability-user-changed` (100 mounted) | ~5 changed | ~100 changed (list refetch) | ~100 changed (list refetch) |
| `sorted-view-update-entity` | Fast (Query memoization skips re-sort) | Re-sorts on every issue change | Re-sorts on every issue change |
| Category | Scenarios (representative) | data-client | tanstack-query | swr |
|---|---|---:|---:|---:|
| Navigation | `getlist-100`, `getlist-500`, `getlist-500-sorted` | ~103% | ~102% | **100%** |
| Navigation | `list-detail-switch` | ~21% | ~102% | **100%** |
| Mutations | `update-entity`, `update-user`, `update-entity-sorted`, `update-entity-multi-view`, `unshift-item`, `delete-item`, `move-item` | ~2% | ~102% | **100%** |
| Scaling (10k items) | `update-user-10000` | ~5% | ~122% | **100%** |


## Latest measured results (network simulation on)

Median per metric; range is approximate 95% CI margin from the runner (`stats.ts`). **Network simulation** applies the per-RPC delays in `bench/scenarios.ts` (`NETWORK_SIM_DELAYS`, e.g. `fetchIssueList` 80 ms, `updateUser` 50 ms) so list refetches after an author update pay extra latency compared to normalized propagation.

Run: **2026-03-21**, Linux (WSL2), `yarn build:benchmark-react`, static preview + `env -u CI npx tsx bench/runner.ts --network-sim true` (all libraries; memory scenarios not included). Numbers are **machine-specific**; use them for relative comparison between libraries, not as absolutes.

| Scenario | Unit | data-client | tanstack-query | swr |
|---|---|---:|---:|---:|
| `getlist-100` | ms | 89.3 ± 0.22 | 88.7 ± 0.15 | 87.5 ± 0.50 |
| `getlist-500` | ms | 102.3 ± 1.25 | 99.9 ± 1.25 | 98.4 ± 1.25 |
| `getlist-500-sorted` | ms | 101.8 ± 1.61 | 99.2 ± 1.29 | 97.9 ± 0.63 |
| `list-detail-switch` | ms | 144.4 ± 21.22 | 689.4 ± 20.83 | 674.5 ± 35.67 |
| `update-entity` | ms | 2.8 ± 0.09 | 142.6 ± 0.31 | 142.4 ± 0.34 |
| `update-user` | ms | 3.0 ± 0.13 | 142.7 ± 0.43 | 139.4 ± 0.51 |
| `update-entity-sorted` | ms | 3.2 ± 0.24 | 141.3 ± 0.07 | 141.4 ± 0.56 |
| `update-entity-multi-view` | ms | 2.8 ± 0.41 | 146.6 ± 7.25 | 145.3 ± 8.21 |
| `update-user-10000` | ms | 10.3 ± 0.82 | 246.0 ± 1.35 | 201.2 ± 0.75 |
| `unshift-item` | ms | 3.5 ± 0.06 | 144.5 ± 0.38 | 139.7 ± 0.07 |
| `delete-item` | ms | 3.2 ± 0.10 | 144.4 ± 0.11 | 139.9 ± 0.11 |
| `move-item` | ms | 3.5 ± 0.13 | 156.4 ± 0.50 | 146.4 ± 0.05 |

## Expected variance

| Category | Scenarios | Typical run-to-run spread |
|---|---|---|
| **Stable** | `getlist-*`, `update-single-entity`, `ref-stability-*`, `sorted-view-mount-*` | 2-5% |
| **Moderate** | `update-shared-user-*`, `sorted-view-update-*` | 5-10% |
| **Stable** | `getlist-*`, `update-entity`, `ref-stability-*` | 2-5% |
| **Moderate** | `update-user-*`, `update-entity-sorted`, `update-entity-multi-view` | 5-10% |
| **Volatile** | `memory-mount-unmount-cycle`, `startup-*`, `(react commit)` suffixes | 10-25% |

Regressions >5% on stable scenarios or >15% on volatile scenarios are worth investigating.
Expand Down Expand Up @@ -107,24 +129,24 @@ Regressions >5% on stable scenarios or >15% on volatile scenarios are worth inve

Or from repo root after a build: start preview in one terminal, then in another run `yarn workspace example-benchmark-react bench`.

3. **With React Compiler**
3. **Without React Compiler**

To measure the impact of React Compiler, build and bench with it enabled:
The default build includes React Compiler. To measure impact without it:

```bash
cd examples/benchmark-react
yarn build:compiler # builds with babel-plugin-react-compiler
yarn build:no-compiler # builds without babel-plugin-react-compiler
yarn preview &
sleep 5
yarn bench:compiler # labels results with [compiler] suffix
yarn bench:no-compiler # labels results with [no-compiler] suffix
```

Or as a single command: `yarn bench:run:compiler`.
Or as a single command: `yarn bench:run:no-compiler`.

Results are labelled `[compiler]` so you can compare side-by-side with a normal run by loading both JSON files into the report viewer's history feature.
Results are labelled `[no-compiler]` so you can compare side-by-side with the default run by loading both JSON files into the report viewer's history feature.

You can also set the env vars directly for custom combinations:
- `REACT_COMPILER=true` — enables the Babel plugin at build time
Env vars for custom combinations:
- `REACT_COMPILER=false` — disables the Babel plugin at build time
- `BENCH_LABEL=<tag>` — appends `[<tag>]` to all result names at bench time
- `BENCH_PORT=<port>` — port for `preview` server and bench runner (default `5173`)
- `BENCH_BASE_URL=<url>` — full base URL override (takes precedence over `BENCH_PORT`)
Expand Down Expand Up @@ -163,8 +185,8 @@ Regressions >5% on stable scenarios or >15% on volatile scenarios are worth inve

Scenarios are classified as `small` or `large` based on their cost:

- **Small** (3 warmup + 15 measurement): `getlist-100`, `update-single-entity`, `ref-stability-*`, `invalidate-and-resolve`, `unshift-item`, `delete-item`
- **Large** (1 warmup + 4 measurement): `getlist-500`, `update-shared-user-500-mounted`, `update-shared-user-10000-mounted`, `update-shared-user-with-network`, `sorted-view-mount-500`, `sorted-view-update-entity`
- **Small** (3 warmup + 15 measurement): `getlist-100`, `update-entity`, `ref-stability-*`, `invalidate-and-resolve`, `unshift-item`, `delete-item`
- **Large** (1 warmup + 4 measurement): `getlist-500`, `getlist-500-sorted`, `update-user`, `update-user-10000`, `update-entity-sorted`, `update-entity-multi-view`, `list-detail-switch`
- **Memory** (opt-in, 1 warmup + 3 measurement): `memory-mount-unmount-cycle` — run with `--action memory`

When running all scenarios (`yarn bench`), each group runs with its own warmup/measurement count. Use `--size` to run only one group.
Expand Down
Loading
Loading