Cortex `consolidate` — Performance Report

[darval]

Cortex consolidate — Performance Report

Plugin: cortex (v3.9.1, per skill path)
Date: 2026-04-15
Environment: macOS 25.4.0 (Darwin), Mac M2 Max, 64 GB RAM

Summary

A full consolidate({decay, compress, cls, memify}) run on a memory
store of ~66K memories took 35 min 50 s (2,150,496 ms). Most of
the work appears to be O(N) per-row updates across stages that could
plausibly be batched. One stage scanned but produced nothing, and the
homeostatic stage reported health_score: 0.0 and skipped scaling.

I'd like to suggest a few low-risk optimizations and — more usefully —
some per-stage telemetry that would let maintainers (and operators)
see where time actually goes on real stores.

Pre-run state

{
  "total_memories": 65949,
  "episodic_count": 25319,
  "semantic_count": 40630,
  "active_count":   65610,
  "archived_count": 339,
  "protected_count": 3088,
  "avg_heat": 0.6746,
  "total_entities": 10770,
  "total_relationships": 34735,
  "active_triggers": 1456,
  "last_consolidation": "2026-04-13T01:20:15Z",   // ~2 days prior
  "has_vector_search": true
}

Full-run result

{
  "decay":       { "memories_decayed": 62522, "metabolic_updates": 65937,
                   "entities_decayed": 10770, "total_memories": 65949 },
  "plasticity":  { "ltp": 17, "ltd": 33092, "edges_updated": 33109 },
  "pruning":     { "edges_pruned": 32024, "entities_archived": 718 },
  "compression": { "compressed_to_gist": 213, "compressed_to_tag": 0,
                   "protected_skipped": 3088, "semantic_skipped": 38502 },
  "cls":         { "patterns_found": 0, "new_semantics_created": 0,
                   "skipped_inconsistent": 0, "skipped_duplicate": 0,
                   "causal_edges_found": 0 },
  "memify":      { "pruned": 0, "strengthened": 0, "reweighted": 1345 },
  "cascade":     { "advanced": 503, "transitions": [... 503 entries ...] },
  "homeostatic": { "scaling_applied": false, "health_score": 0.0 },
  "duration_ms": 2150496
}

Observations

1. Most work is row-by-row across large result sets

• Decay touched 62,522 memories (94.8 % of the store) plus 10,770
  entities. If this is implemented as per-row UPDATE statements
  inside a loop, it is the most plausible source of the bulk of the
  35 minutes.
• Plasticity updated 33,109 edges (17 LTP + 33,092 LTD).
• Pruning deleted 32,024 edges.

These three stages together account for ~138K row operations. A single
set-based UPDATE ... SET heat = heat * :decay WHERE ... (and the
analogous UPDATE / DELETE for plasticity and pruning) is typically
orders of magnitude faster than per-row application logic.

2. CLS scanned ~25K episodic memories and produced zero output

patterns_found: 0, new_semantics_created: 0,
skipped_inconsistent: 0, skipped_duplicate: 0, causal_edges_found: 0

A stage that finds nothing after a 2-day gap on a 25K-episodic store
is either miscalibrated (threshold too strict) or doing expensive
work with no observable effect. Either way, it's a candidate for
early-exit when the input set hasn't changed meaningfully since the
last run.

3. Homeostatic stage reports health_score: 0.0, scaling skipped

"scaling_applied": false, "health_score": 0.0 looks like either a
divide-by-zero / empty-input guard short-circuiting, or a metric that
legitimately is zero but whose name suggests a bug. Worth a glance.

4. Compression is heavily skipped

protected_skipped: 3088, semantic_skipped: 38502, with only
compressed_to_gist: 213. Skips are ~99 % of the candidate set. If
those skip checks are cheap, this is fine — but if the skip path
re-reads each memory's content/flags from the DB, that's another
O(N) tax.

5. No per-stage duration in the result payload

Today the response includes only duration_ms (total). Everything
above is inferred from row counts. Per-stage timings would make
bottleneck identification mechanical instead of speculative.

Suggested telemetry (the most useful change, probably)

Return duration_ms per stage in the existing result dict:

{
  "decay":       { ..., "duration_ms": 1234567 },
  "plasticity":  { ..., "duration_ms": 234567 },
  "pruning":     { ..., "duration_ms": 45678 },
  "compression": { ..., "duration_ms": 12345 },
  "cls":         { ..., "duration_ms": 456789 },   // scanned but empty
  "memify":      { ..., "duration_ms": 6789 },
  "cascade":     { ..., "duration_ms": 3456 },
  "homeostatic": { ..., "duration_ms": 12 },
  "duration_ms":  2150496
}

If it's useful, I'd be happy to collect and share per-run timing
stats from my installation so you have at least one real-world
large-store data point to tune against.

Additional fields that would be valuable if cheap to collect:

Field	Why
rows_scanned vs rows_modified per stage	Catches the CLS-style "scanned 25K, produced 0" case
db_query_count, db_tx_count per stage	Surfaces N+1 patterns directly
avg_work_per_memory_ms	Normalizes across memory-store sizes
peak_memory_mb	Flags when in-memory structures should be streamed
delta_vs_last_run_ms	Drift signal — "this run was 2× slower than last"

Suggested optimizations (in rough order of likely payoff)

1. Batch the decay update. One SQL statement over the candidate
   set instead of per-row updates. Same for plasticity LTD and pruning.
2. Cooldown window on decay. Skip memories whose last_decayed
   timestamp is within the configured cooldown — on a 2-day gap this
   may not help, but on daily runs it should dramatically shrink the
   working set.
3. Early-exit CLS when the episodic-memory delta (new / accessed
   since last run) is below some minimum. The current run scanned the
   whole 25K to produce zero patterns.
4. Instrument the *_skipped counters in compression so it's
   clear whether the skip check itself is the expense or only the
   (rare) non-skip path.
5. Fix or explain health_score: 0.0 + scaling_applied: false.
   Either the metric name is misleading (rename) or the scaling path
   is silently disabled (fix).

Offer to help

If any of the above is actionable and you'd like:

• a real-world workload for benchmarking (my ~66K-memory store before
  and after a patched run),
• per-stage timings once the telemetry lands, or
• a PR for the telemetry change itself (adding duration_ms per
  stage is a small, contained change),

happy to do it. Let me know what's most useful.

Repro

Tool: mcp__plugin_cortex_cortex__consolidate
Args: { decay: true, compress: true, cls: true, memify: true, deep: false }
Store size at run: 65949 memories, 40630 semantic, 34735 relationships
Time since last consolidate: ~2 days
Wall-clock duration: 2,150,496 ms (35 min 50 s)

[darval]

I had recently, a couple days ago, imported all the existing memories into cortex, so this is only the 2nd or 3rd attempt to consolidate the memories. However it is a pretty large dataset, over a network connection to my sql server, so it makes it easy to see optimization opportunities. ;)

[cdeust]

Thanks Rick — this is an exceptionally good report, and the six observations held up under a full review (code-reviewer, test-engineer, and four genius-reasoning passes). Short version: you uncovered one real bug and five performance pathologies, and the diagnostic-first instinct was correct.

What shipped in 69d81fb (v3.10.1 + 1)

1. Per-stage duration_ms — your top ask. Every stage in the consolidate response now carries its own timing; stage-local exceptions are captured as {"error": "...", "duration_ms": N} instead of crashing the whole handler. Covers decay, plasticity, pruning, compression, cls, memify, deep_sleep, cascade, homeostatic, transfer, emergence.

2. Top-level failed_stages + status rollup — so a caller that only reads duration_ms can still see that one or more stages errored inside the _timed capture. Result shape is now {..., "duration_ms": N, "failed_stages": ["stage-a"], "status": "ok" | "partial"}.

3. The homeostatic health_score: 0.0 you flagged was a real bug — and not a divide-by-zero guard. Your review re-derivation caught it:

# before — wrong module
from mcp_server.core import homeostatic_plasticity
health = homeostatic_plasticity.compute_distribution_health(...)  # AttributeError

# after — function lives in homeostatic_health
from mcp_server.core import homeostatic_health
health = homeostatic_health.compute_distribution_health(...)

The except Exception: logger.debug(...) path was swallowing an AttributeError on every real-store run and returning {"health_score": 0.0, "scaling_applied": False}. On a synthetic 66K-memory distribution matching your numbers, the fixed handler now returns:

{
  "scaling_applied": true,
  "health_score": 0.3653,
  "mean_heat": 0.5450,
  "std_heat": 0.0287,
  "bimodality": 0.5632,
  "memories_scanned": 65949
}

The old logger.debug is now logger.warning with exc_info=True, and the result dict explicitly reports "reason": "no_memories" vs "error": "..." so the two cases are no longer indistinguishable.

4. 8 new regression tests (tests_py/handlers/test_consolidate_telemetry.py) cover the wrapper contract and both homeostatic outcome paths. 247 tests pass (was 239).

Out of scope for this patch — tracked as v3.11 work

I deliberately kept this PR to telemetry + the one confirmed bug, because the agents were unanimous that measurement-before-surgery is the right order. Five pathologies came out of the review that account for most of your 35 minutes:

#	Pathology	Evidence	Est. impact
1	pg_store.update_memory_heat commits per row (fsync amplification on ~100K calls)	Feynman's integrity audit, pg_store.py:236-238	60-80% of wall-clock; batching → 100-500× speedup on decay
2	Plasticity get_hot_memories(limit=50) starves the co-access set → 99.95% LTD is distribution collapse, not healthy plasticity	17:33,092 LTP:LTD ratio with 50-memory window across 10,770 entities	Correctness, not just perf
3	CLS _discover_causal_edges hard-caps at 500 episodic on your 25K store (2% sample), does ~5.4M substring ops for a result that's structurally 0	cls.py:132-169, scanned-vs-modified	Mostly correctness: the stage is cargo-culted at this scale
4	Cascade emits per-transition UPDATE + commit and the 503-transition payload duplicates the queryable stage_transitions table	cascade.py:135, :172	Response size + fsync-per-transition
5	get_all_memories_for_decay() called 2-3× per run (decay, compression, emergence)		Load-bound cost, easy win via consolidation-scoped cache

Erlang's queuing analysis on your numbers: ~64 ops/s sustained throughput, 1,100 s on decay alone — consistent with ~15-20 ms/row dominated by per-commit fsync on macOS SSD. Batching the decay/plasticity/pruning UPDATEs into a single UPDATE ... WHERE id = ANY($1) collapses that stage from ~1,100 s to an estimated 3-10 s.

Help we'd welcome

• Per-stage timings from your store once you rerun with this release — that's the ground truth we need to size each follow-up fix. The rows_scanned/rows_modified fields you suggested are on the list for v3.11 once the batching PR lands (they're most useful after the load-dominated vs commit-dominated question is settled).
• PR for the batching change — if you'd like to take that one, happy to pair on the SQL shape + a benchmark harness. The largest win and the cleanest scope.

Thanks again. The telemetry you asked for is now in place and the homeostatic bug you surfaced is fixed.

[darval]

Sounds great. I will try to get into that later today. I am knee deep in trying to optmize my use of the AI pipeline (third run now) looking primarily at token optimization (volume and cost) and will update that repo/issue with my report once done. After that, I can update cortex and consolidate and see what we learn.

[darval]

Just for reference, I am running Claude Code, etc.. on a Mac Studio M2, but the sql server is a v17 postgresql running on a LXC container on a linux hypervisor. It has 8 cores assigned and 10GB memory, but in observing it during this, the bottle neck is generally network and/or query design. I can also collect sql stats from the server if we think that will help.

[cdeust]

Rick — v3.11.0 is out with all five deferred fixes. Summary:

Phase A — Set-based SQL batching on every per-row loop the audit found. One UPDATE … FROM UNNEST replaces 62k per-row UPDATEs on decay; same pattern for plasticity (33k), pruning (32k DELETEs), and cascade transitions (503 INSERTs). Erlang predicts decay alone drops from ~1100 s → 3–10 s on your store.

Phase B — get_all_memories_for_decay() was called 6× per run (decay, compression, memify, homeostatic, sleep, emergence). Now loaded once, threaded through every stage.

Phase C — Plasticity get_hot_memories(limit=50) was sampling 0.5% of your 10k-entity store — the 17:33,092 LTP:LTD ratio was sampling collapse, not biology. Raised to 2000, reuses Phase B cache, precomputes the lowercase name index.

Phase D — CLS episodic cap raised 500 → 2000 and added an early-exit gate: if fewer than 5 entities clear the min-observations threshold, skip the O(E²) PC pass. Also restricted the PC vocabulary to qualifying entities so the matrix is E_qualifying² not E_all².

Phase E — Cascade now skips the heartbeat UPDATE when |Δhours| < 1 (the write was pure fsync amplification below that delta). stage_transitions INSERTs are batched into one statement. Response payload capped at transitions_preview (first 50) + transitions_count.

SQLite parity — Your report is CLI-mode Postgres so this doesn't affect you directly, but the cowork fallback path previously crashed cascade silently because the SQLite schema was missing stage_transitions + stage_entered_at. Fixed in this release too.

Agent review on the diff: code-reviewer APPROVE, test-engineer 2188 tests pass, genius:erlang/feinstein/feynman verified.

If you rerun consolidate on your 66K-memory store, the per-stage duration_ms from v3.10.1 will now let us confirm the batching win against the real measurement. Would very much appreciate a post-upgrade datapoint — it's the ground truth we need to validate Erlang's 100–500× prediction.

Release: https://github.com/cdeust/Cortex/releases/tag/v3.11.0

[darval]

Just to clarify, do you want me to run it twice? Once with 3.10.1 to collect telemetry and then upgrade to 3.11 and run it again to see the improvement? Or just upgrade once to 3.11 and run it once?

[cdeust]

do you want me to run it twice?

Just once on 3.11 is enough, thanks. Your original report already gives us the before-state baseline (35m50s, the counters in the consolidate result, and the homeostatic health_score: 0.0), so a single run post-upgrade is the after-measurement. The v3.10.1 telemetry patch was only needed as the instrument — v3.11 carries it forward, so one run on 3.11 produces all the numbers we need to validate the Erlang prediction.

---

Closeout checklist

All six observations from the original report have code-side fixes shipped. Items traced to the Phase labels in the v3.11.0 release notes.

Original observation	Status	Where
1. Decay per-row UPDATE (62k rows)	✅ Fixed	Phase A — update_memories_heat_batch (single UPDATE … FROM UNNEST)
2. CLS scanned ~25K episodic, produced 0	✅ Fixed	Phase D — episodic cap 500→2000, early-exit gate on qualifying entities, PC vocabulary restricted
3. Homeostatic health_score: 0.0	✅ Fixed	v3.10.1 — root cause was a wrong-module import (compute_distribution_health lives in homeostatic_health, not homeostatic_plasticity); explicit reason/error fields now
4. Compression heavily skipped	✅ Clarified	Phase B — skips confirmed cheap (dict lookups on already-loaded rows); added rows_scanned and removed the redundant reload
5. No per-stage duration	✅ Fixed	v3.10.1 — _timed wrapper, plus failed_stages / status rollup

Suggested optimizations

Suggestion	Status	Notes
1. Batch the decay update	✅ Shipped	Phase A — also batched plasticity, pruning, cascade
2. Cooldown window on decay	🔶 Deferred (by design)	Erlang analysis: filters ~0% of rows for your 2-day cadence, marginal for sub-daily. Will revisit if post-batching measurements still show decay as the hotspot
3. Early-exit CLS	✅ Shipped	Phase D gate on _MIN_ENTITIES_FOR_PC
4. Instrument compression skip counters	✅ Partial	rows_scanned added; the requested *_skipped ratios fall out of the existing protected_skipped/semantic_skipped ÷ rows_scanned
5. Fix health_score: 0.0	✅ Fixed	Root cause was an import bug, not a divide-by-zero as hypothesized

Beyond the original report

The audit (feinstein/feynman/erlang) surfaced three pathologies the report didn't directly call out, all shipped in v3.11.0:

• Plasticity co-access starvation — get_hot_memories(limit=50) was sampling 0.5% of your 10k-entity store; the 17:33,092 LTP:LTD ratio was sampling collapse, not plasticity. Fixed by raising the window to 2000 and reusing the Phase B cache.
• Redundant memory reload — get_all_memories_for_decay() was called 6× per consolidate run. Now loaded once, threaded through every stage.
• Cascade heartbeat UPDATEs — cascade wrote an UPDATE on every scanned memory (~2000) even when nothing advanced. Skipped when |Δhours| < 1h.

---

Closing

Closing as all code-side items are addressed. Please reopen (or ping) if your rerun shows the Erlang 100–500× prediction doesn't hold, if any new edge appears, or if the telemetry surfaces something unexpected — your ~66K-memory store is the most useful real-world data point we have.

Thanks again for the exceptionally clear report. It improved Cortex by more than the fix list suggests.

[darval]

Done — ran both in a clean head-to-head. TL;DR: v3.11.0 is 2.2× faster overall (37.4 min → 16.8 min), decay batching works but gives 6.3×, not the predicted 100–500×. The headline is actually that homeostatic is now the #1 bottleneck and there's a second emergence_tracker import bug of the same shape as the one 69d81fb fixed.

Setup

To avoid Run B starting from Run A's post-consolidation state (which would deflate the speedup), I pg_dump'd my live cortex DB and restored into two independent DBs (cortex_a, cortex_b — both 66,064 memories). Pre-vacuumed both, disabled autovacuum on the significant tables, installed pg_stat_statements scoped per-DB. Spawned each version as an MCP subprocess with its own DATABASE_URL, drove it via JSON-RPC stdio, captured app telemetry + pg_stat_statements snapshot after each consolidate call. Full methodology + repro script at the bottom.

Results

Stage	Run A (69d81fb)	Run B (v3.11.0)	Speedup
decay	23.4 min	3.7 min	6.3×
homeostatic	11.4 min	11.5 min	1.0×
memify	37.4 s	19.6 s	1.9×
plasticity	29.3 s	16.3 s	1.8×
cascade	21.5 s	16.2 s	1.3×
compression	19.2 s	2.1 s	9.3×
pruning	18.0 s	2.8 s	6.4×
cls	12.2 s	15.5 s	0.79× (slightly slower)
emergence	17.1 s (ERR)	0 s (ERR)	—
TOTAL	2245 s (37.4 min)	1006 s (16.8 min)	2.2×

Both runs returned status: "partial", failed_stages: ["emergence"] — the _timed wrapper from 69d81fb did exactly what it was supposed to (captured the exception, reported duration_ms, didn't crash siblings, surfaced in the rollup).

Batching evidence from pg_stat_statements

Aggregated across all heat-UPDATE query shapes on the memories table:

	calls	rows	SQL time
Run A per-row (UPDATE memories SET heat = $1 WHERE id = $2)	194,360	194,360	425.7 s
Run B per-row	65,697	65,697	151.1 s
Run B batched (UPDATE memories ... FROM (SELECT UNNEST($1::int[])...)	14	128,673	222.8 s

So: batching absolutely landed. 128K+14 rows/calls on the batched path, mean 15.9 s per call (batches of ~9K rows). Decay stage dropped from ~1400 s to ~220 s as predicted by the SQL — app-level duration_ms matches within noise.

But Run B still issues 65,697 per-row heat UPDATEs. These are coming from stages that weren't in the Phase A scope — plasticity and homeostatic still write heat one row at a time. Plasticity's relatively small (~16 s) but homeostatic is the new #1.

Why 6.3× on decay, not 100–500×

Single-trial, so take with salt, but the gap is real. Plausible causes for the shortfall:

• vector index maintenance. The 489 MB memories table has hnsw/ivfflat indexes from vector 0.8.2 that update per-row-touched, not per-statement. A 9K-row UPDATE in one statement still does 9K index updates internally.
• WAL volume. Batching collapses commit count, but every row update still writes a WAL record. On macOS/ZFS-backed server storage, WAL flush is ~30 ms for our batch sizes (observed WalSync wait events in Run A).
• Network RTT. LAN RTT < 1 ms, so the per-round-trip overhead in Run A is bounded by ~194K × 1 ms = ~3 min max — consistent with Run A's total-vs-SQL gap.

The 6.3× is still the biggest single-commit wall-clock win in the release; just might want to soften the 100–500× language in the v3.11 notes since that was the fsync-dominated model.

Bigger finding: homeostatic is now 69% of total runtime

In Run B, homeostatic takes 11.5 min of the 16.8-min total — it didn't get faster because it wasn't in scope. Now that decay dropped out of #1, homeostatic is the elephant.

SQL-level: homeostatic is one of the sources of the 65,697 per-row heat UPDATEs in Run B. That's ~151 s of SQL time visible in pg_stat_statements. The stage reports duration_ms: 690646 (11.5 min), so 539 s is unaccounted for by the top 100 SQL statements. Something in the stage's Python side is slow — distribution-health computation on 66K memories, scaling loop, or the dispatch that decides which memories to touch.

The same UPDATE ... FROM UNNEST pattern you used in Phase A would collapse the 151 s of SQL to a handful of seconds, but it wouldn't touch the 539 s of unaccounted stage time. Might need profiling on that side specifically.

Second latent bug: emergence_tracker AttributeError

Same shape as the homeostatic import bug from 69d81fb:

AttributeError: module 'mcp_server.core.emergence_tracker' has no attribute ...

(Full error text truncated in my capture to 60 chars — happy to re-run and grab it if useful.) It appears in both Run A and Run B — v3.11.0 didn't fix it. In Run B the stage returns duration_ms: 0 because the error is raised before any work happens. _timed captures it correctly; status: partial and failed_stages: ["emergence"] surface it at the top level exactly as designed. But whatever emergence is supposed to compute, it hasn't been computing it.

Worth checking whether homeostatic_health was the only module that had a bad-import caller pre-#69d81fb fix.

Ranked targets for v3.12 from the Run B numbers

#	Stage	Run B time	Candidate fix
1	homeostatic	11.5 min	Batch the scaling UPDATEs; profile the 539 s of Python time
2	plasticity	16.3 s	Batch its heat writes (same UNNEST pattern)
3	emergence	0 s (err)	Fix the emergence_tracker import (same class of bug as homeostatic)
4	cls	15.5 s	Slight regression from v3.10.1; probably within noise but the early-exit gate didn't trip on this store

One thing you might want to check

Neither of my runs hit the stage_transitions INSERT batching you mentioned for cascade, because in Run B I only saw ~500 transitions total across the stage and the pre-batched single-INSERT shape didn't appear in pg_stat_statements. Might want to verify the batching is firing on cascade's actual production workload, not just the unit tests.

Repro

Full methodology + setup + cleanup docs in the gist below. Shortest possible repro:

git clone https://github.com/cdeust/Cortex.git /tmp/cortex-bench
cd /tmp/cortex-bench
git worktree add ../cortex-69d81fb 69d81fb
git worktree add ../cortex-v3110   v3.11.0
# pg_dump your cortex DB, restore twice into cortex_a and cortex_b (incl. extensions)
# VACUUM ANALYZE + disable autovacuum on both
# install pg_stat_statements per-DB
python3 run_consolidate.py /tmp/cortex-69d81fb cortex_a run_a run_a.json
python3 run_consolidate.py /tmp/cortex-v3110   cortex_b run_b run_b.json
python3 compare.py run_a.json run_b.json report.md

Harness + methodology + analysis attached separately. Let me know if you want the raw run_a.json / run_b.json dumps — they're small (~220 KB each) and have the full telemetry + SQL snapshots.