Add reward-modulated learning and adaptive sleep budget controls

CHANGELOG.md

@@ -9,6 +9,14 @@ for versioning even while in research-stage development.
 
 ### Added
 
+- Review-driven circadian updates in NumPy and ResNet circadian cores:
+  - optional reward-modulated wake learning (`use_reward_modulated_learning`)
+  - optional adaptive sleep budget scaling (`use_adaptive_sleep_budget`)
+  - `get_last_reward_scale()` telemetry helper
+- Baseline and ResNet benchmark CLI flags for reward modulation and adaptive sleep budget controls.
+- Review follow-up docs:
+  - `docs/circadian-model-review-notes.md`
+  - `docs/adr/ADR-0004-reward-modulated-wake-and-adaptive-sleep-budget.md`
 - Open-source community baseline files:
   - `LICENSE` (MIT)
   - `CODE_OF_CONDUCT.md`
@@ -41,6 +49,9 @@ for versioning even while in research-stage development.
 
 ### Changed
 
+- ResNet benchmark defaults now enable adaptive sleep budget scaling by default while keeping reward-modulated learning disabled by default.
+- Updated circadian unit tests (NumPy + Torch) with coverage for reward scaling and adaptive budget behavior.
+- Updated README, model card, and core module docs to document new circadian controls.
 - Repositioned repository messaging to Circadian Predictive Coding as the primary focus.
 - Updated `README.md` with:
   - circadian-first project framing

README.md

@@ -48,6 +48,7 @@ This lets model capacity adapt over time instead of staying fixed.
 - NumPy circadian predictive coding baseline for small-scale experiments
 - Torch ResNet-50 benchmark pipeline for speed and accuracy comparisons
 - Adaptive sleep triggers, adaptive split/prune thresholds, dual-timescale chemical dynamics
+- Reward-modulated wake learning and adaptive sleep budget scaling (NumPy + ResNet circadian head)
 - Function-preserving split behavior and guarded sleep rollback
 - Multi-seed benchmark runner with JSON/CSV output
 
@@ -141,6 +142,12 @@ Toy baseline:
 python predictive_coding_experiment.py
 ```
 
+Toy baseline with review-driven circadian controls:
+
+```powershell
+python predictive_coding_experiment.py --adaptive-sleep-trigger --adaptive-sleep-budget --reward-modulated-learning --reward-scale-min 0.8 --reward-scale-max 1.4
+```
+
 ResNet benchmark (all 3 models):
 
 ```powershell
@@ -183,6 +190,7 @@ pytest -q
 - Governance: [GOVERNANCE.md](GOVERNANCE.md)
 - Support process: [SUPPORT.md](SUPPORT.md)
 - Model Card: [docs/model-card.md](docs/model-card.md)
+- Review Notes: [docs/circadian-model-review-notes.md](docs/circadian-model-review-notes.md)
 
 ## Citation
 

docs/adr/ADR-0004-reward-modulated-wake-and-adaptive-sleep-budget.md

@@ -0,0 +1,47 @@
+# ADR-0004: Add Reward-Modulated Wake Updates and Adaptive Sleep Budget Scaling
+
+## Context
+
+External model review highlighted two practical gaps in the circadian learning loop:
+
+- wake learning lacks an explicit task-relevance modulation signal
+- sleep structural budgets rely heavily on fixed schedules/hyperparameters
+
+We need incremental improvements that keep the model deterministic, lightweight, and easy to test.
+
+## Decision
+
+Add two optional mechanisms to both circadian implementations:
+
+- `CircadianPredictiveCodingNetwork` (NumPy baseline)
+- `CircadianPredictiveCodingHead` (ResNet benchmark path)
+
+1. Reward-modulated wake updates
+- Compute batch difficulty from mean absolute output error.
+- Compare difficulty against an EMA baseline.
+- Scale learning updates by a clipped reward factor.
+
+2. Adaptive sleep budget scaling
+- Compute a budget scale from:
+  - recent energy plateau severity
+  - current hidden chemical variance
+- Apply the scale before enforcing configured split/prune limits and fraction caps.
+
+Expose both controls through baseline CLI and ResNet benchmark CLI flags. Keep defaults conservative (`off`) for backward compatibility.
+
+## Alternatives Considered
+
+1. Add RL/Bayesian meta-controller for sleep scheduling.
+- Rejected for now: larger complexity and harder reproducibility.
+
+2. Modulate split/prune ranking directly with reward first.
+- Deferred: needs clearer attribution of neuron-level contribution and stronger evaluation harness.
+
+3. Apply reward/adaptive budget in ResNet path first.
+- Initially rejected for sequencing reasons, then implemented after NumPy validation.
+
+## Consequences
+
+- Circadian wake updates can prioritize harder batches without changing model topology.
+- Sleep events become less dependent on manual tuning while staying deterministic.
+- New behavior is opt-in, preserving existing experiments by default.

docs/circadian-model-review-notes.md

@@ -0,0 +1,38 @@
+# Circadian Model Review Notes
+
+Review source: `C:\Users\Avery\Downloads\Circadian model review.pdf`
+
+## Scope
+
+This note maps review recommendations to concrete repository changes and follow-up work.
+
+## Implemented in this pass
+
+1. Reward-modulated wake learning
+- Added optional reward scaling in `CircadianPredictiveCodingNetwork` wake updates.
+- Batch difficulty is measured from mean absolute output error relative to an EMA baseline.
+- Scale is clipped via config to keep updates stable.
+- Why this: gives a simple task-relevance signal without changing the core predictive-coding math.
+
+2. Adaptive sleep budget scaling
+- Added optional scaling for split/prune budgets based on:
+  - energy plateau severity
+  - hidden chemical variance
+- Preserves `max_split_per_sleep` and `max_prune_per_sleep` as hard caps.
+- Why this: reduces manual schedule sensitivity while staying deterministic and lightweight.
+
+3. CLI exposure for new controls
+- Added baseline CLI flags to toggle reward modulation and adaptive sleep budget behavior.
+
+4. ResNet circadian parity
+- Added the same reward-modulated wake learning and adaptive sleep budget scaling to `CircadianPredictiveCodingHead`.
+- Exposed benchmark CLI/config knobs so ResNet benchmark runs can enable the mechanisms.
+
+5. Test coverage
+- Added unit tests for reward scaling behavior and adaptive budget expansion/contraction in both NumPy and Torch circadian paths.
+
+## Still pending (next iterations)
+
+1. Explore deeper-layer structural plasticity (current ResNet adaptation remains head-focused).
+2. Prototype faster inference variants (reduced steps or amortized inference).
+3. Evaluate reward-biased split/prune ranking directly (currently reward influences wake updates and importance EMA).

docs/model-card.md

@@ -21,7 +21,9 @@ It tracks per-neuron chemical usage, modulates plasticity during wake, and appli
 - Extension: circadian dynamics:
   - chemical accumulation and decay
   - plasticity gating
+  - reward-modulated wake learning (optional)
   - adaptive sleep triggers
+  - adaptive sleep budget scaling (optional)
   - structural split/prune
   - optional rollback and homeostatic controls
 

docs/modules/core.md

@@ -4,6 +4,7 @@
 
 - Define model behavior (`BackpropMLP`, `PredictiveCodingNetwork`, `CircadianPredictiveCodingNetwork`)
 - Define ResNet-50 benchmark variants (`BackpropResNet50Classifier`, `PredictiveCodingResNet50Classifier`, `CircadianPredictiveCodingResNet50Classifier`)
+- Implement circadian mechanisms such as chemical gating, reward-modulated wake updates, and adaptive sleep budgeting
 - Provide activation utilities
 - Define neuron adaptation interfaces and traffic summaries
 

src/adapters/cli.py

@@ -117,6 +117,37 @@ def build_argument_parser() -> argparse.ArgumentParser:
         type=float,
         default=circadian_defaults.sleep_chemical_variance_threshold,
     )
+    parser.add_argument(
+        "--adaptive-sleep-budget",
+        action="store_true",
+        help="Scale split/prune budgets based on plateau severity and chemical variance.",
+    )
+    parser.add_argument(
+        "--adaptive-sleep-budget-min-scale",
+        type=float,
+        default=circadian_defaults.adaptive_sleep_budget_min_scale,
+    )
+    parser.add_argument(
+        "--adaptive-sleep-budget-max-scale",
+        type=float,
+        default=circadian_defaults.adaptive_sleep_budget_max_scale,
+    )
+
+    parser.add_argument(
+        "--reward-modulated-learning",
+        action="store_true",
+        help="Scale wake learning rate by batch difficulty relative to recent error baseline.",
+    )
+    parser.add_argument(
+        "--reward-scale-min",
+        type=float,
+        default=circadian_defaults.reward_scale_min,
+    )
+    parser.add_argument(
+        "--reward-scale-max",
+        type=float,
+        default=circadian_defaults.reward_scale_max,
+    )
 
     parser.add_argument(
         "--split-weight-norm-mix",
@@ -188,6 +219,12 @@ def main() -> None:
         sleep_energy_window=arguments.sleep_energy_window,
         sleep_plateau_delta=arguments.sleep_plateau_delta,
         sleep_chemical_variance_threshold=arguments.sleep_chemical_variance_threshold,
+        use_adaptive_sleep_budget=arguments.adaptive_sleep_budget,
+        adaptive_sleep_budget_min_scale=arguments.adaptive_sleep_budget_min_scale,
+        adaptive_sleep_budget_max_scale=arguments.adaptive_sleep_budget_max_scale,
+        use_reward_modulated_learning=arguments.reward_modulated_learning,
+        reward_scale_min=arguments.reward_scale_min,
+        reward_scale_max=arguments.reward_scale_max,
         split_weight_norm_mix=arguments.split_weight_norm_mix,
         prune_weight_norm_mix=arguments.prune_weight_norm_mix,
         prune_decay_steps=arguments.prune_decay_steps,

src/adapters/resnet_benchmark_cli.py

@@ -113,6 +113,23 @@ def build_argument_parser() -> argparse.ArgumentParser:
     parser.add_argument("--circ-sleep-energy-window", type=int, default=48)
     parser.add_argument("--circ-sleep-plateau-delta", type=float, default=5e-5)
     parser.add_argument("--circ-sleep-chemical-variance-threshold", type=float, default=0.02)
+    parser.add_argument(
+        "--circ-use-adaptive-sleep-budget",
+        dest="circ_use_adaptive_sleep_budget",
+        action="store_true",
+        help="Enable adaptive split/prune budget scaling by plateau and chemical variance.",
+    )
+    parser.add_argument(
+        "--circ-disable-adaptive-sleep-budget",
+        dest="circ_use_adaptive_sleep_budget",
+        action="store_false",
+        help="Disable adaptive split/prune budget scaling.",
+    )
+    parser.set_defaults(circ_use_adaptive_sleep_budget=True)
+    parser.add_argument("--circ-adaptive-sleep-budget-min-scale", type=float, default=0.25)
+    parser.add_argument("--circ-adaptive-sleep-budget-max-scale", type=float, default=1.0)
+    parser.add_argument("--circ-adaptive-sleep-budget-plateau-weight", type=float, default=0.6)
+    parser.add_argument("--circ-adaptive-sleep-budget-variance-weight", type=float, default=0.4)
     parser.add_argument(
         "--circ-force-sleep",
         dest="circ_force_sleep",
@@ -168,6 +185,15 @@ def build_argument_parser() -> argparse.ArgumentParser:
     parser.add_argument("--circ-plasticity-sensitivity-max", type=float, default=0.55)
     parser.add_argument("--circ-plasticity-importance-mix", type=float, default=0.50)
     parser.add_argument("--circ-min-plasticity", type=float, default=0.5)
+    parser.add_argument(
+        "--circ-use-reward-modulated-learning",
+        action="store_true",
+        help="Scale wake learning rate by batch difficulty relative to a moving baseline.",
+    )
+    parser.add_argument("--circ-reward-baseline-decay", type=float, default=0.95)
+    parser.add_argument("--circ-reward-difficulty-exponent", type=float, default=1.0)
+    parser.add_argument("--circ-reward-scale-min", type=float, default=0.75)
+    parser.add_argument("--circ-reward-scale-max", type=float, default=1.5)
     parser.add_argument("--circ-use-adaptive-thresholds", action="store_true", default=None)
     parser.add_argument("--circ-adaptive-split-percentile", type=float, default=92.0)
     parser.add_argument("--circ-adaptive-prune-percentile", type=float, default=8.0)
@@ -260,6 +286,15 @@ def main() -> None:
         circadian_sleep_chemical_variance_threshold=(
             args.circ_sleep_chemical_variance_threshold
         ),
+        circadian_use_adaptive_sleep_budget=args.circ_use_adaptive_sleep_budget,
+        circadian_adaptive_sleep_budget_min_scale=args.circ_adaptive_sleep_budget_min_scale,
+        circadian_adaptive_sleep_budget_max_scale=args.circ_adaptive_sleep_budget_max_scale,
+        circadian_adaptive_sleep_budget_plateau_weight=(
+            args.circ_adaptive_sleep_budget_plateau_weight
+        ),
+        circadian_adaptive_sleep_budget_variance_weight=(
+            args.circ_adaptive_sleep_budget_variance_weight
+        ),
         circadian_enable_sleep_rollback=args.circ_enable_sleep_rollback,
         circadian_sleep_rollback_tolerance=args.circ_sleep_rollback_tolerance,
         circadian_sleep_rollback_metric=args.circ_sleep_rollback_metric,
@@ -291,6 +326,11 @@ def main() -> None:
         circadian_plasticity_sensitivity_max=args.circ_plasticity_sensitivity_max,
         circadian_plasticity_importance_mix=args.circ_plasticity_importance_mix,
         circadian_min_plasticity=args.circ_min_plasticity,
+        circadian_use_reward_modulated_learning=args.circ_use_reward_modulated_learning,
+        circadian_reward_baseline_decay=args.circ_reward_baseline_decay,
+        circadian_reward_difficulty_exponent=args.circ_reward_difficulty_exponent,
+        circadian_reward_scale_min=args.circ_reward_scale_min,
+        circadian_reward_scale_max=args.circ_reward_scale_max,
         circadian_use_adaptive_thresholds=(
             True
             if args.circ_use_adaptive_thresholds is None

src/app/resnet50_benchmark.py

@@ -70,6 +70,11 @@ class ResNet50BenchmarkConfig:
     circadian_sleep_energy_window: int = 48
     circadian_sleep_plateau_delta: float = 5e-5
     circadian_sleep_chemical_variance_threshold: float = 0.02
+    circadian_use_adaptive_sleep_budget: bool = True
+    circadian_adaptive_sleep_budget_min_scale: float = 0.25
+    circadian_adaptive_sleep_budget_max_scale: float = 1.0
+    circadian_adaptive_sleep_budget_plateau_weight: float = 0.6
+    circadian_adaptive_sleep_budget_variance_weight: float = 0.4
     circadian_enable_sleep_rollback: bool = True
     circadian_sleep_rollback_tolerance: float = 0.002
     circadian_sleep_rollback_metric: str = "cross_entropy"
@@ -91,6 +96,11 @@ class ResNet50BenchmarkConfig:
     circadian_plasticity_sensitivity_max: float = 0.55
     circadian_plasticity_importance_mix: float = 0.50
     circadian_min_plasticity: float = 0.5
+    circadian_use_reward_modulated_learning: bool = False
+    circadian_reward_baseline_decay: float = 0.95
+    circadian_reward_difficulty_exponent: float = 1.0
+    circadian_reward_scale_min: float = 0.75
+    circadian_reward_scale_max: float = 1.5
     circadian_use_adaptive_thresholds: bool = True
     circadian_adaptive_split_percentile: float = 92.0
     circadian_adaptive_prune_percentile: float = 8.0
@@ -484,6 +494,11 @@ def _benchmark_circadian(
         plasticity_sensitivity_max=config.circadian_plasticity_sensitivity_max,
         plasticity_importance_mix=config.circadian_plasticity_importance_mix,
         min_plasticity=config.circadian_min_plasticity,
+        use_reward_modulated_learning=config.circadian_use_reward_modulated_learning,
+        reward_baseline_decay=config.circadian_reward_baseline_decay,
+        reward_difficulty_exponent=config.circadian_reward_difficulty_exponent,
+        reward_scale_min=config.circadian_reward_scale_min,
+        reward_scale_max=config.circadian_reward_scale_max,
         use_adaptive_thresholds=config.circadian_use_adaptive_thresholds,
         adaptive_split_percentile=config.circadian_adaptive_split_percentile,
         adaptive_prune_percentile=config.circadian_adaptive_prune_percentile,
@@ -517,6 +532,11 @@ def _benchmark_circadian(
         sleep_energy_window=config.circadian_sleep_energy_window,
         sleep_plateau_delta=config.circadian_sleep_plateau_delta,
         sleep_chemical_variance_threshold=config.circadian_sleep_chemical_variance_threshold,
+        use_adaptive_sleep_budget=config.circadian_use_adaptive_sleep_budget,
+        adaptive_sleep_budget_min_scale=config.circadian_adaptive_sleep_budget_min_scale,
+        adaptive_sleep_budget_max_scale=config.circadian_adaptive_sleep_budget_max_scale,
+        adaptive_sleep_budget_plateau_weight=config.circadian_adaptive_sleep_budget_plateau_weight,
+        adaptive_sleep_budget_variance_weight=config.circadian_adaptive_sleep_budget_variance_weight,
     )
     model = CircadianPredictiveCodingResNet50Classifier(
         num_classes=loaders.num_classes,
@@ -879,6 +899,14 @@ def _validate_benchmark_config(config: ResNet50BenchmarkConfig) -> None:
         )
     if not (0.0 <= config.circadian_plasticity_importance_mix <= 1.0):
         raise ValueError("circadian_plasticity_importance_mix must be between 0 and 1.")
+    if not (0.0 <= config.circadian_reward_baseline_decay < 1.0):
+        raise ValueError("circadian_reward_baseline_decay must be in [0, 1).")
+    if config.circadian_reward_difficulty_exponent <= 0.0:
+        raise ValueError("circadian_reward_difficulty_exponent must be positive.")
+    if config.circadian_reward_scale_min <= 0.0:
+        raise ValueError("circadian_reward_scale_min must be positive.")
+    if config.circadian_reward_scale_max < config.circadian_reward_scale_min:
+        raise ValueError("circadian_reward_scale_max must be >= circadian_reward_scale_min.")
     if config.circadian_sleep_warmup_steps < 0:
         raise ValueError("circadian_sleep_warmup_steps must be non-negative.")
     if not (0.0 <= config.circadian_sleep_split_only_until_fraction <= 1.0):
@@ -907,6 +935,22 @@ def _validate_benchmark_config(config: ResNet50BenchmarkConfig) -> None:
         raise ValueError("circadian_sleep_plateau_delta must be non-negative.")
     if config.circadian_sleep_chemical_variance_threshold < 0.0:
         raise ValueError("circadian_sleep_chemical_variance_threshold must be non-negative.")
+    if config.circadian_adaptive_sleep_budget_min_scale <= 0.0:
+        raise ValueError("circadian_adaptive_sleep_budget_min_scale must be positive.")
+    if (
+        config.circadian_adaptive_sleep_budget_max_scale
+        < config.circadian_adaptive_sleep_budget_min_scale
+    ):
+        raise ValueError(
+            "circadian_adaptive_sleep_budget_max_scale must be >= "
+            "circadian_adaptive_sleep_budget_min_scale."
+        )
+    if config.circadian_adaptive_sleep_budget_max_scale > 1.0:
+        raise ValueError("circadian_adaptive_sleep_budget_max_scale must be <= 1.0.")
+    if config.circadian_adaptive_sleep_budget_plateau_weight < 0.0:
+        raise ValueError("circadian_adaptive_sleep_budget_plateau_weight must be non-negative.")
+    if config.circadian_adaptive_sleep_budget_variance_weight < 0.0:
+        raise ValueError("circadian_adaptive_sleep_budget_variance_weight must be non-negative.")
     if config.circadian_sleep_rollback_tolerance < 0.0:
         raise ValueError("circadian_sleep_rollback_tolerance must be non-negative.")
     if config.circadian_sleep_rollback_eval_batches < 0: